is this module include a resource management cente r where jars can be chosen to provide to create a job
---- 回复的原邮件 ---- | 发件人 | Huajie Wang<[email protected]> | | 日期 | 2022年10月08日 12:07 | | 收件人 | [email protected]<[email protected]> | | 抄送至 | | | 主题 | Re: [DISCUSS][issue-1477] Add variable management module | hi 阿穆: Thank you for the clarification, My question is answered, but now I have one more concern, I'm currently using {var} for variables, and I'm worried that one of the user's parameters includes "{" and "}", e.g : --password 'mypass{123}abc' So how do we parse parse this? How about ${var} Best, Huajie Wang 阿穆 <[email protected]> 于2022年10月8日周六 11:36写道： > Hi, monster, Huajie Wang, Rui Fan: > > Thanks for your reply. > > I have integrated your replies. I really don't pay much attention to SQL, > so I will do the following. > > 1.Each team has independent variables that do not affect each other and > are isolated from each other, but the same variable is unique under the > same team, so you need to specify the team when creating variables (users > may belong to multiple teams, and the teams of the current user will be > displayed here for users to choose) > 2.For Flink jar jobs, there are two ways to pass variables > - By using placeholders, such as adding --kafka {mykafka} to the args of > the application, 'mykafka' is the variable code of a variable > - Search by variable code or variable name. After a variable is selected, > the parameter transfer method is displayed > 3.Flink sql will use variables as placeholders, E.g: > > CREATE TABLE KafkaTableA ( > `user_id` BIGINT, > `item_id` BIGINT, > `type` STRING > ) WITH ( > 'connector' = 'kafka', > 'topic' = 'user_behaviorA', > 'properties.bootstrap.servers' = '{kafka_clusterA}', -- > 'kafka_ ClusterA' is the variable code > 'properties.group.id' = 'testGroupA', > 'scan.startup.mode' = 'earliest-offset', > > CREATE TABLE KafkaTableB ( > `user_id` BIGINT, > `item_id` BIGINT > ) WITH ( > 'connector' = 'kafka', > 'topic' = 'user_behaviorB', > 'properties.bootstrap.servers' = '{kafka_clusterB}', -- > 'kafka_ ClusterB' is the variable code > 'properties.group.id' = 'testGroupB', > 'scan.startup.mode' = 'earliest-offset', > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM > KafkaTableB -- 'type' is the variable code > > 4.In the future, we can consider setting some system built-in variables > > Best, > ChunJin Mu > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <[email protected]> wrote: > > > Hi 阿穆: > > > > It's a great proposal. > > > > For usage scenarios, I strongly recommend support for replacing > variables > > in sql. E.g: > > > > CREATE TABLE KafkaTable ( > > `user_id` BIGINT, > > `item_id` BIGINT > > ) WITH ( > > 'connector' = 'kafka', > > 'topic' = 'user_behavior', > > 'properties.bootstrap.servers' = '{kafka_clusterA}', > > 'properties.group.id' = 'testGroup', > > 'scan.startup.mode' = 'earliest-offset', > > 'format' = 'csv' > > ) > > > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}', > > > > For variable management, I have some question: > > > > 1. Is the parameter used within the team? If all teams use a common > > parameter, it > > needs to be defined for each team, right? > > 2. Which users can modify variables? Can all users modify all > variables of > > their team? > > After the variable is modified incorrectly, many jobs may be > affected. So I > > suggest that > > only team admin or system admin have permission to modify, add and > delete > > variables. > > > > Best > > fanrui > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <[email protected]> wrote: > > > hi 阿穆: > > > > > > Thanks for starting a very useful discussion, I have a some questions: > > > > > > 1. The access boundary problem of variable management, is this all > > people in the platform can access these variables( no matter who > > defines) or that only team members can be access these variables. > > > > 2. There are two usage scenarios for variables that I know of, one is > > in the entry parameter of main, and the other is in flinksql. Are both > > supported, or only supported main entry ? > > > > about first question, my opinion: > > > > 1. The variable management is associated with the team. After the user > > logs in to the system, he must select a team (a user can belong to one > > or more teams), The variable belongs to a team > > > > 2. All resources between teams are isolated, which means that > > variables are also isolated. In short, members of a team can use and > > define these variables, which are invisible to other members, and > > other members cannot use these variables. > > > > > > > > Welcome to your views > > > > > > Best, > > Huajie Wang > > > > > > > > > > 陈卓宇 <[email protected]> 于2022年10月7日周五 14:18写道： > > > > Is it possible to do this variable support in sql as well > > > > > > 阿穆 <[email protected]> 于2022年10月7日周五 13:54写道： > > > > > > > Hi guys, > > > > > > > > > > > > I want to start a discussion about the variable management > module. Please > > > > see this issue issue-1477. > > > > > > > > > > > > Background: > > > > > > > > > > > > In the actual production environment, Flink jobs are > generally complex, > > > > and usually require multiple external components. For > example, Flink jobs > > > > consume data from Kafka, then connect external components > such as HBase > > > or > > > > Redis to obtain additional business information, and then > write it to the > > > > downstream external components. There are the following > problems: > > > > > > > > > > > > 1. The connection information of external components, such > as IP, port > > > and > > > > user password, needs to be configured in the application > args and > > > > transferred to the Flink job, so that the connection > information of > > > > external components is distributed in multiple > applications. Once the > > > > connection information of external components changes, many > application > > > > args parameters need to be modified, which will lead to > high operation > > > and > > > > maintenance costs. > > > > 2. There are many people in the team developing real-time > computing jobs. > > > > There is no uniform specification for the connection > information of the > > > > external components passed to the job, resulting in > different parameter > > > > names of the same component. This is difficult to count > which external > > > > components are dependent. > > > > 3. In production practice, there are usually multiple sets > of > > > > environments, such as test environment and production > environment. It is > > > > not intuitive to judge whether a component belongs to a > test environment > > > or > > > > a production environment simply through IP and ports. Even > if it can be > > > > judged, there are some omissions. In this way, the > connection information > > > > online to the production environment may be external > components of the > > > test > > > > environment, or components of the production environment are > > > inadvertently > > > > configured during testing, leading to production failures. > > > > > > > > > > > > Proposal: > > > > > > > > > > > > 1. Create a unified module to manage variables. Users can > define their > > > own > > > > variables in this module. Variable information includes > variable code > > > > (passed to Flink program as parameter name), variable > value, variable > > name, > > > > variable description, and which applications depend on > > > > 2. In the same Team module, the variable code or variable > name is unique > > > > - By using placeholders, such as adding -- kafka {mykafka} > to the args of > > > > the application, mykafka is the variable code of a variable > > > > - Search by variable code or variable name. After a > variable is selected, > > > > the parameter transfer method is displayed > > > > > > > > > > > > > > > > If you have any suggestions, welcome to participate in the > discussion on > > > > the mail list or issue, I look forward to your feedback. > > > > > > > > > > > > Best wishes > > >
