Hi guys, I agree with monster. And I suggest use variable name, variable value and description.
By the way, there are some problems still need to confirm. What’s the placeholder? And how to use them when edit sql or datastream job? It’s better to discuss these asap. Best fanrui On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <[email protected]> wrote: > hi, huajie, fan rui, monster > > Thank you for your suggestions. > > Now please Huajie to confirm how to do it. If remove the variable name, I > can also accept it here. I will do small secondary development on our > production system based on the community to meet internal needs. > > 陈卓宇 <[email protected]> 于2022年10月14日周五 15:58写道： > > > "外部采集整合kafka" is the Chinese description of variable, so it makes sense > to > > put it under `Description`. > > > > Chunjin Mu <[email protected]> 于2022年10月14日周五 12:01写道： > > > > > The variable management module has three properties, including variable > > > code, variable name, and description, For example, the variable code is > > > "external.kafka.collect.brokers", and the variable name is > "外部采集整合kafka". > > > When we created the application, I forgot what the variable code was, > > but I > > > knew it was called "采集整合", so I can search Enter the "采集整合" keywords in > > the > > > box, and of course you can also search for kafka keywords. > > > > > > At the same time, I want to know more detailed information about this > > > kafka, such as who the operator is, what is the data retention period, > > and > > > how big is the cluster size, but I can't use the description to search, > > so > > > I can only search or variable code or variable name, and the variable > > name > > > is also for a simple display. > > > > > > Although external.kafka.collect.brokers already contains a lot of > > > information, I think this is just a naming convention. When we discuss > > this > > > variable, we will discuss it by using the "采集整合 kafka" > > > > > > We can discuss whether there is a problem with this design, what do you > > > think? > > > > > > Rui Fan <[email protected]> 于2022年10月10日周一 11:45写道： > > > > > > > Hi huajie: > > > > > > > > > If it is manually entered, it will always be treated as ordinary > > text. > > > > > > > > Users always copy some code from an old job to a new job. > > > > I'm afraid it might not meet the user's expectations. > > > > > > > > Best, > > > > fanrui > > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <[email protected]> > wrote: > > > > > > > > > hi 田向阳: > > > > > > > > > > The current discussion is about variable management, which is a > > > > relatively > > > > > independent function, No other modules involved, you can check the > > > thread > > > > > [1] > > > > > > > > > > > > > > > [1] thread: > > > > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m > > > > > > > > > > > > > > > Best, > > > > > Huajie Wang > > > > > > > > > > > > > > > > > > > > 田向阳 <[email protected]> 于2022年10月8日周六 13:13写道： > > > > > > > > > > > is this module include a resource management cente r where jars > can > > > be > > > > > > chosen to provide to create a job > > > > > > > > > > > > > > > > > > > > > > > > ---- 回复的原邮件 ---- > > > > > > | 发件人 | Huajie Wang<[email protected]> | > > > > > > | 日期 | 2022年10月08日 12:07 | > > > > > > | 收件人 | [email protected]<[email protected]> | > > > > > > | 抄送至 | | > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management module | > > > > > > hi 阿穆: > > > > > > Thank you for the clarification, My question is answered, but > now I > > > > have > > > > > > one more concern, > > > > > > I'm currently using {var} for variables, and I'm worried that one > > of > > > > the > > > > > > user's parameters includes "{" and "}", e.g : > > > > > > > > > > > > --password 'mypass{123}abc' > > > > > > > > > > > > So how do we parse parse this? How about ${var} > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > Huajie Wang > > > > > > > > > > > > > > > > > > > > > > > > 阿穆 <[email protected]> 于2022年10月8日周六 11:36写道： > > > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan: > > > > > > > > > > > > > > Thanks for your reply. > > > > > > > > > > > > > > I have integrated your replies. I really don't pay much > attention > > > to > > > > > SQL, > > > > > > > so I will do the following. > > > > > > > > > > > > > > 1.Each team has independent variables that do not affect each > > other > > > > and > > > > > > > are isolated from each other, but the same variable is unique > > under > > > > the > > > > > > > same team, so you need to specify the team when creating > > variables > > > > > (users > > > > > > > may belong to multiple teams, and the teams of the current user > > > will > > > > be > > > > > > > displayed here for users to choose) > > > > > > > 2.For Flink jar jobs, there are two ways to pass variables > > > > > > > - By using placeholders, such as adding --kafka {mykafka} to > the > > > args > > > > > of > > > > > > > the application, 'mykafka' is the variable code of a variable > > > > > > > - Search by variable code or variable name. After a variable is > > > > > selected, > > > > > > > the parameter transfer method is displayed > > > > > > > 3.Flink sql will use variables as placeholders, E.g: > > > > > > > > > > > > > > CREATE TABLE KafkaTableA ( > > > > > > > `user_id` BIGINT, > > > > > > > `item_id` BIGINT, > > > > > > > `type` STRING > > > > > > > ) WITH ( > > > > > > > 'connector' = 'kafka', > > > > > > > 'topic' = 'user_behaviorA', > > > > > > > 'properties.bootstrap.servers' = > > > > '{kafka_clusterA}', > > > > > > -- > > > > > > > 'kafka_ ClusterA' is the variable code > > > > > > > 'properties.group.id' = 'testGroupA', > > > > > > > 'scan.startup.mode' = 'earliest-offset', > > > > > > > > > > > > > > CREATE TABLE KafkaTableB ( > > > > > > > `user_id` BIGINT, > > > > > > > `item_id` BIGINT > > > > > > > ) WITH ( > > > > > > > 'connector' = 'kafka', > > > > > > > 'topic' = 'user_behaviorB', > > > > > > > 'properties.bootstrap.servers' = > > > > '{kafka_clusterB}', > > > > > > -- > > > > > > > 'kafka_ ClusterB' is the variable code > > > > > > > 'properties.group.id' = 'testGroupB', > > > > > > > 'scan.startup.mode' = 'earliest-offset', > > > > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM > > > > > > > KafkaTableB -- 'type' is the variable code > > > > > > > > > > > > > > 4.In the future, we can consider setting some system built-in > > > > variables > > > > > > > > > > > > > > Best, > > > > > > > ChunJin Mu > > > > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <[email protected] > > > > > > > wrote: > > > > > > > > > > > > > > > Hi 阿穆: > > > > > > > > > > > > > > > > It's a great proposal. > > > > > > > > > > > > > > > > For usage scenarios, I strongly recommend support for > > > replacing > > > > > > > variables > > > > > > > > in sql. E.g: > > > > > > > > > > > > > > > > CREATE TABLE KafkaTable ( > > > > > > > > `user_id` BIGINT, > > > > > > > > `item_id` BIGINT > > > > > > > > ) WITH ( > > > > > > > > 'connector' = 'kafka', > > > > > > > > 'topic' = 'user_behavior', > > > > > > > > 'properties.bootstrap.servers' = > > > > > > '{kafka_clusterA}', > > > > > > > > 'properties.group.id' = 'testGroup', > > > > > > > > 'scan.startup.mode' = 'earliest-offset', > > > > > > > > 'format' = 'csv' > > > > > > > > ) > > > > > > > > > > > > > > > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}', > > > > > > > > > > > > > > > > For variable management, I have some question: > > > > > > > > > > > > > > > > 1. Is the parameter used within the team? If all teams > use a > > > > > common > > > > > > > > parameter, it > > > > > > > > needs to be defined for each team, right? > > > > > > > > 2. Which users can modify variables? Can all users modify > > all > > > > > > > variables of > > > > > > > > their team? > > > > > > > > After the variable is modified incorrectly, many jobs may > be > > > > > > > affected. So I > > > > > > > > suggest that > > > > > > > > only team admin or system admin have permission to modify, > > add > > > > and > > > > > > > delete > > > > > > > > variables. > > > > > > > > > > > > > > > > Best > > > > > > > > fanrui > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <[email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > hi 阿穆: > > > > > > > > > > > > > > > > > > > > > > > > Thanks for starting a very useful discussion, I have a > some > > > > > > questions: > > > > > > > > > > > > > > > > > > > > > > > > 1. The access boundary problem of variable management, is > > this > > > > all > > > > > > > > people in the platform can access these variables( no > matter > > > who > > > > > > > > defines) or that only team members can be access these > > > > variables. > > > > > > > > > > > > > > > > 2. There are two usage scenarios for variables that I know > > of, > > > > one > > > > > > is > > > > > > > > in the entry parameter of main, and the other is in > > flinksql. > > > > Are > > > > > > both > > > > > > > > supported, or only supported main entry ? > > > > > > > > > > > > > > > > about first question, my opinion: > > > > > > > > > > > > > > > > 1. The variable management is associated with the team. > > After > > > > the > > > > > > user > > > > > > > > logs in to the system, he must select a team (a user can > > > belong > > > > to > > > > > > one > > > > > > > > or more teams), The variable belongs to a team > > > > > > > > > > > > > > > > 2. All resources between teams are isolated, which means > > that > > > > > > > > variables are also isolated. In short, members of a team > can > > > use > > > > > and > > > > > > > > define these variables, which are invisible to other > > members, > > > > and > > > > > > > > other members cannot use these variables. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Welcome to your views > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > Huajie Wang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 陈卓宇 <[email protected]> 于2022年10月7日周五 > > 14:18写道： > > > > > > > > > > > > > > > > Is it possible to do this variable support in sql as > > well > > > > > > > > > > > > > > > > > > 阿穆 <[email protected]> 于2022年10月7日周五 > > 13:54写道： > > > > > > > > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I want to start a discussion about the variable > > > > > management > > > > > > > module. Please > > > > > > > > > > see this issue issue-1477. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Background: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the actual production environment, Flink jobs > > are > > > > > > > generally complex, > > > > > > > > > > and usually require multiple external > components. > > > For > > > > > > > example, Flink jobs > > > > > > > > > > consume data from Kafka, then connect external > > > > > components > > > > > > > such as HBase > > > > > > > > > or > > > > > > > > > > Redis to obtain additional business information, > > and > > > > > then > > > > > > > write it to the > > > > > > > > > > downstream external components. There are the > > > > following > > > > > > > problems: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. The connection information of external > > > components, > > > > > such > > > > > > > as IP, port > > > > > > > > > and > > > > > > > > > > user password, needs to be configured in the > > > > application > > > > > > > args and > > > > > > > > > > transferred to the Flink job, so that the > > connection > > > > > > > information of > > > > > > > > > > external components is distributed in multiple > > > > > > > applications. Once the > > > > > > > > > > connection information of external components > > > changes, > > > > > > many > > > > > > > application > > > > > > > > > > args parameters need to be modified, which will > > lead > > > > to > > > > > > > high operation > > > > > > > > > and > > > > > > > > > > maintenance costs. > > > > > > > > > > 2. There are many people in the team developing > > > > > real-time > > > > > > > computing jobs. > > > > > > > > > > There is no uniform specification for the > > connection > > > > > > > information of the > > > > > > > > > > external components passed to the job, resulting > > in > > > > > > > different parameter > > > > > > > > > > names of the same component. This is difficult > to > > > > count > > > > > > > which external > > > > > > > > > > components are dependent. > > > > > > > > > > 3. In production practice, there are usually > > > multiple > > > > > sets > > > > > > > of > > > > > > > > > > environments, such as test environment and > > > production > > > > > > > environment. It is > > > > > > > > > > not intuitive to judge whether a component > belongs > > > to > > > > a > > > > > > > test environment > > > > > > > > > or > > > > > > > > > > a production environment simply through IP and > > > ports. > > > > > Even > > > > > > > if it can be > > > > > > > > > > judged, there are some omissions. In this way, > the > > > > > > > connection information > > > > > > > > > > online to the production environment may be > > external > > > > > > > components of the > > > > > > > > > test > > > > > > > > > > environment, or components of the production > > > > environment > > > > > > are > > > > > > > > > inadvertently > > > > > > > > > > configured during testing, leading to production > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Proposal: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Create a unified module to manage variables. > > > Users > > > > > can > > > > > > > define their > > > > > > > > > own > > > > > > > > > > variables in this module. Variable information > > > > includes > > > > > > > variable code > > > > > > > > > > (passed to Flink program as parameter name), > > > variable > > > > > > > value, variable > > > > > > > > name, > > > > > > > > > > variable description, and which applications > > depend > > > on > > > > > > > > > > 2. In the same Team module, the variable code or > > > > > variable > > > > > > > name is unique > > > > > > > > > > - By using placeholders, such as adding -- kafka > > > > > {mykafka} > > > > > > > to the args of > > > > > > > > > > the application, mykafka is the variable code > of a > > > > > > variable > > > > > > > > > > - Search by variable code or variable name. > After > > a > > > > > > > variable is selected, > > > > > > > > > > the parameter transfer method is displayed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If you have any suggestions, welcome to > > participate > > > in > > > > > the > > > > > > > discussion on > > > > > > > > > > the mail list or issue, I look forward to your > > > > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best wishes > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
