hi, huajie, fan rui, monster Thank you for your suggestions.
Now please Huajie to confirm how to do it. If remove the variable name, I can also accept it here. I will do small secondary development on our production system based on the community to meet internal needs. 陈卓宇 <[email protected]> 于2022年10月14日周五 15:58写道： > "外部采集整合kafka" is the Chinese description of variable, so it makes sense to > put it under `Description`. > > Chunjin Mu <[email protected]> 于2022年10月14日周五 12:01写道： > > > The variable management module has three properties, including variable > > code, variable name, and description, For example, the variable code is > > "external.kafka.collect.brokers", and the variable name is "外部采集整合kafka". > > When we created the application, I forgot what the variable code was, > but I > > knew it was called "采集整合", so I can search Enter the "采集整合" keywords in > the > > box, and of course you can also search for kafka keywords. > > > > At the same time, I want to know more detailed information about this > > kafka, such as who the operator is, what is the data retention period, > and > > how big is the cluster size, but I can't use the description to search, > so > > I can only search or variable code or variable name, and the variable > name > > is also for a simple display. > > > > Although external.kafka.collect.brokers already contains a lot of > > information, I think this is just a naming convention. When we discuss > this > > variable, we will discuss it by using the "采集整合 kafka" > > > > We can discuss whether there is a problem with this design, what do you > > think? > > > > Rui Fan <[email protected]> 于2022年10月10日周一 11:45写道： > > > > > Hi huajie: > > > > > > > If it is manually entered, it will always be treated as ordinary > text. > > > > > > Users always copy some code from an old job to a new job. > > > I'm afraid it might not meet the user's expectations. > > > > > > Best, > > > fanrui > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <[email protected]> wrote: > > > > > > > hi 田向阳: > > > > > > > > The current discussion is about variable management, which is a > > > relatively > > > > independent function, No other modules involved, you can check the > > thread > > > > [1] > > > > > > > > > > > > [1] thread: > > > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m > > > > > > > > > > > > Best, > > > > Huajie Wang > > > > > > > > > > > > > > > > 田向阳 <[email protected]> 于2022年10月8日周六 13:13写道： > > > > > > > > > is this module include a resource management cente r where jars can > > be > > > > > chosen to provide to create a job > > > > > > > > > > > > > > > > > > > > ---- 回复的原邮件 ---- > > > > > | 发件人 | Huajie Wang<[email protected]> | > > > > > | 日期 | 2022年10月08日 12:07 | > > > > > | 收件人 | [email protected]<[email protected]> | > > > > > | 抄送至 | | > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management module | > > > > > hi 阿穆: > > > > > Thank you for the clarification, My question is answered, but now I > > > have > > > > > one more concern, > > > > > I'm currently using {var} for variables, and I'm worried that one > of > > > the > > > > > user's parameters includes "{" and "}", e.g : > > > > > > > > > > --password 'mypass{123}abc' > > > > > > > > > > So how do we parse parse this? How about ${var} > > > > > > > > > > > > > > > > > > > > Best, > > > > > Huajie Wang > > > > > > > > > > > > > > > > > > > > 阿穆 <[email protected]> 于2022年10月8日周六 11:36写道： > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan: > > > > > > > > > > > > Thanks for your reply. > > > > > > > > > > > > I have integrated your replies. I really don't pay much attention > > to > > > > SQL, > > > > > > so I will do the following. > > > > > > > > > > > > 1.Each team has independent variables that do not affect each > other > > > and > > > > > > are isolated from each other, but the same variable is unique > under > > > the > > > > > > same team, so you need to specify the team when creating > variables > > > > (users > > > > > > may belong to multiple teams, and the teams of the current user > > will > > > be > > > > > > displayed here for users to choose) > > > > > > 2.For Flink jar jobs, there are two ways to pass variables > > > > > > - By using placeholders, such as adding --kafka {mykafka} to the > > args > > > > of > > > > > > the application, 'mykafka' is the variable code of a variable > > > > > > - Search by variable code or variable name. After a variable is > > > > selected, > > > > > > the parameter transfer method is displayed > > > > > > 3.Flink sql will use variables as placeholders, E.g: > > > > > > > > > > > > CREATE TABLE KafkaTableA ( > > > > > > `user_id` BIGINT, > > > > > > `item_id` BIGINT, > > > > > > `type` STRING > > > > > > ) WITH ( > > > > > > 'connector' = 'kafka', > > > > > > 'topic' = 'user_behaviorA', > > > > > > 'properties.bootstrap.servers' = > > > '{kafka_clusterA}', > > > > > -- > > > > > > 'kafka_ ClusterA' is the variable code > > > > > > 'properties.group.id' = 'testGroupA', > > > > > > 'scan.startup.mode' = 'earliest-offset', > > > > > > > > > > > > CREATE TABLE KafkaTableB ( > > > > > > `user_id` BIGINT, > > > > > > `item_id` BIGINT > > > > > > ) WITH ( > > > > > > 'connector' = 'kafka', > > > > > > 'topic' = 'user_behaviorB', > > > > > > 'properties.bootstrap.servers' = > > > '{kafka_clusterB}', > > > > > -- > > > > > > 'kafka_ ClusterB' is the variable code > > > > > > 'properties.group.id' = 'testGroupB', > > > > > > 'scan.startup.mode' = 'earliest-offset', > > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM > > > > > > KafkaTableB -- 'type' is the variable code > > > > > > > > > > > > 4.In the future, we can consider setting some system built-in > > > variables > > > > > > > > > > > > Best, > > > > > > ChunJin Mu > > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <[email protected]> > > > > wrote: > > > > > > > > > > > > > Hi 阿穆: > > > > > > > > > > > > > > It's a great proposal. > > > > > > > > > > > > > > For usage scenarios, I strongly recommend support for > > replacing > > > > > > variables > > > > > > > in sql. E.g: > > > > > > > > > > > > > > CREATE TABLE KafkaTable ( > > > > > > > `user_id` BIGINT, > > > > > > > `item_id` BIGINT > > > > > > > ) WITH ( > > > > > > > 'connector' = 'kafka', > > > > > > > 'topic' = 'user_behavior', > > > > > > > 'properties.bootstrap.servers' = > > > > > '{kafka_clusterA}', > > > > > > > 'properties.group.id' = 'testGroup', > > > > > > > 'scan.startup.mode' = 'earliest-offset', > > > > > > > 'format' = 'csv' > > > > > > > ) > > > > > > > > > > > > > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}', > > > > > > > > > > > > > > For variable management, I have some question: > > > > > > > > > > > > > > 1. Is the parameter used within the team? If all teams use a > > > > common > > > > > > > parameter, it > > > > > > > needs to be defined for each team, right? > > > > > > > 2. Which users can modify variables? Can all users modify > all > > > > > > variables of > > > > > > > their team? > > > > > > > After the variable is modified incorrectly, many jobs may be > > > > > > affected. So I > > > > > > > suggest that > > > > > > > only team admin or system admin have permission to modify, > add > > > and > > > > > > delete > > > > > > > variables. > > > > > > > > > > > > > > Best > > > > > > > fanrui > > > > > > > > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <[email protected] > > > > > > wrote: > > > > > > > > > > > > > hi 阿穆: > > > > > > > > > > > > > > > > > > > > > Thanks for starting a very useful discussion, I have a some > > > > > questions: > > > > > > > > > > > > > > > > > > > > > 1. The access boundary problem of variable management, is > this > > > all > > > > > > > people in the platform can access these variables( no matter > > who > > > > > > > defines) or that only team members can be access these > > > variables. > > > > > > > > > > > > > > 2. There are two usage scenarios for variables that I know > of, > > > one > > > > > is > > > > > > > in the entry parameter of main, and the other is in > flinksql. > > > Are > > > > > both > > > > > > > supported, or only supported main entry ? > > > > > > > > > > > > > > about first question, my opinion: > > > > > > > > > > > > > > 1. The variable management is associated with the team. > After > > > the > > > > > user > > > > > > > logs in to the system, he must select a team (a user can > > belong > > > to > > > > > one > > > > > > > or more teams), The variable belongs to a team > > > > > > > > > > > > > > 2. All resources between teams are isolated, which means > that > > > > > > > variables are also isolated. In short, members of a team can > > use > > > > and > > > > > > > define these variables, which are invisible to other > members, > > > and > > > > > > > other members cannot use these variables. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Welcome to your views > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Huajie Wang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 陈卓宇 <[email protected]> 于2022年10月7日周五 > 14:18写道： > > > > > > > > > > > > > > Is it possible to do this variable support in sql as > well > > > > > > > > > > > > > > > > 阿穆 <[email protected]> 于2022年10月7日周五 > 13:54写道： > > > > > > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > > > > > > > > > > > > > > > > I want to start a discussion about the variable > > > > management > > > > > > module. Please > > > > > > > > > see this issue issue-1477. > > > > > > > > > > > > > > > > > > > > > > > > > > > Background: > > > > > > > > > > > > > > > > > > > > > > > > > > > In the actual production environment, Flink jobs > are > > > > > > generally complex, > > > > > > > > > and usually require multiple external components. > > For > > > > > > example, Flink jobs > > > > > > > > > consume data from Kafka, then connect external > > > > components > > > > > > such as HBase > > > > > > > > or > > > > > > > > > Redis to obtain additional business information, > and > > > > then > > > > > > write it to the > > > > > > > > > downstream external components. There are the > > > following > > > > > > problems: > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. The connection information of external > > components, > > > > such > > > > > > as IP, port > > > > > > > > and > > > > > > > > > user password, needs to be configured in the > > > application > > > > > > args and > > > > > > > > > transferred to the Flink job, so that the > connection > > > > > > information of > > > > > > > > > external components is distributed in multiple > > > > > > applications. Once the > > > > > > > > > connection information of external components > > changes, > > > > > many > > > > > > application > > > > > > > > > args parameters need to be modified, which will > lead > > > to > > > > > > high operation > > > > > > > > and > > > > > > > > > maintenance costs. > > > > > > > > > 2. There are many people in the team developing > > > > real-time > > > > > > computing jobs. > > > > > > > > > There is no uniform specification for the > connection > > > > > > information of the > > > > > > > > > external components passed to the job, resulting > in > > > > > > different parameter > > > > > > > > > names of the same component. This is difficult to > > > count > > > > > > which external > > > > > > > > > components are dependent. > > > > > > > > > 3. In production practice, there are usually > > multiple > > > > sets > > > > > > of > > > > > > > > > environments, such as test environment and > > production > > > > > > environment. It is > > > > > > > > > not intuitive to judge whether a component belongs > > to > > > a > > > > > > test environment > > > > > > > > or > > > > > > > > > a production environment simply through IP and > > ports. > > > > Even > > > > > > if it can be > > > > > > > > > judged, there are some omissions. In this way, the > > > > > > connection information > > > > > > > > > online to the production environment may be > external > > > > > > components of the > > > > > > > > test > > > > > > > > > environment, or components of the production > > > environment > > > > > are > > > > > > > > inadvertently > > > > > > > > > configured during testing, leading to production > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > Proposal: > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Create a unified module to manage variables. > > Users > > > > can > > > > > > define their > > > > > > > > own > > > > > > > > > variables in this module. Variable information > > > includes > > > > > > variable code > > > > > > > > > (passed to Flink program as parameter name), > > variable > > > > > > value, variable > > > > > > > name, > > > > > > > > > variable description, and which applications > depend > > on > > > > > > > > > 2. In the same Team module, the variable code or > > > > variable > > > > > > name is unique > > > > > > > > > - By using placeholders, such as adding -- kafka > > > > {mykafka} > > > > > > to the args of > > > > > > > > > the application, mykafka is the variable code of a > > > > > variable > > > > > > > > > - Search by variable code or variable name. After > a > > > > > > variable is selected, > > > > > > > > > the parameter transfer method is displayed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If you have any suggestions, welcome to > participate > > in > > > > the > > > > > > discussion on > > > > > > > > > the mail list or issue, I look forward to your > > > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > > Best wishes > > > > > > > > > > > > > > > > > > > > > > >
