Hi fanru, Thank you very much for your hard review. You have considered it very comprehensively, I also saw your question in the PR, and I have already responded, please check this reply [1]
[1] https://github.com/apache/incubator-streampark/pull/1831#discussion_r996293537 Best, Chunjin Mu Rui Fan <[email protected]> 于2022年10月15日周六 18:54写道： > Hi chunjin, > > Very sorry for slowing down your progress, I found some > small problems when I reviewed your code. > > Who can create the variable and update the variable? > System Admin and team_admin? Or every developer? > > As I understand, we didn't discuss this point, right? If I > miss any information please let me know, thanks a lot. > > Best, > fanrui > > On Sat, Oct 15, 2022 at 4:44 PM Chunjin Mu <[email protected]> wrote: > > > Hi fanru, > > > > > I have a little question. If I copy some sql from a old app, and the > sql > > includes some placeholders. These placeholders will take effect in the > > new app, right? > > > > yes, the copied placeholder will take effect if the variable exists. > > > > I have added all the information to proposal issue-1477[1]，include the > > questions you asked above， if there is more discussion later, I will add > > this to the proposal. > > > > [1] https://github.com/apache/incubator-streampark/issues/1477 > > > > > > Rui Fan <[email protected]> 于2022年10月15日周六 13:17写道： > > > > > Hi chunjin, > > > > > > Thanks for your detailed information. > > > > > > > When the focus is lost, it is judged whether there is a variable > > > placeholder, and if so, a verification button is displayed to prompt > the > > > user to perform verification. When the user clicks the verification > > button, > > > the replaced content of the placeholder is displayed, and only the user > > > clicks the verification button to click the submit button > > > > > > I have a little question. If I copy some sql from a old app, and the > sql > > > includes some placeholders. These placeholders will take effect in the > > > new app, right? > > > > > > BTW, could you add all information to your proposal issue-1477[1]? > > > The mail list is for discussion only, and we should maintain the final > > > solution in the Proposal. > > > > > > > > > [1] https://github.com/apache/incubator-streampark/issues/1477 > > > > > > Best > > > fanrui > > > > > > On Sat, Oct 15, 2022 at 11:56 AM Chunjin Mu <[email protected]> > > wrote: > > > > > > > Hi guys, > > > > > > > > Thanks for the discussion and great suggestions. > > > > > > > > Here I summarize the functions to be completed in version 1.2.4 > > > > > > > > 1.The table structure of the variable contains the following main > > > contents > > > > - variable code is placeholder, For example the variable code is > > > > "collect.kafka.cluster", it will be referenced as > > > ${collect.kafka.cluster}, > > > > it is unique within a team > > > > - variable name is removed, after discussion, it is not necessarily > > > > retained > > > > - variable value is the value of a specific variable, such as ip and > > port > > > > - description is a detailed introduction > > > > - team is the team the variable belongs to > > > > > > > > 2.Basic functions of variable management > > > > - Add and modify variables: need to check whether the variable code > is > > > > unique under the current team > > > > - Deleting a variable: It is necessary to check whether it has been > > > > referenced by the Application under the team to which the variable > > > belongs. > > > > If there is metadata management in the future, it is also necessary > to > > > > check whether it has been referenced by the metadata. If it is > > > referenced, > > > > it will prompt that the variable is referenced and cannot be deleted. > > > > > > > > 3.References in Flink sql and program args > > > > - In the application, the variables are uniformly referenced in the > > form > > > of > > > > placeholders ${xxx}, such as in Flink sql, program args > > > > - For a better experience, it is necessary to provide the function of > > > > following the search prompt when entering variable placeholders. For > > > > example, when entering "${", the variable list is displayed, and when > > > > "${kafka" is entered, the display contains variables for kafka > strings, > > > in > > > > "variable code" or "description". > > > > - When the focus is lost, it is judged whether there is a variable > > > > placeholder, and if so, a verification button is displayed to prompt > > the > > > > user to perform verification. When the user clicks the verification > > > button, > > > > the replaced content of the placeholder is displayed, and only the > user > > > > clicks the verification button to click the submit button > > > > - If the user enters "okir23${j2Rts}pEt", this placeholder "${j2Rts}" > > > will > > > > not be replaced if "j2Rts" is not a variable we defined > > > > > > > > 4.Future plan > > > > - Display the application or metadata list of a referenced variable > in > > > the > > > > variable basic management module > > > > - Improve user experience > > > > > > > > Best, > > > > Chunjin Mu > > > > > > > > Chunjin Mu <[email protected]> 于2022年10月14日周五 22:28写道： > > > > > > > > > When we have dozens or dozens of variables, this phenomenon may be > > more > > > > > prominent > > > > > > > > > > > > > > > Best, > > > > > Chunjin Mu > > > > > > > > > > Chunjin Mu <[email protected]> 于2022年10月14日周五 22:26写道： > > > > > > > > > >> When a person asks which kafka should be used, I tell him in > Chinese > > > > that > > > > >> he should use "采集整合的kafka", in fact the variable code is > > > collect.kafka, > > > > but > > > > >> he is searching for gather keyword, it is difficult to just pass > > > > >> collect.kafka The intuitive understanding is to "采集整合的kafka", > > because > > > > >> the information contained in the variable code is still not that > > much. > > > > >> > > > > >> Huajie Wang <[email protected]> 于2022年10月14日周五 22:17写道： > > > > >> > > > > >>> I just thought about it, there is really no need for this name. > > > > >>> > > > > >>> > > > > >>> > > > > >>> Best, > > > > >>> Huajie Wang > > > > >>> > > > > >>> > > > > >>> > > > > >>> Rui Fan <[email protected]> 于2022年10月14日周五 21:54写道： > > > > >>> > > > > >>> > Hi guys, > > > > >>> > > > > > >>> > I understand chunjin's solution is this: > > > > >>> > > > > > >>> > - variable code is placeholder > > > > >>> > - variable name is an introduction > > > > >>> > - Description is a detailed introduction > > > > >>> > > > > > >>> > I think introduction and detail are redundant. > > > > >>> > > > > > >>> > For example: there are two kafka clusters, the Beijing kafka > > > > >>> > cluster and the Hangzhou kafka cluster. > > > > >>> > > > > > >>> > If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why > > > > >>> > does the introduction still need `北京 kafka 集群` and > > > > >>> > `杭州 kafka 集群`? placeholder is already clear. > > > > >>> > > > > > >>> > If dev may forget the placeholder, then dev may also > > > > >>> > forget the introduction. > > > > >>> > > > > > >>> > Of course, if the placeholder is kafka1 kafka2, an > > > > >>> > introduction is required. Why do we need an introduction > > > > >>> > at this point, because the placeholder name is unreasonable. > > > > >>> > > > > > >>> > Just like variable naming in Java code, there is no need to > > > > >>> > introduce an introduction with a clear name. If the meaning of > > > > >>> > the variable is complicated, we also need to write a detailed > > > > >>> > description. > > > > >>> > > > > > >>> > Finally, I want to say: less is more. If we want to make > > > > >>> > something complicated, please give everyone a strong reason. > > > > >>> > > > > > >>> > Best > > > > >>> > fanrui > > > > >>> > > > > > >>> > On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <[email protected] > > > > > > wrote: > > > > >>> > > > > > >>> > > Hi guys, > > > > >>> > > > > > > >>> > > I agree with Chunjin, I think variable name is useful, > Chunjin > > > also > > > > >>> > > explained the reason. > > > > >>> > > > > > > >>> > > > > > > >>> > > Best, > > > > >>> > > Huajie Wang > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > Chunjin Mu <[email protected]> 于2022年10月14日周五 20:22写道： > > > > >>> > > > > > > >>> > > > Hi, Rui Fan > > > > >>> > > > I'll start a discussion later. > > > > >>> > > > > > > > >>> > > > Best > > > > >>> > > > Chunjin Mu > > > > >>> > > > > > > > >>> > > > Rui Fan <[email protected]> 于2022年10月14日周五 19:42写道： > > > > >>> > > > > > > > >>> > > > > Hi guys, > > > > >>> > > > > > > > > >>> > > > > I agree with monster. And I suggest use variable name, > > > variable > > > > >>> value > > > > >>> > > and > > > > >>> > > > > description. > > > > >>> > > > > > > > > >>> > > > > By the way, there are some problems still need to > confirm. > > > > >>> What’s the > > > > >>> > > > > placeholder? And how to use them when edit sql or > > datastream > > > > job? > > > > >>> > It’s > > > > >>> > > > > better to discuss these asap. > > > > >>> > > > > > > > > >>> > > > > Best > > > > >>> > > > > fanrui > > > > >>> > > > > > > > > >>> > > > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu < > > > [email protected] > > > > > > > > > >>> > wrote: > > > > >>> > > > > > > > > >>> > > > > > hi, huajie, fan rui, monster > > > > >>> > > > > > > > > > >>> > > > > > Thank you for your suggestions. > > > > >>> > > > > > > > > > >>> > > > > > Now please Huajie to confirm how to do it. If remove > the > > > > >>> variable > > > > >>> > > > name, I > > > > >>> > > > > > can also accept it here. I will do small secondary > > > > development > > > > >>> on > > > > >>> > our > > > > >>> > > > > > production system based on the community to meet > internal > > > > >>> needs. > > > > >>> > > > > > > > > > >>> > > > > > 陈卓宇 <[email protected]> 于2022年10月14日周五 > > 15:58写道： > > > > >>> > > > > > > > > > >>> > > > > > > "外部采集整合kafka" is the Chinese description of variable, > > so > > > it > > > > >>> makes > > > > >>> > > > sense > > > > >>> > > > > > to > > > > >>> > > > > > > put it under `Description`. > > > > >>> > > > > > > > > > > >>> > > > > > > Chunjin Mu <[email protected]> 于2022年10月14日周五 > > > 12:01写道： > > > > >>> > > > > > > > > > > >>> > > > > > > > The variable management module has three > properties, > > > > >>> including > > > > >>> > > > > variable > > > > >>> > > > > > > > code, variable name, and description, For example, > > the > > > > >>> variable > > > > >>> > > > code > > > > >>> > > > > is > > > > >>> > > > > > > > "external.kafka.collect.brokers", and the variable > > name > > > > is > > > > >>> > > > > > "外部采集整合kafka". > > > > >>> > > > > > > > When we created the application, I forgot what the > > > > variable > > > > >>> > code > > > > >>> > > > was, > > > > >>> > > > > > > but I > > > > >>> > > > > > > > knew it was called "采集整合", so I can search Enter > the > > > > "采集整合" > > > > >>> > > > keywords > > > > >>> > > > > in > > > > >>> > > > > > > the > > > > >>> > > > > > > > box, and of course you can also search for kafka > > > > keywords. > > > > >>> > > > > > > > > > > > >>> > > > > > > > At the same time, I want to know more detailed > > > > information > > > > >>> > about > > > > >>> > > > this > > > > >>> > > > > > > > kafka, such as who the operator is, what is the > data > > > > >>> retention > > > > >>> > > > > period, > > > > >>> > > > > > > and > > > > >>> > > > > > > > how big is the cluster size, but I can't use the > > > > >>> description to > > > > >>> > > > > search, > > > > >>> > > > > > > so > > > > >>> > > > > > > > I can only search or variable code or variable > name, > > > and > > > > >>> the > > > > >>> > > > variable > > > > >>> > > > > > > name > > > > >>> > > > > > > > is also for a simple display. > > > > >>> > > > > > > > > > > > >>> > > > > > > > Although external.kafka.collect.brokers already > > > contains > > > > a > > > > >>> lot > > > > >>> > of > > > > >>> > > > > > > > information, I think this is just a naming > > convention. > > > > >>> When we > > > > >>> > > > > discuss > > > > >>> > > > > > > this > > > > >>> > > > > > > > variable, we will discuss it by using the "采集整合 > > kafka" > > > > >>> > > > > > > > > > > > >>> > > > > > > > We can discuss whether there is a problem with this > > > > design, > > > > >>> > what > > > > >>> > > do > > > > >>> > > > > you > > > > >>> > > > > > > > think? > > > > >>> > > > > > > > > > > > >>> > > > > > > > Rui Fan <[email protected]> 于2022年10月10日周一 > > 11:45写道： > > > > >>> > > > > > > > > > > > >>> > > > > > > > > Hi huajie: > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > If it is manually entered, it will always be > > > treated > > > > as > > > > >>> > > > ordinary > > > > >>> > > > > > > text. > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > Users always copy some code from an old job to a > > new > > > > job. > > > > >>> > > > > > > > > I'm afraid it might not meet the user's > > expectations. > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > Best, > > > > >>> > > > > > > > > fanrui > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang < > > > > >>> > [email protected]> > > > > >>> > > > > > wrote: > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > hi 田向阳: > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > The current discussion is about variable > > > management, > > > > >>> which > > > > >>> > > is a > > > > >>> > > > > > > > > relatively > > > > >>> > > > > > > > > > independent function, No other modules > involved, > > > you > > > > >>> can > > > > >>> > > check > > > > >>> > > > > the > > > > >>> > > > > > > > thread > > > > >>> > > > > > > > > > [1] > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > [1] thread: > > > > >>> > > > > > > > > > > > > > >>> > > > > > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > Best, > > > > >>> > > > > > > > > > Huajie Wang > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > 田向阳 <[email protected]> 于2022年10月8日周六 13:13写道： > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > is this module include a resource management > > > cente > > > > r > > > > >>> > where > > > > >>> > > > jars > > > > >>> > > > > > can > > > > >>> > > > > > > > be > > > > >>> > > > > > > > > > > chosen to provide to create a job > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > ---- 回复的原邮件 ---- > > > > >>> > > > > > > > > > > | 发件人 | Huajie Wang<[email protected]> | > > > > >>> > > > > > > > > > > | 日期 | 2022年10月08日 12:07 | > > > > >>> > > > > > > > > > > | 收件人 | [email protected]< > > > > >>> > > [email protected]> > > > > >>> > > > | > > > > >>> > > > > > > > > > > | 抄送至 | | > > > > >>> > > > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable > > > > >>> management > > > > >>> > > > > module | > > > > >>> > > > > > > > > > > hi 阿穆: > > > > >>> > > > > > > > > > > Thank you for the clarification, My question > is > > > > >>> answered, > > > > >>> > > but > > > > >>> > > > > > now I > > > > >>> > > > > > > > > have > > > > >>> > > > > > > > > > > one more concern, > > > > >>> > > > > > > > > > > I'm currently using {var} for variables, and > > I'm > > > > >>> worried > > > > >>> > > that > > > > >>> > > > > one > > > > >>> > > > > > > of > > > > >>> > > > > > > > > the > > > > >>> > > > > > > > > > > user's parameters includes "{" and "}", e.g : > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > --password 'mypass{123}abc' > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > So how do we parse parse this? How about > > ${var} > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > Best, > > > > >>> > > > > > > > > > > Huajie Wang > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > 阿穆 <[email protected]> 于2022年10月8日周六 > > > > 11:36写道： > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan: > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > Thanks for your reply. > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > I have integrated your replies. I really > > don't > > > > pay > > > > >>> much > > > > >>> > > > > > attention > > > > >>> > > > > > > > to > > > > >>> > > > > > > > > > SQL, > > > > >>> > > > > > > > > > > > so I will do the following. > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > 1.Each team has independent variables that > do > > > not > > > > >>> > affect > > > > >>> > > > each > > > > >>> > > > > > > other > > > > >>> > > > > > > > > and > > > > >>> > > > > > > > > > > > are isolated from each other, but the same > > > > >>> variable is > > > > >>> > > > unique > > > > >>> > > > > > > under > > > > >>> > > > > > > > > the > > > > >>> > > > > > > > > > > > same team, so you need to specify the team > > when > > > > >>> > creating > > > > >>> > > > > > > variables > > > > >>> > > > > > > > > > (users > > > > >>> > > > > > > > > > > > may belong to multiple teams, and the teams > > of > > > > the > > > > >>> > > current > > > > >>> > > > > user > > > > >>> > > > > > > > will > > > > >>> > > > > > > > > be > > > > >>> > > > > > > > > > > > displayed here for users to choose) > > > > >>> > > > > > > > > > > > 2.For Flink jar jobs, there are two ways to > > > pass > > > > >>> > > variables > > > > >>> > > > > > > > > > > > - By using placeholders, such as adding > > --kafka > > > > >>> > {mykafka} > > > > >>> > > > to > > > > >>> > > > > > the > > > > >>> > > > > > > > args > > > > >>> > > > > > > > > > of > > > > >>> > > > > > > > > > > > the application, 'mykafka' is the variable > > code > > > > of > > > > >>> a > > > > >>> > > > variable > > > > >>> > > > > > > > > > > > - Search by variable code or variable name. > > > > After a > > > > >>> > > > variable > > > > >>> > > > > is > > > > >>> > > > > > > > > > selected, > > > > >>> > > > > > > > > > > > the parameter transfer method is displayed > > > > >>> > > > > > > > > > > > 3.Flink sql will use variables as > > placeholders, > > > > >>> E.g: > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > CREATE TABLE KafkaTableA ( > > > > >>> > > > > > > > > > > > `user_id` BIGINT, > > > > >>> > > > > > > > > > > > `item_id` BIGINT, > > > > >>> > > > > > > > > > > > `type` STRING > > > > >>> > > > > > > > > > > > ) WITH ( > > > > >>> > > > > > > > > > > > 'connector' = 'kafka', > > > > >>> > > > > > > > > > > > 'topic' = 'user_behaviorA', > > > > >>> > > > > > > > > > > > > 'properties.bootstrap.servers' = > > > > >>> > > > > > > > > '{kafka_clusterA}', > > > > >>> > > > > > > > > > > -- > > > > >>> > > > > > > > > > > > 'kafka_ ClusterA' is the variable code > > > > >>> > > > > > > > > > > > 'properties.group.id' = > > > > 'testGroupA', > > > > >>> > > > > > > > > > > > 'scan.startup.mode' = > > > > >>> 'earliest-offset', > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > CREATE TABLE KafkaTableB ( > > > > >>> > > > > > > > > > > > `user_id` BIGINT, > > > > >>> > > > > > > > > > > > `item_id` BIGINT > > > > >>> > > > > > > > > > > > ) WITH ( > > > > >>> > > > > > > > > > > > 'connector' = 'kafka', > > > > >>> > > > > > > > > > > > 'topic' = 'user_behaviorB', > > > > >>> > > > > > > > > > > > > 'properties.bootstrap.servers' = > > > > >>> > > > > > > > > '{kafka_clusterB}', > > > > >>> > > > > > > > > > > -- > > > > >>> > > > > > > > > > > > 'kafka_ ClusterB' is the variable code > > > > >>> > > > > > > > > > > > 'properties.group.id' = > > > > 'testGroupB', > > > > >>> > > > > > > > > > > > 'scan.startup.mode' = > > > > >>> 'earliest-offset', > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id, > > > item_id, > > > > >>> > '{type}' > > > > >>> > > > > FROM > > > > >>> > > > > > > > > > > > KafkaTableB -- 'type' is the > > > variable > > > > >>> code > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > 4.In the future, we can consider setting > some > > > > >>> system > > > > >>> > > > built-in > > > > >>> > > > > > > > > variables > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > Best, > > > > >>> > > > > > > > > > > > ChunJin Mu > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan < > > > > >>> > > > [email protected] > > > > >>> > > > > > > > > > > >>> > > > > > > > > > wrote: > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > Hi 阿穆: > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > It's a great proposal. > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > For usage scenarios, I strongly > > recommend > > > > >>> support > > > > >>> > > for > > > > >>> > > > > > > > replacing > > > > >>> > > > > > > > > > > > variables > > > > >>> > > > > > > > > > > > > in sql. E.g: > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > CREATE TABLE KafkaTable ( > > > > >>> > > > > > > > > > > > > `user_id` BIGINT, > > > > >>> > > > > > > > > > > > > `item_id` BIGINT > > > > >>> > > > > > > > > > > > > ) WITH ( > > > > >>> > > > > > > > > > > > > 'connector' = > 'kafka', > > > > >>> > > > > > > > > > > > > 'topic' = > > > 'user_behavior', > > > > >>> > > > > > > > > > > > > > > > > >>> 'properties.bootstrap.servers' = > > > > >>> > > > > > > > > > > '{kafka_clusterA}', > > > > >>> > > > > > > > > > > > > ' > properties.group.id' > > = > > > > >>> > > > 'testGroup', > > > > >>> > > > > > > > > > > > > 'scan.startup.mode' > = > > > > >>> > > > > 'earliest-offset', > > > > >>> > > > > > > > > > > > > 'format' = 'csv' > > > > >>> > > > > > > > > > > > > ) > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > Note: 'properties.bootstrap.servers' = > > > > >>> > > > > '{kafka_clusterA}', > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > For variable management, I have some > > > > question: > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > 1. Is the parameter used within the > > team? > > > If > > > > >>> all > > > > >>> > > teams > > > > >>> > > > > > use a > > > > >>> > > > > > > > > > common > > > > >>> > > > > > > > > > > > > parameter, it > > > > >>> > > > > > > > > > > > > needs to be defined for each team, > > right? > > > > >>> > > > > > > > > > > > > 2. Which users can modify variables? > Can > > > all > > > > >>> users > > > > >>> > > > > modify > > > > >>> > > > > > > all > > > > >>> > > > > > > > > > > > variables of > > > > >>> > > > > > > > > > > > > their team? > > > > >>> > > > > > > > > > > > > After the variable is modified > > > incorrectly, > > > > >>> many > > > > >>> > > jobs > > > > >>> > > > > may > > > > >>> > > > > > be > > > > >>> > > > > > > > > > > > affected. So I > > > > >>> > > > > > > > > > > > > suggest that > > > > >>> > > > > > > > > > > > > only team admin or system admin have > > > > >>> permission to > > > > >>> > > > > modify, > > > > >>> > > > > > > add > > > > >>> > > > > > > > > and > > > > >>> > > > > > > > > > > > delete > > > > >>> > > > > > > > > > > > > variables. > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > Best > > > > >>> > > > > > > > > > > > > fanrui > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang > < > > > > >>> > > > > [email protected] > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > wrote: > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > hi 阿穆: > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > Thanks for starting a very useful > > > > discussion, > > > > >>> I > > > > >>> > > have a > > > > >>> > > > > > some > > > > >>> > > > > > > > > > > questions: > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > 1. The access boundary problem of > > variable > > > > >>> > > management, > > > > >>> > > > > is > > > > >>> > > > > > > this > > > > >>> > > > > > > > > all > > > > >>> > > > > > > > > > > > > people in the platform can access > these > > > > >>> variables( > > > > >>> > > no > > > > >>> > > > > > matter > > > > >>> > > > > > > > who > > > > >>> > > > > > > > > > > > > defines) or that only team members can > > be > > > > >>> access > > > > >>> > > these > > > > >>> > > > > > > > > variables. > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > 2. There are two usage scenarios for > > > > variables > > > > >>> > that > > > > >>> > > I > > > > >>> > > > > know > > > > >>> > > > > > > of, > > > > >>> > > > > > > > > one > > > > >>> > > > > > > > > > > is > > > > >>> > > > > > > > > > > > > in the entry parameter of main, and > the > > > > other > > > > >>> is > > > > >>> > in > > > > >>> > > > > > > flinksql. > > > > >>> > > > > > > > > Are > > > > >>> > > > > > > > > > > both > > > > >>> > > > > > > > > > > > > supported, or only supported main > entry > > ? > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > about first question, my opinion: > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > 1. The variable management is > associated > > > > with > > > > >>> the > > > > >>> > > > team. > > > > >>> > > > > > > After > > > > >>> > > > > > > > > the > > > > >>> > > > > > > > > > > user > > > > >>> > > > > > > > > > > > > logs in to the system, he must select > a > > > team > > > > >>> (a > > > > >>> > user > > > > >>> > > > can > > > > >>> > > > > > > > belong > > > > >>> > > > > > > > > to > > > > >>> > > > > > > > > > > one > > > > >>> > > > > > > > > > > > > or more teams), The variable belongs > to > > a > > > > team > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > 2. All resources between teams are > > > isolated, > > > > >>> which > > > > >>> > > > means > > > > >>> > > > > > > that > > > > >>> > > > > > > > > > > > > variables are also isolated. In short, > > > > >>> members of > > > > >>> > a > > > > >>> > > > team > > > > >>> > > > > > can > > > > >>> > > > > > > > use > > > > >>> > > > > > > > > > and > > > > >>> > > > > > > > > > > > > define these variables, which are > > > invisible > > > > to > > > > >>> > other > > > > >>> > > > > > > members, > > > > >>> > > > > > > > > and > > > > >>> > > > > > > > > > > > > other members cannot use these > > variables. > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > Welcome to your views > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > Best, > > > > >>> > > > > > > > > > > > > Huajie Wang > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > 陈卓宇 <[email protected]> > > > > >>> > 于2022年10月7日周五 > > > > >>> > > > > > > 14:18写道： > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > Is it possible to do this > variable > > > > >>> support in > > > > >>> > > sql > > > > >>> > > > > as > > > > >>> > > > > > > well > > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > 阿穆 <[email protected]> > > > > >>> > 于2022年10月7日周五 > > > > >>> > > > > > > 13:54写道： > > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > Hi guys, > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > I want to start a discussion > > > about > > > > >>> the > > > > >>> > > > > variable > > > > >>> > > > > > > > > > management > > > > >>> > > > > > > > > > > > module. Please > > > > >>> > > > > > > > > > > > > > > see this issue issue-1477. > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > Background: > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > In the actual production > > > > >>> environment, > > > > >>> > > Flink > > > > >>> > > > > jobs > > > > >>> > > > > > > are > > > > >>> > > > > > > > > > > > generally complex, > > > > >>> > > > > > > > > > > > > > > and usually require multiple > > > > >>> external > > > > >>> > > > > > components. > > > > >>> > > > > > > > For > > > > >>> > > > > > > > > > > > example, Flink jobs > > > > >>> > > > > > > > > > > > > > > consume data from Kafka, > then > > > > >>> connect > > > > >>> > > > external > > > > >>> > > > > > > > > > components > > > > >>> > > > > > > > > > > > such as HBase > > > > >>> > > > > > > > > > > > > > or > > > > >>> > > > > > > > > > > > > > > Redis to obtain additional > > > > business > > > > >>> > > > > information, > > > > >>> > > > > > > and > > > > >>> > > > > > > > > > then > > > > >>> > > > > > > > > > > > write it to the > > > > >>> > > > > > > > > > > > > > > downstream external > > components. > > > > >>> There > > > > >>> > are > > > > >>> > > > the > > > > >>> > > > > > > > > following > > > > >>> > > > > > > > > > > > problems: > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > 1. The connection > information > > of > > > > >>> > external > > > > >>> > > > > > > > components, > > > > >>> > > > > > > > > > such > > > > >>> > > > > > > > > > > > as IP, port > > > > >>> > > > > > > > > > > > > > and > > > > >>> > > > > > > > > > > > > > > user password, needs to be > > > > >>> configured in > > > > >>> > > the > > > > >>> > > > > > > > > application > > > > >>> > > > > > > > > > > > args and > > > > >>> > > > > > > > > > > > > > > transferred to the Flink > job, > > so > > > > >>> that > > > > >>> > the > > > > >>> > > > > > > connection > > > > >>> > > > > > > > > > > > information of > > > > >>> > > > > > > > > > > > > > > external components is > > > distributed > > > > >>> in > > > > >>> > > > multiple > > > > >>> > > > > > > > > > > > applications. Once the > > > > >>> > > > > > > > > > > > > > > connection information of > > > external > > > > >>> > > > components > > > > >>> > > > > > > > changes, > > > > >>> > > > > > > > > > > many > > > > >>> > > > > > > > > > > > application > > > > >>> > > > > > > > > > > > > > > args parameters need to be > > > > modified, > > > > >>> > which > > > > >>> > > > > will > > > > >>> > > > > > > lead > > > > >>> > > > > > > > > to > > > > >>> > > > > > > > > > > > high operation > > > > >>> > > > > > > > > > > > > > and > > > > >>> > > > > > > > > > > > > > > maintenance costs. > > > > >>> > > > > > > > > > > > > > > 2. There are many people in > > the > > > > team > > > > >>> > > > > developing > > > > >>> > > > > > > > > > real-time > > > > >>> > > > > > > > > > > > computing jobs. > > > > >>> > > > > > > > > > > > > > > There is no uniform > > > specification > > > > >>> for > > > > >>> > the > > > > >>> > > > > > > connection > > > > >>> > > > > > > > > > > > information of the > > > > >>> > > > > > > > > > > > > > > external components passed > to > > > the > > > > >>> job, > > > > >>> > > > > resulting > > > > >>> > > > > > > in > > > > >>> > > > > > > > > > > > different parameter > > > > >>> > > > > > > > > > > > > > > names of the same component. > > > This > > > > is > > > > >>> > > > difficult > > > > >>> > > > > > to > > > > >>> > > > > > > > > count > > > > >>> > > > > > > > > > > > which external > > > > >>> > > > > > > > > > > > > > > components are dependent. > > > > >>> > > > > > > > > > > > > > > 3. In production practice, > > there > > > > are > > > > >>> > > usually > > > > >>> > > > > > > > multiple > > > > >>> > > > > > > > > > sets > > > > >>> > > > > > > > > > > > of > > > > >>> > > > > > > > > > > > > > > environments, such as test > > > > >>> environment > > > > >>> > and > > > > >>> > > > > > > > production > > > > >>> > > > > > > > > > > > environment. It is > > > > >>> > > > > > > > > > > > > > > not intuitive to judge > > whether a > > > > >>> > component > > > > >>> > > > > > belongs > > > > >>> > > > > > > > to > > > > >>> > > > > > > > > a > > > > >>> > > > > > > > > > > > test environment > > > > >>> > > > > > > > > > > > > > or > > > > >>> > > > > > > > > > > > > > > a production environment > > simply > > > > >>> through > > > > >>> > IP > > > > >>> > > > and > > > > >>> > > > > > > > ports. > > > > >>> > > > > > > > > > Even > > > > >>> > > > > > > > > > > > if it can be > > > > >>> > > > > > > > > > > > > > > judged, there are some > > > omissions. > > > > In > > > > >>> > this > > > > >>> > > > way, > > > > >>> > > > > > the > > > > >>> > > > > > > > > > > > connection information > > > > >>> > > > > > > > > > > > > > > online to the production > > > > >>> environment may > > > > >>> > > be > > > > >>> > > > > > > external > > > > >>> > > > > > > > > > > > components of the > > > > >>> > > > > > > > > > > > > > test > > > > >>> > > > > > > > > > > > > > > environment, or components > of > > > the > > > > >>> > > production > > > > >>> > > > > > > > > environment > > > > >>> > > > > > > > > > > are > > > > >>> > > > > > > > > > > > > > inadvertently > > > > >>> > > > > > > > > > > > > > > configured during testing, > > > leading > > > > >>> to > > > > >>> > > > > production > > > > >>> > > > > > > > > > failures. > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > Proposal: > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > 1. Create a unified module > to > > > > manage > > > > >>> > > > > variables. > > > > >>> > > > > > > > Users > > > > >>> > > > > > > > > > can > > > > >>> > > > > > > > > > > > define their > > > > >>> > > > > > > > > > > > > > own > > > > >>> > > > > > > > > > > > > > > variables in this module. > > > Variable > > > > >>> > > > information > > > > >>> > > > > > > > > includes > > > > >>> > > > > > > > > > > > variable code > > > > >>> > > > > > > > > > > > > > > (passed to Flink program as > > > > >>> parameter > > > > >>> > > name), > > > > >>> > > > > > > > variable > > > > >>> > > > > > > > > > > > value, variable > > > > >>> > > > > > > > > > > > > name, > > > > >>> > > > > > > > > > > > > > > variable description, and > > which > > > > >>> > > applications > > > > >>> > > > > > > depend > > > > >>> > > > > > > > on > > > > >>> > > > > > > > > > > > > > > 2. In the same Team module, > > the > > > > >>> variable > > > > >>> > > > code > > > > >>> > > > > or > > > > >>> > > > > > > > > > variable > > > > >>> > > > > > > > > > > > name is unique > > > > >>> > > > > > > > > > > > > > > - By using placeholders, > such > > as > > > > >>> adding > > > > >>> > -- > > > > >>> > > > > kafka > > > > >>> > > > > > > > > > {mykafka} > > > > >>> > > > > > > > > > > > to the args of > > > > >>> > > > > > > > > > > > > > > the application, mykafka is > > the > > > > >>> variable > > > > >>> > > > code > > > > >>> > > > > > of a > > > > >>> > > > > > > > > > > variable > > > > >>> > > > > > > > > > > > > > > - Search by variable code or > > > > >>> variable > > > > >>> > > name. > > > > >>> > > > > > After > > > > >>> > > > > > > a > > > > >>> > > > > > > > > > > > variable is selected, > > > > >>> > > > > > > > > > > > > > > the parameter transfer > method > > is > > > > >>> > displayed > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > If you have any suggestions, > > > > >>> welcome to > > > > >>> > > > > > > participate > > > > >>> > > > > > > > in > > > > >>> > > > > > > > > > the > > > > >>> > > > > > > > > > > > discussion on > > > > >>> > > > > > > > > > > > > > > the mail list or issue, I > look > > > > >>> forward > > > > >>> > to > > > > >>> > > > your > > > > >>> > > > > > > > > feedback. > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > Best wishes > > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >> > > > > > > > > > >
