hi guys.
When referencing variables in program args or Flink sql, in order to
enhance the user experience, follow-up search occurs when the user enters
"${", the user can support both variable code and description fuzzy search,
for example, if there is a variable code of a variable is
"internal.kafka.brokers", description is "内部kafka主机列表", when the user
enters "${kaf" and "${内部", this variable can be searched.
Do you think this design is feasible?
Best,
Chunjin Mu
Chunjin Mu <[email protected]> 于2022年10月15日周六 20:50写道：
> Hi fanru,
>
> Thank you very much for your hard review.
> You have considered it very comprehensively, I also saw your question in
> the PR, and I have already responded, please check this reply [1]
>
> [1]
> https://github.com/apache/incubator-streampark/pull/1831#discussion_r996293537
>
> Best,
> Chunjin Mu
>
> Rui Fan <[email protected]> 于2022年10月15日周六 18:54写道：
>
>> Hi chunjin,
>>
>> Very sorry for slowing down your progress, I found some
>> small problems when I reviewed your code.
>>
>> Who can create the variable and update the variable?
>> System Admin and team_admin? Or every developer?
>>
>> As I understand, we didn't discuss this point, right? If I
>> miss any information please let me know, thanks a lot.
>>
>> Best,
>> fanrui
>>
>> On Sat, Oct 15, 2022 at 4:44 PM Chunjin Mu <[email protected]> wrote:
>>
>> > Hi fanru,
>> >
>> > > I have a little question. If I copy some sql from a old app, and the
>> sql
>> > includes some placeholders. These placeholders will take effect in the
>> > new app, right?
>> >
>> > yes, the copied placeholder will take effect if the variable exists.
>> >
>> > I have added all the information to proposal issue-1477[1]，include the
>> > questions you asked above， if there is more discussion later, I will add
>> > this to the proposal.
>> >
>> > [1] https://github.com/apache/incubator-streampark/issues/1477
>> >
>> >
>> > Rui Fan <[email protected]> 于2022年10月15日周六 13:17写道：
>> >
>> > > Hi chunjin,
>> > >
>> > > Thanks for your detailed information.
>> > >
>> > > > When the focus is lost, it is judged whether there is a variable
>> > > placeholder, and if so, a verification button is displayed to prompt
>> the
>> > > user to perform verification. When the user clicks the verification
>> > button,
>> > > the replaced content of the placeholder is displayed, and only the
>> user
>> > > clicks the verification button to click the submit button
>> > >
>> > > I have a little question. If I copy some sql from a old app, and the
>> sql
>> > > includes some placeholders. These placeholders will take effect in the
>> > > new app, right?
>> > >
>> > > BTW, could you add all information to your proposal issue-1477[1]?
>> > > The mail list is for discussion only, and we should maintain the final
>> > > solution in the Proposal.
>> > >
>> > >
>> > > [1] https://github.com/apache/incubator-streampark/issues/1477
>> > >
>> > > Best
>> > > fanrui
>> > >
>> > > On Sat, Oct 15, 2022 at 11:56 AM Chunjin Mu <[email protected]>
>> > wrote:
>> > >
>> > > > Hi guys,
>> > > >
>> > > > Thanks for the discussion and great suggestions.
>> > > >
>> > > > Here I summarize the functions to be completed in version 1.2.4
>> > > >
>> > > > 1.The table structure of the variable contains the following main
>> > > contents
>> > > > - variable code is placeholder, For example the variable code is
>> > > > "collect.kafka.cluster", it will be referenced as
>> > > ${collect.kafka.cluster},
>> > > > it is unique within a team
>> > > > - variable name is removed, after discussion, it is not necessarily
>> > > > retained
>> > > > - variable value is the value of a specific variable, such as ip and
>> > port
>> > > > - description is a detailed introduction
>> > > > - team is the team the variable belongs to
>> > > >
>> > > > 2.Basic functions of variable management
>> > > > - Add and modify variables: need to check whether the variable code
>> is
>> > > > unique under the current team
>> > > > - Deleting a variable: It is necessary to check whether it has been
>> > > > referenced by the Application under the team to which the variable
>> > > belongs.
>> > > > If there is metadata management in the future, it is also necessary
>> to
>> > > > check whether it has been referenced by the metadata. If it is
>> > > referenced,
>> > > > it will prompt that the variable is referenced and cannot be
>> deleted.
>> > > >
>> > > > 3.References in Flink sql and program args
>> > > > - In the application, the variables are uniformly referenced in the
>> > form
>> > > of
>> > > > placeholders ${xxx}, such as in Flink sql, program args
>> > > > - For a better experience, it is necessary to provide the function
>> of
>> > > > following the search prompt when entering variable placeholders. For
>> > > > example, when entering "${", the variable list is displayed, and
>> when
>> > > > "${kafka" is entered, the display contains variables for kafka
>> strings,
>> > > in
>> > > > "variable code" or "description".
>> > > > - When the focus is lost, it is judged whether there is a variable
>> > > > placeholder, and if so, a verification button is displayed to prompt
>> > the
>> > > > user to perform verification. When the user clicks the verification
>> > > button,
>> > > > the replaced content of the placeholder is displayed, and only the
>> user
>> > > > clicks the verification button to click the submit button
>> > > > - If the user enters "okir23${j2Rts}pEt", this placeholder
>> "${j2Rts}"
>> > > will
>> > > > not be replaced if "j2Rts" is not a variable we defined
>> > > >
>> > > > 4.Future plan
>> > > > - Display the application or metadata list of a referenced variable
>> in
>> > > the
>> > > > variable basic management module
>> > > > - Improve user experience
>> > > >
>> > > > Best,
>> > > > Chunjin Mu
>> > > >
>> > > > Chunjin Mu <[email protected]> 于2022年10月14日周五 22:28写道：
>> > > >
>> > > > > When we have dozens or dozens of variables, this phenomenon may be
>> > more
>> > > > > prominent
>> > > > >
>> > > > >
>> > > > > Best,
>> > > > > Chunjin Mu
>> > > > >
>> > > > > Chunjin Mu <[email protected]> 于2022年10月14日周五 22:26写道：
>> > > > >
>> > > > >> When a person asks which kafka should be used, I tell him in
>> Chinese
>> > > > that
>> > > > >> he should use "采集整合的kafka", in fact the variable code is
>> > > collect.kafka,
>> > > > but
>> > > > >> he is searching for gather keyword, it is difficult to just pass
>> > > > >> collect.kafka The intuitive understanding is to "采集整合的kafka",
>> > because
>> > > > >> the information contained in the variable code is still not that
>> > much.
>> > > > >>
>> > > > >> Huajie Wang <[email protected]> 于2022年10月14日周五 22:17写道：
>> > > > >>
>> > > > >>> I just thought about it, there is really no need for this name.
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> Best,
>> > > > >>> Huajie Wang
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> Rui Fan <[email protected]> 于2022年10月14日周五 21:54写道：
>> > > > >>>
>> > > > >>> > Hi guys,
>> > > > >>> >
>> > > > >>> > I understand chunjin's solution is this:
>> > > > >>> >
>> > > > >>> > - variable code is placeholder
>> > > > >>> > - variable name is an introduction
>> > > > >>> > - Description is a detailed introduction
>> > > > >>> >
>> > > > >>> > I think introduction and detail are redundant.
>> > > > >>> >
>> > > > >>> > For example: there are two kafka clusters, the Beijing kafka
>> > > > >>> > cluster and the Hangzhou kafka cluster.
>> > > > >>> >
>> > > > >>> > If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why
>> > > > >>> > does the introduction still need `北京 kafka 集群` and
>> > > > >>> > `杭州 kafka 集群`? placeholder is already clear.
>> > > > >>> >
>> > > > >>> > If dev may forget the placeholder, then dev may also
>> > > > >>> > forget the introduction.
>> > > > >>> >
>> > > > >>> > Of course, if the placeholder is kafka1 kafka2, an
>> > > > >>> > introduction is required. Why do we need an introduction
>> > > > >>> > at this point, because the placeholder name is unreasonable.
>> > > > >>> >
>> > > > >>> > Just like variable naming in Java code, there is no need to
>> > > > >>> > introduce an introduction with a clear name. If the meaning of
>> > > > >>> > the variable is complicated, we also need to write a detailed
>> > > > >>> > description.
>> > > > >>> >
>> > > > >>> > Finally, I want to say: less is more. If we want to make
>> > > > >>> > something complicated, please give everyone a strong reason.
>> > > > >>> >
>> > > > >>> > Best
>> > > > >>> > fanrui
>> > > > >>> >
>> > > > >>> > On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <
>> [email protected]>
>> > > > wrote:
>> > > > >>> >
>> > > > >>> > > Hi guys,
>> > > > >>> > >
>> > > > >>> > > I agree with Chunjin, I think variable name is useful,
>> Chunjin
>> > > also
>> > > > >>> > > explained the reason.
>> > > > >>> > >
>> > > > >>> > >
>> > > > >>> > > Best,
>> > > > >>> > > Huajie Wang
>> > > > >>> > >
>> > > > >>> > >
>> > > > >>> > >
>> > > > >>> > > Chunjin Mu <[email protected]> 于2022年10月14日周五 20:22写道：
>> > > > >>> > >
>> > > > >>> > > > Hi, Rui Fan
>> > > > >>> > > > I'll start a discussion later.
>> > > > >>> > > >
>> > > > >>> > > > Best
>> > > > >>> > > > Chunjin Mu
>> > > > >>> > > >
>> > > > >>> > > > Rui Fan <[email protected]> 于2022年10月14日周五 19:42写道：
>> > > > >>> > > >
>> > > > >>> > > > > Hi guys,
>> > > > >>> > > > >
>> > > > >>> > > > > I agree with monster. And I suggest use variable name,
>> > > variable
>> > > > >>> value
>> > > > >>> > > and
>> > > > >>> > > > > description.
>> > > > >>> > > > >
>> > > > >>> > > > > By the way, there are some problems still need to
>> confirm.
>> > > > >>> What’s the
>> > > > >>> > > > > placeholder? And how to use them when edit sql or
>> > datastream
>> > > > job?
>> > > > >>> > It’s
>> > > > >>> > > > > better to discuss these asap.
>> > > > >>> > > > >
>> > > > >>> > > > > Best
>> > > > >>> > > > > fanrui
>> > > > >>> > > > >
>> > > > >>> > > > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <
>> > > [email protected]
>> > > > >
>> > > > >>> > wrote:
>> > > > >>> > > > >
>> > > > >>> > > > > > hi, huajie, fan rui, monster
>> > > > >>> > > > > >
>> > > > >>> > > > > > Thank you for your suggestions.
>> > > > >>> > > > > >
>> > > > >>> > > > > > Now please Huajie to confirm how to do it. If remove
>> the
>> > > > >>> variable
>> > > > >>> > > > name, I
>> > > > >>> > > > > > can also accept it here. I will do small secondary
>> > > > development
>> > > > >>> on
>> > > > >>> > our
>> > > > >>> > > > > > production system based on the community to meet
>> internal
>> > > > >>> needs.
>> > > > >>> > > > > >
>> > > > >>> > > > > > 陈卓宇 <[email protected]> 于2022年10月14日周五
>> > 15:58写道：
>> > > > >>> > > > > >
>> > > > >>> > > > > > > "外部采集整合kafka" is the Chinese description of
>> variable,
>> > so
>> > > it
>> > > > >>> makes
>> > > > >>> > > > sense
>> > > > >>> > > > > > to
>> > > > >>> > > > > > > put it under `Description`.
>> > > > >>> > > > > > >
>> > > > >>> > > > > > > Chunjin Mu <[email protected]> 于2022年10月14日周五
>> > > 12:01写道：
>> > > > >>> > > > > > >
>> > > > >>> > > > > > > > The variable management module has three
>> properties,
>> > > > >>> including
>> > > > >>> > > > > variable
>> > > > >>> > > > > > > > code, variable name, and description, For example,
>> > the
>> > > > >>> variable
>> > > > >>> > > > code
>> > > > >>> > > > > is
>> > > > >>> > > > > > > > "external.kafka.collect.brokers", and the variable
>> > name
>> > > > is
>> > > > >>> > > > > > "外部采集整合kafka".
>> > > > >>> > > > > > > > When we created the application, I forgot what the
>> > > > variable
>> > > > >>> > code
>> > > > >>> > > > was,
>> > > > >>> > > > > > > but I
>> > > > >>> > > > > > > > knew it was called "采集整合", so I can search Enter
>> the
>> > > > "采集整合"
>> > > > >>> > > > keywords
>> > > > >>> > > > > in
>> > > > >>> > > > > > > the
>> > > > >>> > > > > > > > box, and of course you can also search for kafka
>> > > > keywords.
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > At the same time, I want to know more detailed
>> > > > information
>> > > > >>> > about
>> > > > >>> > > > this
>> > > > >>> > > > > > > > kafka, such as who the operator is, what is the
>> data
>> > > > >>> retention
>> > > > >>> > > > > period,
>> > > > >>> > > > > > > and
>> > > > >>> > > > > > > > how big is the cluster size, but I can't use the
>> > > > >>> description to
>> > > > >>> > > > > search,
>> > > > >>> > > > > > > so
>> > > > >>> > > > > > > > I can only search or variable code or variable
>> name,
>> > > and
>> > > > >>> the
>> > > > >>> > > > variable
>> > > > >>> > > > > > > name
>> > > > >>> > > > > > > > is also for a simple display.
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > Although external.kafka.collect.brokers already
>> > > contains
>> > > > a
>> > > > >>> lot
>> > > > >>> > of
>> > > > >>> > > > > > > > information, I think this is just a naming
>> > convention.
>> > > > >>> When we
>> > > > >>> > > > > discuss
>> > > > >>> > > > > > > this
>> > > > >>> > > > > > > > variable, we will discuss it by using the "采集整合
>> > kafka"
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > We can discuss whether there is a problem with
>> this
>> > > > design,
>> > > > >>> > what
>> > > > >>> > > do
>> > > > >>> > > > > you
>> > > > >>> > > > > > > > think?
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > Rui Fan <[email protected]> 于2022年10月10日周一
>> > 11:45写道：
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > > Hi huajie:
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > > > > If it is manually entered, it will always be
>> > > treated
>> > > > as
>> > > > >>> > > > ordinary
>> > > > >>> > > > > > > text.
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > > > Users always copy some code from an old job to a
>> > new
>> > > > job.
>> > > > >>> > > > > > > > > I'm afraid it might not meet the user's
>> > expectations.
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > > > Best,
>> > > > >>> > > > > > > > > fanrui
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <
>> > > > >>> > [email protected]>
>> > > > >>> > > > > > wrote:
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > > > > hi 田向阳:
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > > The current discussion is about variable
>> > > management,
>> > > > >>> which
>> > > > >>> > > is a
>> > > > >>> > > > > > > > > relatively
>> > > > >>> > > > > > > > > > independent function, No other modules
>> involved,
>> > > you
>> > > > >>> can
>> > > > >>> > > check
>> > > > >>> > > > > the
>> > > > >>> > > > > > > > thread
>> > > > >>> > > > > > > > > > [1]
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > > [1] thread:
>> > > > >>> > > > > > > > > >
>> > > > >>> > > >
>> > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > > Best,
>> > > > >>> > > > > > > > > > Huajie Wang
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > > 田向阳 <[email protected]> 于2022年10月8日周六
>> 13:13写道：
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > > > is this module include a resource management
>> > > cente
>> > > > r
>> > > > >>> > where
>> > > > >>> > > > jars
>> > > > >>> > > > > > can
>> > > > >>> > > > > > > > be
>> > > > >>> > > > > > > > > > > chosen to provide to create a job
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > ---- 回复的原邮件 ----
>> > > > >>> > > > > > > > > > > | 发件人 | Huajie Wang<[email protected]> |
>> > > > >>> > > > > > > > > > > | 日期 | 2022年10月08日 12:07 |
>> > > > >>> > > > > > > > > > > | 收件人 | [email protected]<
>> > > > >>> > > [email protected]>
>> > > > >>> > > > |
>> > > > >>> > > > > > > > > > > | 抄送至 | |
>> > > > >>> > > > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add
>> variable
>> > > > >>> management
>> > > > >>> > > > > module |
>> > > > >>> > > > > > > > > > > hi 阿穆:
>> > > > >>> > > > > > > > > > > Thank you for the clarification, My
>> question is
>> > > > >>> answered,
>> > > > >>> > > but
>> > > > >>> > > > > > now I
>> > > > >>> > > > > > > > > have
>> > > > >>> > > > > > > > > > > one more concern,
>> > > > >>> > > > > > > > > > > I'm currently using {var} for variables, and
>> > I'm
>> > > > >>> worried
>> > > > >>> > > that
>> > > > >>> > > > > one
>> > > > >>> > > > > > > of
>> > > > >>> > > > > > > > > the
>> > > > >>> > > > > > > > > > > user's parameters includes "{" and "}", e.g
>> :
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > --password 'mypass{123}abc'
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > So how do we parse parse this? How about
>> > ${var}
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > Best,
>> > > > >>> > > > > > > > > > > Huajie Wang
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > 阿穆 <[email protected]> 于2022年10月8日周六
>> > > > 11:36写道：
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > Thanks for your reply.
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > I have integrated your replies. I really
>> > don't
>> > > > pay
>> > > > >>> much
>> > > > >>> > > > > > attention
>> > > > >>> > > > > > > > to
>> > > > >>> > > > > > > > > > SQL,
>> > > > >>> > > > > > > > > > > > so I will do the following.
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > 1.Each team has independent variables
>> that do
>> > > not
>> > > > >>> > affect
>> > > > >>> > > > each
>> > > > >>> > > > > > > other
>> > > > >>> > > > > > > > > and
>> > > > >>> > > > > > > > > > > > are isolated from each other, but the same
>> > > > >>> variable is
>> > > > >>> > > > unique
>> > > > >>> > > > > > > under
>> > > > >>> > > > > > > > > the
>> > > > >>> > > > > > > > > > > > same team, so you need to specify the team
>> > when
>> > > > >>> > creating
>> > > > >>> > > > > > > variables
>> > > > >>> > > > > > > > > > (users
>> > > > >>> > > > > > > > > > > > may belong to multiple teams, and the
>> teams
>> > of
>> > > > the
>> > > > >>> > > current
>> > > > >>> > > > > user
>> > > > >>> > > > > > > > will
>> > > > >>> > > > > > > > > be
>> > > > >>> > > > > > > > > > > > displayed here for users to choose)
>> > > > >>> > > > > > > > > > > > 2.For Flink jar jobs, there are two ways
>> to
>> > > pass
>> > > > >>> > > variables
>> > > > >>> > > > > > > > > > > > - By using placeholders, such as adding
>> > --kafka
>> > > > >>> > {mykafka}
>> > > > >>> > > > to
>> > > > >>> > > > > > the
>> > > > >>> > > > > > > > args
>> > > > >>> > > > > > > > > > of
>> > > > >>> > > > > > > > > > > > the application, 'mykafka' is the variable
>> > code
>> > > > of
>> > > > >>> a
>> > > > >>> > > > variable
>> > > > >>> > > > > > > > > > > > - Search by variable code or variable
>> name.
>> > > > After a
>> > > > >>> > > > variable
>> > > > >>> > > > > is
>> > > > >>> > > > > > > > > > selected,
>> > > > >>> > > > > > > > > > > > the parameter transfer method is displayed
>> > > > >>> > > > > > > > > > > > 3.Flink sql will use variables as
>> > placeholders,
>> > > > >>> E.g:
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > CREATE TABLE KafkaTableA (
>> > > > >>> > > > > > > > > > > > `user_id` BIGINT,
>> > > > >>> > > > > > > > > > > > `item_id` BIGINT,
>> > > > >>> > > > > > > > > > > > `type` STRING
>> > > > >>> > > > > > > > > > > > ) WITH (
>> > > > >>> > > > > > > > > > > > 'connector' = 'kafka',
>> > > > >>> > > > > > > > > > > > 'topic' = 'user_behaviorA',
>> > > > >>> > > > > > > > > > > >
>> 'properties.bootstrap.servers' =
>> > > > >>> > > > > > > > > '{kafka_clusterA}',
>> > > > >>> > > > > > > > > > > --
>> > > > >>> > > > > > > > > > > > 'kafka_ ClusterA' is the variable code
>> > > > >>> > > > > > > > > > > > 'properties.group.id' =
>> > > > 'testGroupA',
>> > > > >>> > > > > > > > > > > > 'scan.startup.mode' =
>> > > > >>> 'earliest-offset',
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > CREATE TABLE KafkaTableB (
>> > > > >>> > > > > > > > > > > > `user_id` BIGINT,
>> > > > >>> > > > > > > > > > > > `item_id` BIGINT
>> > > > >>> > > > > > > > > > > > ) WITH (
>> > > > >>> > > > > > > > > > > > 'connector' = 'kafka',
>> > > > >>> > > > > > > > > > > > 'topic' = 'user_behaviorB',
>> > > > >>> > > > > > > > > > > >
>> 'properties.bootstrap.servers' =
>> > > > >>> > > > > > > > > '{kafka_clusterB}',
>> > > > >>> > > > > > > > > > > --
>> > > > >>> > > > > > > > > > > > 'kafka_ ClusterB' is the variable code
>> > > > >>> > > > > > > > > > > > 'properties.group.id' =
>> > > > 'testGroupB',
>> > > > >>> > > > > > > > > > > > 'scan.startup.mode' =
>> > > > >>> 'earliest-offset',
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id,
>> > > item_id,
>> > > > >>> > '{type}'
>> > > > >>> > > > > FROM
>> > > > >>> > > > > > > > > > > > KafkaTableB -- 'type' is the
>> > > variable
>> > > > >>> code
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > 4.In the future, we can consider setting
>> some
>> > > > >>> system
>> > > > >>> > > > built-in
>> > > > >>> > > > > > > > > variables
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > Best,
>> > > > >>> > > > > > > > > > > > ChunJin Mu
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
>> > > > >>> > > > [email protected]
>> > > > >>> > > > > > >
>> > > > >>> > > > > > > > > > wrote:
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Hi 阿穆:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > It's a great proposal.
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > For usage scenarios, I strongly
>> > recommend
>> > > > >>> support
>> > > > >>> > > for
>> > > > >>> > > > > > > > replacing
>> > > > >>> > > > > > > > > > > > variables
>> > > > >>> > > > > > > > > > > > > in sql. E.g:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > CREATE TABLE KafkaTable (
>> > > > >>> > > > > > > > > > > > > `user_id` BIGINT,
>> > > > >>> > > > > > > > > > > > > `item_id` BIGINT
>> > > > >>> > > > > > > > > > > > > ) WITH (
>> > > > >>> > > > > > > > > > > > > 'connector' =
>> 'kafka',
>> > > > >>> > > > > > > > > > > > > 'topic' =
>> > > 'user_behavior',
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> 'properties.bootstrap.servers' =
>> > > > >>> > > > > > > > > > > '{kafka_clusterA}',
>> > > > >>> > > > > > > > > > > > > '
>> properties.group.id'
>> > =
>> > > > >>> > > > 'testGroup',
>> > > > >>> > > > > > > > > > > > >
>> 'scan.startup.mode' =
>> > > > >>> > > > > 'earliest-offset',
>> > > > >>> > > > > > > > > > > > > 'format' = 'csv'
>> > > > >>> > > > > > > > > > > > > )
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Note: 'properties.bootstrap.servers'
>> =
>> > > > >>> > > > > '{kafka_clusterA}',
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > For variable management, I have some
>> > > > question:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 1. Is the parameter used within the
>> > team?
>> > > If
>> > > > >>> all
>> > > > >>> > > teams
>> > > > >>> > > > > > use a
>> > > > >>> > > > > > > > > > common
>> > > > >>> > > > > > > > > > > > > parameter, it
>> > > > >>> > > > > > > > > > > > > needs to be defined for each team,
>> > right?
>> > > > >>> > > > > > > > > > > > > 2. Which users can modify variables?
>> Can
>> > > all
>> > > > >>> users
>> > > > >>> > > > > modify
>> > > > >>> > > > > > > all
>> > > > >>> > > > > > > > > > > > variables of
>> > > > >>> > > > > > > > > > > > > their team?
>> > > > >>> > > > > > > > > > > > > After the variable is modified
>> > > incorrectly,
>> > > > >>> many
>> > > > >>> > > jobs
>> > > > >>> > > > > may
>> > > > >>> > > > > > be
>> > > > >>> > > > > > > > > > > > affected. So I
>> > > > >>> > > > > > > > > > > > > suggest that
>> > > > >>> > > > > > > > > > > > > only team admin or system admin have
>> > > > >>> permission to
>> > > > >>> > > > > modify,
>> > > > >>> > > > > > > add
>> > > > >>> > > > > > > > > and
>> > > > >>> > > > > > > > > > > > delete
>> > > > >>> > > > > > > > > > > > > variables.
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Best
>> > > > >>> > > > > > > > > > > > > fanrui
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie
>> Wang <
>> > > > >>> > > > > [email protected]
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > > > wrote:
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > hi 阿穆:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Thanks for starting a very useful
>> > > > discussion,
>> > > > >>> I
>> > > > >>> > > have a
>> > > > >>> > > > > > some
>> > > > >>> > > > > > > > > > > questions:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 1. The access boundary problem of
>> > variable
>> > > > >>> > > management,
>> > > > >>> > > > > is
>> > > > >>> > > > > > > this
>> > > > >>> > > > > > > > > all
>> > > > >>> > > > > > > > > > > > > people in the platform can access
>> these
>> > > > >>> variables(
>> > > > >>> > > no
>> > > > >>> > > > > > matter
>> > > > >>> > > > > > > > who
>> > > > >>> > > > > > > > > > > > > defines) or that only team members
>> can
>> > be
>> > > > >>> access
>> > > > >>> > > these
>> > > > >>> > > > > > > > > variables.
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 2. There are two usage scenarios for
>> > > > variables
>> > > > >>> > that
>> > > > >>> > > I
>> > > > >>> > > > > know
>> > > > >>> > > > > > > of,
>> > > > >>> > > > > > > > > one
>> > > > >>> > > > > > > > > > > is
>> > > > >>> > > > > > > > > > > > > in the entry parameter of main, and
>> the
>> > > > other
>> > > > >>> is
>> > > > >>> > in
>> > > > >>> > > > > > > flinksql.
>> > > > >>> > > > > > > > > Are
>> > > > >>> > > > > > > > > > > both
>> > > > >>> > > > > > > > > > > > > supported, or only supported main
>> entry
>> > ?
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > about first question, my opinion:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 1. The variable management is
>> associated
>> > > > with
>> > > > >>> the
>> > > > >>> > > > team.
>> > > > >>> > > > > > > After
>> > > > >>> > > > > > > > > the
>> > > > >>> > > > > > > > > > > user
>> > > > >>> > > > > > > > > > > > > logs in to the system, he must
>> select a
>> > > team
>> > > > >>> (a
>> > > > >>> > user
>> > > > >>> > > > can
>> > > > >>> > > > > > > > belong
>> > > > >>> > > > > > > > > to
>> > > > >>> > > > > > > > > > > one
>> > > > >>> > > > > > > > > > > > > or more teams), The variable belongs
>> to
>> > a
>> > > > team
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 2. All resources between teams are
>> > > isolated,
>> > > > >>> which
>> > > > >>> > > > means
>> > > > >>> > > > > > > that
>> > > > >>> > > > > > > > > > > > > variables are also isolated. In
>> short,
>> > > > >>> members of
>> > > > >>> > a
>> > > > >>> > > > team
>> > > > >>> > > > > > can
>> > > > >>> > > > > > > > use
>> > > > >>> > > > > > > > > > and
>> > > > >>> > > > > > > > > > > > > define these variables, which are
>> > > invisible
>> > > > to
>> > > > >>> > other
>> > > > >>> > > > > > > members,
>> > > > >>> > > > > > > > > and
>> > > > >>> > > > > > > > > > > > > other members cannot use these
>> > variables.
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Welcome to your views
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Best,
>> > > > >>> > > > > > > > > > > > > Huajie Wang
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 陈卓宇 <[email protected]
>> >
>> > > > >>> > 于2022年10月7日周五
>> > > > >>> > > > > > > 14:18写道：
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > Is it possible to do this
>> variable
>> > > > >>> support in
>> > > > >>> > > sql
>> > > > >>> > > > > as
>> > > > >>> > > > > > > well
>> > > > >>> > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > 阿穆 <[email protected]
>> >
>> > > > >>> > 于2022年10月7日周五
>> > > > >>> > > > > > > 13:54写道：
>> > > > >>> > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > Hi guys,
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > I want to start a
>> discussion
>> > > about
>> > > > >>> the
>> > > > >>> > > > > variable
>> > > > >>> > > > > > > > > > management
>> > > > >>> > > > > > > > > > > > module. Please
>> > > > >>> > > > > > > > > > > > > > > see this issue issue-1477.
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > Background:
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > In the actual production
>> > > > >>> environment,
>> > > > >>> > > Flink
>> > > > >>> > > > > jobs
>> > > > >>> > > > > > > are
>> > > > >>> > > > > > > > > > > > generally complex,
>> > > > >>> > > > > > > > > > > > > > > and usually require
>> multiple
>> > > > >>> external
>> > > > >>> > > > > > components.
>> > > > >>> > > > > > > > For
>> > > > >>> > > > > > > > > > > > example, Flink jobs
>> > > > >>> > > > > > > > > > > > > > > consume data from Kafka,
>> then
>> > > > >>> connect
>> > > > >>> > > > external
>> > > > >>> > > > > > > > > > components
>> > > > >>> > > > > > > > > > > > such as HBase
>> > > > >>> > > > > > > > > > > > > > or
>> > > > >>> > > > > > > > > > > > > > > Redis to obtain additional
>> > > > business
>> > > > >>> > > > > information,
>> > > > >>> > > > > > > and
>> > > > >>> > > > > > > > > > then
>> > > > >>> > > > > > > > > > > > write it to the
>> > > > >>> > > > > > > > > > > > > > > downstream external
>> > components.
>> > > > >>> There
>> > > > >>> > are
>> > > > >>> > > > the
>> > > > >>> > > > > > > > > following
>> > > > >>> > > > > > > > > > > > problems:
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > 1. The connection
>> information
>> > of
>> > > > >>> > external
>> > > > >>> > > > > > > > components,
>> > > > >>> > > > > > > > > > such
>> > > > >>> > > > > > > > > > > > as IP, port
>> > > > >>> > > > > > > > > > > > > > and
>> > > > >>> > > > > > > > > > > > > > > user password, needs to be
>> > > > >>> configured in
>> > > > >>> > > the
>> > > > >>> > > > > > > > > application
>> > > > >>> > > > > > > > > > > > args and
>> > > > >>> > > > > > > > > > > > > > > transferred to the Flink
>> job,
>> > so
>> > > > >>> that
>> > > > >>> > the
>> > > > >>> > > > > > > connection
>> > > > >>> > > > > > > > > > > > information of
>> > > > >>> > > > > > > > > > > > > > > external components is
>> > > distributed
>> > > > >>> in
>> > > > >>> > > > multiple
>> > > > >>> > > > > > > > > > > > applications. Once the
>> > > > >>> > > > > > > > > > > > > > > connection information of
>> > > external
>> > > > >>> > > > components
>> > > > >>> > > > > > > > changes,
>> > > > >>> > > > > > > > > > > many
>> > > > >>> > > > > > > > > > > > application
>> > > > >>> > > > > > > > > > > > > > > args parameters need to be
>> > > > modified,
>> > > > >>> > which
>> > > > >>> > > > > will
>> > > > >>> > > > > > > lead
>> > > > >>> > > > > > > > > to
>> > > > >>> > > > > > > > > > > > high operation
>> > > > >>> > > > > > > > > > > > > > and
>> > > > >>> > > > > > > > > > > > > > > maintenance costs.
>> > > > >>> > > > > > > > > > > > > > > 2. There are many people in
>> > the
>> > > > team
>> > > > >>> > > > > developing
>> > > > >>> > > > > > > > > > real-time
>> > > > >>> > > > > > > > > > > > computing jobs.
>> > > > >>> > > > > > > > > > > > > > > There is no uniform
>> > > specification
>> > > > >>> for
>> > > > >>> > the
>> > > > >>> > > > > > > connection
>> > > > >>> > > > > > > > > > > > information of the
>> > > > >>> > > > > > > > > > > > > > > external components passed
>> to
>> > > the
>> > > > >>> job,
>> > > > >>> > > > > resulting
>> > > > >>> > > > > > > in
>> > > > >>> > > > > > > > > > > > different parameter
>> > > > >>> > > > > > > > > > > > > > > names of the same
>> component.
>> > > This
>> > > > is
>> > > > >>> > > > difficult
>> > > > >>> > > > > > to
>> > > > >>> > > > > > > > > count
>> > > > >>> > > > > > > > > > > > which external
>> > > > >>> > > > > > > > > > > > > > > components are dependent.
>> > > > >>> > > > > > > > > > > > > > > 3. In production practice,
>> > there
>> > > > are
>> > > > >>> > > usually
>> > > > >>> > > > > > > > multiple
>> > > > >>> > > > > > > > > > sets
>> > > > >>> > > > > > > > > > > > of
>> > > > >>> > > > > > > > > > > > > > > environments, such as test
>> > > > >>> environment
>> > > > >>> > and
>> > > > >>> > > > > > > > production
>> > > > >>> > > > > > > > > > > > environment. It is
>> > > > >>> > > > > > > > > > > > > > > not intuitive to judge
>> > whether a
>> > > > >>> > component
>> > > > >>> > > > > > belongs
>> > > > >>> > > > > > > > to
>> > > > >>> > > > > > > > > a
>> > > > >>> > > > > > > > > > > > test environment
>> > > > >>> > > > > > > > > > > > > > or
>> > > > >>> > > > > > > > > > > > > > > a production environment
>> > simply
>> > > > >>> through
>> > > > >>> > IP
>> > > > >>> > > > and
>> > > > >>> > > > > > > > ports.
>> > > > >>> > > > > > > > > > Even
>> > > > >>> > > > > > > > > > > > if it can be
>> > > > >>> > > > > > > > > > > > > > > judged, there are some
>> > > omissions.
>> > > > In
>> > > > >>> > this
>> > > > >>> > > > way,
>> > > > >>> > > > > > the
>> > > > >>> > > > > > > > > > > > connection information
>> > > > >>> > > > > > > > > > > > > > > online to the production
>> > > > >>> environment may
>> > > > >>> > > be
>> > > > >>> > > > > > > external
>> > > > >>> > > > > > > > > > > > components of the
>> > > > >>> > > > > > > > > > > > > > test
>> > > > >>> > > > > > > > > > > > > > > environment, or components
>> of
>> > > the
>> > > > >>> > > production
>> > > > >>> > > > > > > > > environment
>> > > > >>> > > > > > > > > > > are
>> > > > >>> > > > > > > > > > > > > > inadvertently
>> > > > >>> > > > > > > > > > > > > > > configured during testing,
>> > > leading
>> > > > >>> to
>> > > > >>> > > > > production
>> > > > >>> > > > > > > > > > failures.
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > Proposal:
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > 1. Create a unified module
>> to
>> > > > manage
>> > > > >>> > > > > variables.
>> > > > >>> > > > > > > > Users
>> > > > >>> > > > > > > > > > can
>> > > > >>> > > > > > > > > > > > define their
>> > > > >>> > > > > > > > > > > > > > own
>> > > > >>> > > > > > > > > > > > > > > variables in this module.
>> > > Variable
>> > > > >>> > > > information
>> > > > >>> > > > > > > > > includes
>> > > > >>> > > > > > > > > > > > variable code
>> > > > >>> > > > > > > > > > > > > > > (passed to Flink program as
>> > > > >>> parameter
>> > > > >>> > > name),
>> > > > >>> > > > > > > > variable
>> > > > >>> > > > > > > > > > > > value, variable
>> > > > >>> > > > > > > > > > > > > name,
>> > > > >>> > > > > > > > > > > > > > > variable description, and
>> > which
>> > > > >>> > > applications
>> > > > >>> > > > > > > depend
>> > > > >>> > > > > > > > on
>> > > > >>> > > > > > > > > > > > > > > 2. In the same Team module,
>> > the
>> > > > >>> variable
>> > > > >>> > > > code
>> > > > >>> > > > > or
>> > > > >>> > > > > > > > > > variable
>> > > > >>> > > > > > > > > > > > name is unique
>> > > > >>> > > > > > > > > > > > > > > - By using placeholders,
>> such
>> > as
>> > > > >>> adding
>> > > > >>> > --
>> > > > >>> > > > > kafka
>> > > > >>> > > > > > > > > > {mykafka}
>> > > > >>> > > > > > > > > > > > to the args of
>> > > > >>> > > > > > > > > > > > > > > the application, mykafka is
>> > the
>> > > > >>> variable
>> > > > >>> > > > code
>> > > > >>> > > > > > of a
>> > > > >>> > > > > > > > > > > variable
>> > > > >>> > > > > > > > > > > > > > > - Search by variable code
>> or
>> > > > >>> variable
>> > > > >>> > > name.
>> > > > >>> > > > > > After
>> > > > >>> > > > > > > a
>> > > > >>> > > > > > > > > > > > variable is selected,
>> > > > >>> > > > > > > > > > > > > > > the parameter transfer
>> method
>> > is
>> > > > >>> > displayed
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > If you have any
>> suggestions,
>> > > > >>> welcome to
>> > > > >>> > > > > > > participate
>> > > > >>> > > > > > > > in
>> > > > >>> > > > > > > > > > the
>> > > > >>> > > > > > > > > > > > discussion on
>> > > > >>> > > > > > > > > > > > > > > the mail list or issue, I
>> look
>> > > > >>> forward
>> > > > >>> > to
>> > > > >>> > > > your
>> > > > >>> > > > > > > > > feedback.
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > Best wishes
>> > > > >>> > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > >
>> > > > >>> > > > > >
>> > > > >>> > > > >
>> > > > >>> > > >
>> > > > >>> > >
>> > > > >>> >
>> > > > >>>
>> > > > >>
>> > > >
>> > >
>> >
>>
>
