Hi, Rui Fan
I'll start a discussion later.
Best
Chunjin Mu
Rui Fan <[email protected]> 于2022年10月14日周五 19:42写道：
> Hi guys,
>
> I agree with monster. And I suggest use variable name, variable value and
> description.
>
> By the way, there are some problems still need to confirm. What’s the
> placeholder? And how to use them when edit sql or datastream job? It’s
> better to discuss these asap.
>
> Best
> fanrui
>
> On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <[email protected]> wrote:
>
> > hi, huajie, fan rui, monster
> >
> > Thank you for your suggestions.
> >
> > Now please Huajie to confirm how to do it. If remove the variable name, I
> > can also accept it here. I will do small secondary development on our
> > production system based on the community to meet internal needs.
> >
> > 陈卓宇 <[email protected]> 于2022年10月14日周五 15:58写道：
> >
> > > "外部采集整合kafka" is the Chinese description of variable, so it makes sense
> > to
> > > put it under `Description`.
> > >
> > > Chunjin Mu <[email protected]> 于2022年10月14日周五 12:01写道：
> > >
> > > > The variable management module has three properties, including
> variable
> > > > code, variable name, and description, For example, the variable code
> is
> > > > "external.kafka.collect.brokers", and the variable name is
> > "外部采集整合kafka".
> > > > When we created the application, I forgot what the variable code was,
> > > but I
> > > > knew it was called "采集整合", so I can search Enter the "采集整合" keywords
> in
> > > the
> > > > box, and of course you can also search for kafka keywords.
> > > >
> > > > At the same time, I want to know more detailed information about this
> > > > kafka, such as who the operator is, what is the data retention
> period,
> > > and
> > > > how big is the cluster size, but I can't use the description to
> search,
> > > so
> > > > I can only search or variable code or variable name, and the variable
> > > name
> > > > is also for a simple display.
> > > >
> > > > Although external.kafka.collect.brokers already contains a lot of
> > > > information, I think this is just a naming convention. When we
> discuss
> > > this
> > > > variable, we will discuss it by using the "采集整合 kafka"
> > > >
> > > > We can discuss whether there is a problem with this design, what do
> you
> > > > think?
> > > >
> > > > Rui Fan <[email protected]> 于2022年10月10日周一 11:45写道：
> > > >
> > > > > Hi huajie:
> > > > >
> > > > > > If it is manually entered, it will always be treated as ordinary
> > > text.
> > > > >
> > > > > Users always copy some code from an old job to a new job.
> > > > > I'm afraid it might not meet the user's expectations.
> > > > >
> > > > > Best,
> > > > > fanrui
> > > > >
> > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <[email protected]>
> > wrote:
> > > > >
> > > > > > hi 田向阳:
> > > > > >
> > > > > > The current discussion is about variable management, which is a
> > > > > relatively
> > > > > > independent function, No other modules involved, you can check
> the
> > > > thread
> > > > > > [1]
> > > > > >
> > > > > >
> > > > > > [1] thread:
> > > > > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Huajie Wang
> > > > > >
> > > > > >
> > > > > >
> > > > > > 田向阳 <[email protected]> 于2022年10月8日周六 13:13写道：
> > > > > >
> > > > > > > is this module include a resource management cente r where jars
> > can
> > > > be
> > > > > > > chosen to provide to create a job
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ---- 回复的原邮件 ----
> > > > > > > | 发件人 | Huajie Wang<[email protected]> |
> > > > > > > | 日期 | 2022年10月08日 12:07 |
> > > > > > > | 收件人 | [email protected]<[email protected]> |
> > > > > > > | 抄送至 | |
> > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management
> module |
> > > > > > > hi 阿穆:
> > > > > > > Thank you for the clarification, My question is answered, but
> > now I
> > > > > have
> > > > > > > one more concern,
> > > > > > > I'm currently using {var} for variables, and I'm worried that
> one
> > > of
> > > > > the
> > > > > > > user's parameters includes "{" and "}", e.g :
> > > > > > >
> > > > > > > --password 'mypass{123}abc'
> > > > > > >
> > > > > > > So how do we parse parse this? How about ${var}
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Huajie Wang
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 阿穆 <[email protected]> 于2022年10月8日周六 11:36写道：
> > > > > > >
> > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > > > > > >
> > > > > > > > Thanks for your reply.
> > > > > > > >
> > > > > > > > I have integrated your replies. I really don't pay much
> > attention
> > > > to
> > > > > > SQL,
> > > > > > > > so I will do the following.
> > > > > > > >
> > > > > > > > 1.Each team has independent variables that do not affect each
> > > other
> > > > > and
> > > > > > > > are isolated from each other, but the same variable is unique
> > > under
> > > > > the
> > > > > > > > same team, so you need to specify the team when creating
> > > variables
> > > > > > (users
> > > > > > > > may belong to multiple teams, and the teams of the current
> user
> > > > will
> > > > > be
> > > > > > > > displayed here for users to choose)
> > > > > > > > 2.For Flink jar jobs, there are two ways to pass variables
> > > > > > > > - By using placeholders, such as adding --kafka {mykafka} to
> > the
> > > > args
> > > > > > of
> > > > > > > > the application, 'mykafka' is the variable code of a variable
> > > > > > > > - Search by variable code or variable name. After a variable
> is
> > > > > > selected,
> > > > > > > > the parameter transfer method is displayed
> > > > > > > > 3.Flink sql will use variables as placeholders, E.g:
> > > > > > > >
> > > > > > > > CREATE TABLE KafkaTableA (
> > > > > > > > `user_id` BIGINT,
> > > > > > > > `item_id` BIGINT,
> > > > > > > > `type` STRING
> > > > > > > > ) WITH (
> > > > > > > > 'connector' = 'kafka',
> > > > > > > > 'topic' = 'user_behaviorA',
> > > > > > > > 'properties.bootstrap.servers' =
> > > > > '{kafka_clusterA}',
> > > > > > > --
> > > > > > > > 'kafka_ ClusterA' is the variable code
> > > > > > > > 'properties.group.id' = 'testGroupA',
> > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > >
> > > > > > > > CREATE TABLE KafkaTableB (
> > > > > > > > `user_id` BIGINT,
> > > > > > > > `item_id` BIGINT
> > > > > > > > ) WITH (
> > > > > > > > 'connector' = 'kafka',
> > > > > > > > 'topic' = 'user_behaviorB',
> > > > > > > > 'properties.bootstrap.servers' =
> > > > > '{kafka_clusterB}',
> > > > > > > --
> > > > > > > > 'kafka_ ClusterB' is the variable code
> > > > > > > > 'properties.group.id' = 'testGroupB',
> > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > >
> > > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}'
> FROM
> > > > > > > > KafkaTableB -- 'type' is the variable code
> > > > > > > >
> > > > > > > > 4.In the future, we can consider setting some system built-in
> > > > > variables
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > ChunJin Mu
> > > > > > > >
> > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <[email protected]
> > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi 阿穆:
> > > > > > > > >
> > > > > > > > > It's a great proposal.
> > > > > > > > >
> > > > > > > > > For usage scenarios, I strongly recommend support for
> > > > replacing
> > > > > > > > variables
> > > > > > > > > in sql. E.g:
> > > > > > > > >
> > > > > > > > > CREATE TABLE KafkaTable (
> > > > > > > > > `user_id` BIGINT,
> > > > > > > > > `item_id` BIGINT
> > > > > > > > > ) WITH (
> > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > 'topic' = 'user_behavior',
> > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > > '{kafka_clusterA}',
> > > > > > > > > 'properties.group.id' = 'testGroup',
> > > > > > > > > 'scan.startup.mode' =
> 'earliest-offset',
> > > > > > > > > 'format' = 'csv'
> > > > > > > > > )
> > > > > > > > >
> > > > > > > > > Note: 'properties.bootstrap.servers' =
> '{kafka_clusterA}',
> > > > > > > > >
> > > > > > > > > For variable management, I have some question:
> > > > > > > > >
> > > > > > > > > 1. Is the parameter used within the team? If all teams
> > use a
> > > > > > common
> > > > > > > > > parameter, it
> > > > > > > > > needs to be defined for each team, right?
> > > > > > > > > 2. Which users can modify variables? Can all users
> modify
> > > all
> > > > > > > > variables of
> > > > > > > > > their team?
> > > > > > > > > After the variable is modified incorrectly, many jobs
> may
> > be
> > > > > > > > affected. So I
> > > > > > > > > suggest that
> > > > > > > > > only team admin or system admin have permission to
> modify,
> > > add
> > > > > and
> > > > > > > > delete
> > > > > > > > > variables.
> > > > > > > > >
> > > > > > > > > Best
> > > > > > > > > fanrui
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <
> [email protected]
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > hi 阿穆:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks for starting a very useful discussion, I have a
> > some
> > > > > > > questions:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1. The access boundary problem of variable management,
> is
> > > this
> > > > > all
> > > > > > > > > people in the platform can access these variables( no
> > matter
> > > > who
> > > > > > > > > defines) or that only team members can be access these
> > > > > variables.
> > > > > > > > >
> > > > > > > > > 2. There are two usage scenarios for variables that I
> know
> > > of,
> > > > > one
> > > > > > > is
> > > > > > > > > in the entry parameter of main, and the other is in
> > > flinksql.
> > > > > Are
> > > > > > > both
> > > > > > > > > supported, or only supported main entry ?
> > > > > > > > >
> > > > > > > > > about first question, my opinion:
> > > > > > > > >
> > > > > > > > > 1. The variable management is associated with the team.
> > > After
> > > > > the
> > > > > > > user
> > > > > > > > > logs in to the system, he must select a team (a user can
> > > > belong
> > > > > to
> > > > > > > one
> > > > > > > > > or more teams), The variable belongs to a team
> > > > > > > > >
> > > > > > > > > 2. All resources between teams are isolated, which means
> > > that
> > > > > > > > > variables are also isolated. In short, members of a team
> > can
> > > > use
> > > > > > and
> > > > > > > > > define these variables, which are invisible to other
> > > members,
> > > > > and
> > > > > > > > > other members cannot use these variables.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Welcome to your views
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Huajie Wang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 陈卓宇 <[email protected]> 于2022年10月7日周五
> > > 14:18写道：
> > > > > > > >
> > > > > > > > > > Is it possible to do this variable support in sql
> as
> > > well
> > > > > > > > > >
> > > > > > > > > > 阿穆 <[email protected]> 于2022年10月7日周五
> > > 13:54写道：
> > > > > > > > > >
> > > > > > > > > > > Hi guys,
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I want to start a discussion about the
> variable
> > > > > > management
> > > > > > > > module. Please
> > > > > > > > > > > see this issue issue-1477.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Background:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > In the actual production environment, Flink
> jobs
> > > are
> > > > > > > > generally complex,
> > > > > > > > > > > and usually require multiple external
> > components.
> > > > For
> > > > > > > > example, Flink jobs
> > > > > > > > > > > consume data from Kafka, then connect external
> > > > > > components
> > > > > > > > such as HBase
> > > > > > > > > > or
> > > > > > > > > > > Redis to obtain additional business
> information,
> > > and
> > > > > > then
> > > > > > > > write it to the
> > > > > > > > > > > downstream external components. There are the
> > > > > following
> > > > > > > > problems:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 1. The connection information of external
> > > > components,
> > > > > > such
> > > > > > > > as IP, port
> > > > > > > > > > and
> > > > > > > > > > > user password, needs to be configured in the
> > > > > application
> > > > > > > > args and
> > > > > > > > > > > transferred to the Flink job, so that the
> > > connection
> > > > > > > > information of
> > > > > > > > > > > external components is distributed in multiple
> > > > > > > > applications. Once the
> > > > > > > > > > > connection information of external components
> > > > changes,
> > > > > > > many
> > > > > > > > application
> > > > > > > > > > > args parameters need to be modified, which
> will
> > > lead
> > > > > to
> > > > > > > > high operation
> > > > > > > > > > and
> > > > > > > > > > > maintenance costs.
> > > > > > > > > > > 2. There are many people in the team
> developing
> > > > > > real-time
> > > > > > > > computing jobs.
> > > > > > > > > > > There is no uniform specification for the
> > > connection
> > > > > > > > information of the
> > > > > > > > > > > external components passed to the job,
> resulting
> > > in
> > > > > > > > different parameter
> > > > > > > > > > > names of the same component. This is difficult
> > to
> > > > > count
> > > > > > > > which external
> > > > > > > > > > > components are dependent.
> > > > > > > > > > > 3. In production practice, there are usually
> > > > multiple
> > > > > > sets
> > > > > > > > of
> > > > > > > > > > > environments, such as test environment and
> > > > production
> > > > > > > > environment. It is
> > > > > > > > > > > not intuitive to judge whether a component
> > belongs
> > > > to
> > > > > a
> > > > > > > > test environment
> > > > > > > > > > or
> > > > > > > > > > > a production environment simply through IP and
> > > > ports.
> > > > > > Even
> > > > > > > > if it can be
> > > > > > > > > > > judged, there are some omissions. In this way,
> > the
> > > > > > > > connection information
> > > > > > > > > > > online to the production environment may be
> > > external
> > > > > > > > components of the
> > > > > > > > > > test
> > > > > > > > > > > environment, or components of the production
> > > > > environment
> > > > > > > are
> > > > > > > > > > inadvertently
> > > > > > > > > > > configured during testing, leading to
> production
> > > > > > failures.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Proposal:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 1. Create a unified module to manage
> variables.
> > > > Users
> > > > > > can
> > > > > > > > define their
> > > > > > > > > > own
> > > > > > > > > > > variables in this module. Variable information
> > > > > includes
> > > > > > > > variable code
> > > > > > > > > > > (passed to Flink program as parameter name),
> > > > variable
> > > > > > > > value, variable
> > > > > > > > > name,
> > > > > > > > > > > variable description, and which applications
> > > depend
> > > > on
> > > > > > > > > > > 2. In the same Team module, the variable code
> or
> > > > > > variable
> > > > > > > > name is unique
> > > > > > > > > > > - By using placeholders, such as adding --
> kafka
> > > > > > {mykafka}
> > > > > > > > to the args of
> > > > > > > > > > > the application, mykafka is the variable code
> > of a
> > > > > > > variable
> > > > > > > > > > > - Search by variable code or variable name.
> > After
> > > a
> > > > > > > > variable is selected,
> > > > > > > > > > > the parameter transfer method is displayed
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > If you have any suggestions, welcome to
> > > participate
> > > > in
> > > > > > the
> > > > > > > > discussion on
> > > > > > > > > > > the mail list or issue, I look forward to your
> > > > > feedback.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best wishes
> > > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
