hi 阿穆:
Thank you for the clarification, My question is answered, but now I have
one more concern,
I'm currently using {var} for variables, and I'm worried that one of the
user's parameters includes "{" and "}", e.g :
--password 'mypass{123}abc'
So how do we parse parse this? How about ${var}
Best,
Huajie Wang
阿穆 <[email protected]> 于2022年10月8日周六 11:36写道：
> Hi, monster, Huajie Wang, Rui Fan:
>
> Thanks for your reply.
>
> I have integrated your replies. I really don't pay much attention to SQL,
> so I will do the following.
>
> 1.Each team has independent variables that do not affect each other and
> are isolated from each other, but the same variable is unique under the
> same team, so you need to specify the team when creating variables (users
> may belong to multiple teams, and the teams of the current user will be
> displayed here for users to choose)
> 2.For Flink jar jobs, there are two ways to pass variables
> - By using placeholders, such as adding --kafka {mykafka} to the args of
> the application, 'mykafka' is the variable code of a variable
> - Search by variable code or variable name. After a variable is selected,
> the parameter transfer method is displayed
> 3.Flink sql will use variables as placeholders, E.g:
>
> CREATE TABLE KafkaTableA (
> `user_id` BIGINT,
> `item_id` BIGINT,
> `type` STRING
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'user_behaviorA',
> 'properties.bootstrap.servers' = '{kafka_clusterA}', --
> 'kafka_ ClusterA' is the variable code
> 'properties.group.id' = 'testGroupA',
> 'scan.startup.mode' = 'earliest-offset',
>
> CREATE TABLE KafkaTableB (
> `user_id` BIGINT,
> `item_id` BIGINT
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'user_behaviorB',
> 'properties.bootstrap.servers' = '{kafka_clusterB}', --
> 'kafka_ ClusterB' is the variable code
> 'properties.group.id' = 'testGroupB',
> 'scan.startup.mode' = 'earliest-offset',
>
> INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM
> KafkaTableB -- 'type' is the variable code
>
> 4.In the future, we can consider setting some system built-in variables
>
> Best,
> ChunJin Mu
>
> On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <[email protected]> wrote:
>
> > Hi 阿穆:
> >
> > It's a great proposal.
> >
> > For usage scenarios, I strongly recommend support for replacing
> variables
> > in sql. E.g:
> >
> > CREATE TABLE KafkaTable (
> > `user_id` BIGINT,
> > `item_id` BIGINT
> > ) WITH (
> > 'connector' = 'kafka',
> > 'topic' = 'user_behavior',
> > 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > 'properties.group.id' = 'testGroup',
> > 'scan.startup.mode' = 'earliest-offset',
> > 'format' = 'csv'
> > )
> >
> > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
> >
> > For variable management, I have some question:
> >
> > 1. Is the parameter used within the team? If all teams use a common
> > parameter, it
> > needs to be defined for each team, right?
> > 2. Which users can modify variables? Can all users modify all
> variables of
> > their team?
> > After the variable is modified incorrectly, many jobs may be
> affected. So I
> > suggest that
> > only team admin or system admin have permission to modify, add and
> delete
> > variables.
> >
> > Best
> > fanrui
>
>
> On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <[email protected]> wrote:
>
> > hi 阿穆:
> >
> >
> > Thanks for starting a very useful discussion, I have a some questions:
> >
> >
> > 1. The access boundary problem of variable management, is this all
> > people in the platform can access these variables( no matter who
> > defines) or that only team members can be access these variables.
> >
> > 2. There are two usage scenarios for variables that I know of, one is
> > in the entry parameter of main, and the other is in flinksql. Are both
> > supported, or only supported main entry ?
> >
> > about first question, my opinion:
> >
> > 1. The variable management is associated with the team. After the user
> > logs in to the system, he must select a team (a user can belong to one
> > or more teams), The variable belongs to a team
> >
> > 2. All resources between teams are isolated, which means that
> > variables are also isolated. In short, members of a team can use and
> > define these variables, which are invisible to other members, and
> > other members cannot use these variables.
> >
> >
> >
> > Welcome to your views
> >
> >
> > Best,
> > Huajie Wang
> >
> >
> >
> >
> > 陈卓宇 <[email protected]> 于2022年10月7日周五 14:18写道：
>
> > > Is it possible to do this variable support in sql as well
> > >
> > > 阿穆 <[email protected]> 于2022年10月7日周五 13:54写道：
> > >
> > > > Hi guys,
> > > >
> > > >
> > > > I want to start a discussion about the variable management
> module. Please
> > > > see this issue issue-1477.
> > > >
> > > >
> > > > Background:
> > > >
> > > >
> > > > In the actual production environment, Flink jobs are
> generally complex,
> > > > and usually require multiple external components. For
> example, Flink jobs
> > > > consume data from Kafka, then connect external components
> such as HBase
> > > or
> > > > Redis to obtain additional business information, and then
> write it to the
> > > > downstream external components. There are the following
> problems:
> > > >
> > > >
> > > > 1. The connection information of external components, such
> as IP, port
> > > and
> > > > user password, needs to be configured in the application
> args and
> > > > transferred to the Flink job, so that the connection
> information of
> > > > external components is distributed in multiple
> applications. Once the
> > > > connection information of external components changes, many
> application
> > > > args parameters need to be modified, which will lead to
> high operation
> > > and
> > > > maintenance costs.
> > > > 2. There are many people in the team developing real-time
> computing jobs.
> > > > There is no uniform specification for the connection
> information of the
> > > > external components passed to the job, resulting in
> different parameter
> > > > names of the same component. This is difficult to count
> which external
> > > > components are dependent.
> > > > 3. In production practice, there are usually multiple sets
> of
> > > > environments, such as test environment and production
> environment. It is
> > > > not intuitive to judge whether a component belongs to a
> test environment
> > > or
> > > > a production environment simply through IP and ports. Even
> if it can be
> > > > judged, there are some omissions. In this way, the
> connection information
> > > > online to the production environment may be external
> components of the
> > > test
> > > > environment, or components of the production environment are
> > > inadvertently
> > > > configured during testing, leading to production failures.
> > > >
> > > >
> > > > Proposal:
> > > >
> > > >
> > > > 1. Create a unified module to manage variables. Users can
> define their
> > > own
> > > > variables in this module. Variable information includes
> variable code
> > > > (passed to Flink program as parameter name), variable
> value, variable
> > name,
> > > > variable description, and which applications depend on
> > > > 2. In the same Team module, the variable code or variable
> name is unique
> > > > - By using placeholders, such as adding -- kafka {mykafka}
> to the args of
> > > > the application, mykafka is the variable code of a variable
> > > > - Search by variable code or variable name. After a
> variable is selected,
> > > > the parameter transfer method is displayed
> > > >
> > > >
> > > >
> > > > If you have any suggestions, welcome to participate in the
> discussion on
> > > > the mail list or issue, I look forward to your feedback.
> > > >
> > > >
> > > > Best wishes
> > >
