Hi Rui Fan:
Thanks for your feedback, This is indeed a problem, no matter what
placeholder rule, it may conflict with the user's input parameters,
I have come up with a better solution:
Need to interact with the front-end, when the user enters "${" an
auto-completion drop-down box will be triggered, giving the user a
list of variables list that can be selected, if a variable is
selected, then the style of "${var}" will have a special highlighting
effect, will remember the variable, the program will do the relevant
processing, and vice versa, if it is manually entered, it will always
be treated as ordinary text.
Best,
Huajie Wang
Rui Fan <[email protected]> 于2022年10月8日周六 12:52写道：
> Hi huajie,
>
> Changing from {var} to ${var} doesn't seem to solve the problem.
> e.g : --password 'mypass${123}abc'
> In other words: no matter what the placeholder rule is, it can be
> ambiguous.
>
> How about adding a config that allows users to disable
> substitution variables for the application?
>
> Best,
> fanrui
>
> On Sat, Oct 8, 2022 at 12:07 PM Huajie Wang <[email protected]> wrote:
>
> > hi 阿穆:
> > Thank you for the clarification, My question is answered, but now I have
> > one more concern,
> > I'm currently using {var} for variables, and I'm worried that one of the
> > user's parameters includes "{" and "}", e.g :
> >
> > --password 'mypass{123}abc'
> >
> > So how do we parse parse this? How about ${var}
> >
> >
> >
> > Best,
> > Huajie Wang
> >
> >
> >
> > 阿穆 <[email protected]> 于2022年10月8日周六 11:36写道：
> >
> > > Hi, monster, Huajie Wang, Rui Fan:
> > >
> > > Thanks for your reply.
> > >
> > > I have integrated your replies. I really don't pay much attention to
> SQL,
> > > so I will do the following.
> > >
> > > 1.Each team has independent variables that do not affect each other and
> > > are isolated from each other, but the same variable is unique under the
> > > same team, so you need to specify the team when creating variables
> (users
> > > may belong to multiple teams, and the teams of the current user will be
> > > displayed here for users to choose)
> > > 2.For Flink jar jobs, there are two ways to pass variables
> > > - By using placeholders, such as adding --kafka {mykafka} to the args
> of
> > > the application, 'mykafka' is the variable code of a variable
> > > - Search by variable code or variable name. After a variable is
> selected,
> > > the parameter transfer method is displayed
> > > 3.Flink sql will use variables as placeholders, E.g:
> > >
> > > CREATE TABLE KafkaTableA (
> > > `user_id` BIGINT,
> > > `item_id` BIGINT,
> > > `type` STRING
> > > ) WITH (
> > > 'connector' = 'kafka',
> > > 'topic' = 'user_behaviorA',
> > > 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > --
> > > 'kafka_ ClusterA' is the variable code
> > > 'properties.group.id' = 'testGroupA',
> > > 'scan.startup.mode' = 'earliest-offset',
> > >
> > > CREATE TABLE KafkaTableB (
> > > `user_id` BIGINT,
> > > `item_id` BIGINT
> > > ) WITH (
> > > 'connector' = 'kafka',
> > > 'topic' = 'user_behaviorB',
> > > 'properties.bootstrap.servers' = '{kafka_clusterB}',
> > --
> > > 'kafka_ ClusterB' is the variable code
> > > 'properties.group.id' = 'testGroupB',
> > > 'scan.startup.mode' = 'earliest-offset',
> > >
> > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM
> > > KafkaTableB -- 'type' is the variable code
> > >
> > > 4.In the future, we can consider setting some system built-in variables
> > >
> > > Best,
> > > ChunJin Mu
> > >
> > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <[email protected]>
> wrote:
> > >
> > > > Hi 阿穆:
> > > >
> > > > It's a great proposal.
> > > >
> > > > For usage scenarios, I strongly recommend support for replacing
> > > variables
> > > > in sql. E.g:
> > > >
> > > > CREATE TABLE KafkaTable (
> > > > `user_id` BIGINT,
> > > > `item_id` BIGINT
> > > > ) WITH (
> > > > 'connector' = 'kafka',
> > > > 'topic' = 'user_behavior',
> > > > 'properties.bootstrap.servers' =
> > '{kafka_clusterA}',
> > > > 'properties.group.id' = 'testGroup',
> > > > 'scan.startup.mode' = 'earliest-offset',
> > > > 'format' = 'csv'
> > > > )
> > > >
> > > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > > >
> > > > For variable management, I have some question:
> > > >
> > > > 1. Is the parameter used within the team? If all teams use a
> common
> > > > parameter, it
> > > > needs to be defined for each team, right?
> > > > 2. Which users can modify variables? Can all users modify all
> > > variables of
> > > > their team?
> > > > After the variable is modified incorrectly, many jobs may be
> > > affected. So I
> > > > suggest that
> > > > only team admin or system admin have permission to modify, add and
> > > delete
> > > > variables.
> > > >
> > > > Best
> > > > fanrui
> > >
> > >
> > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <[email protected]>
> wrote:
> > >
> > > > hi 阿穆:
> > > >
> > > >
> > > > Thanks for starting a very useful discussion, I have a some
> > questions:
> > > >
> > > >
> > > > 1. The access boundary problem of variable management, is this all
> > > > people in the platform can access these variables( no matter who
> > > > defines) or that only team members can be access these variables.
> > > >
> > > > 2. There are two usage scenarios for variables that I know of, one
> > is
> > > > in the entry parameter of main, and the other is in flinksql. Are
> > both
> > > > supported, or only supported main entry ?
> > > >
> > > > about first question, my opinion:
> > > >
> > > > 1. The variable management is associated with the team. After the
> > user
> > > > logs in to the system, he must select a team (a user can belong to
> > one
> > > > or more teams), The variable belongs to a team
> > > >
> > > > 2. All resources between teams are isolated, which means that
> > > > variables are also isolated. In short, members of a team can use
> and
> > > > define these variables, which are invisible to other members, and
> > > > other members cannot use these variables.
> > > >
> > > >
> > > >
> > > > Welcome to your views
> > > >
> > > >
> > > > Best,
> > > > Huajie Wang
> > > >
> > > >
> > > >
> > > >
> > > > 陈卓宇 <[email protected]> 于2022年10月7日周五 14:18写道：
> > >
> > > > > Is it possible to do this variable support in sql as well
> > > > >
> > > > > 阿穆 <[email protected]> 于2022年10月7日周五 13:54写道：
> > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > >
> > > > > > I want to start a discussion about the variable
> management
> > > module. Please
> > > > > > see this issue issue-1477.
> > > > > >
> > > > > >
> > > > > > Background:
> > > > > >
> > > > > >
> > > > > > In the actual production environment, Flink jobs are
> > > generally complex,
> > > > > > and usually require multiple external components. For
> > > example, Flink jobs
> > > > > > consume data from Kafka, then connect external
> components
> > > such as HBase
> > > > > or
> > > > > > Redis to obtain additional business information, and
> then
> > > write it to the
> > > > > > downstream external components. There are the following
> > > problems:
> > > > > >
> > > > > >
> > > > > > 1. The connection information of external components,
> such
> > > as IP, port
> > > > > and
> > > > > > user password, needs to be configured in the application
> > > args and
> > > > > > transferred to the Flink job, so that the connection
> > > information of
> > > > > > external components is distributed in multiple
> > > applications. Once the
> > > > > > connection information of external components changes,
> > many
> > > application
> > > > > > args parameters need to be modified, which will lead to
> > > high operation
> > > > > and
> > > > > > maintenance costs.
> > > > > > 2. There are many people in the team developing
> real-time
> > > computing jobs.
> > > > > > There is no uniform specification for the connection
> > > information of the
> > > > > > external components passed to the job, resulting in
> > > different parameter
> > > > > > names of the same component. This is difficult to count
> > > which external
> > > > > > components are dependent.
> > > > > > 3. In production practice, there are usually multiple
> sets
> > > of
> > > > > > environments, such as test environment and production
> > > environment. It is
> > > > > > not intuitive to judge whether a component belongs to a
> > > test environment
> > > > > or
> > > > > > a production environment simply through IP and ports.
> Even
> > > if it can be
> > > > > > judged, there are some omissions. In this way, the
> > > connection information
> > > > > > online to the production environment may be external
> > > components of the
> > > > > test
> > > > > > environment, or components of the production environment
> > are
> > > > > inadvertently
> > > > > > configured during testing, leading to production
> failures.
> > > > > >
> > > > > >
> > > > > > Proposal:
> > > > > >
> > > > > >
> > > > > > 1. Create a unified module to manage variables. Users
> can
> > > define their
> > > > > own
> > > > > > variables in this module. Variable information includes
> > > variable code
> > > > > > (passed to Flink program as parameter name), variable
> > > value, variable
> > > > name,
> > > > > > variable description, and which applications depend on
> > > > > > 2. In the same Team module, the variable code or
> variable
> > > name is unique
> > > > > > - By using placeholders, such as adding -- kafka
> {mykafka}
> > > to the args of
> > > > > > the application, mykafka is the variable code of a
> > variable
> > > > > > - Search by variable code or variable name. After a
> > > variable is selected,
> > > > > > the parameter transfer method is displayed
> > > > > >
> > > > > >
> > > > > >
> > > > > > If you have any suggestions, welcome to participate in
> the
> > > discussion on
> > > > > > the mail list or issue, I look forward to your feedback.
> > > > > >
> > > > > >
> > > > > > Best wishes
> > > > >
> >
>
