Hi huajie,
Changing from {var} to ${var} doesn't seem to solve the problem.
e.g : --password 'mypass${123}abc'
In other words: no matter what the placeholder rule is, it can be
ambiguous.
How about adding a config that allows users to disable
substitution variables for the application?
Best,
fanrui
On Sat, Oct 8, 2022 at 12:07 PM Huajie Wang <[email protected]> wrote:
> hi 阿穆:
> Thank you for the clarification, My question is answered, but now I have
> one more concern,
> I'm currently using {var} for variables, and I'm worried that one of the
> user's parameters includes "{" and "}", e.g :
>
> --password 'mypass{123}abc'
>
> So how do we parse parse this? How about ${var}
>
>
>
> Best,
> Huajie Wang
>
>
>
> 阿穆 <[email protected]> 于2022年10月8日周六 11:36写道：
>
> > Hi, monster, Huajie Wang, Rui Fan:
> >
> > Thanks for your reply.
> >
> > I have integrated your replies. I really don't pay much attention to SQL,
> > so I will do the following.
> >
> > 1.Each team has independent variables that do not affect each other and
> > are isolated from each other, but the same variable is unique under the
> > same team, so you need to specify the team when creating variables (users
> > may belong to multiple teams, and the teams of the current user will be
> > displayed here for users to choose)
> > 2.For Flink jar jobs, there are two ways to pass variables
> > - By using placeholders, such as adding --kafka {mykafka} to the args of
> > the application, 'mykafka' is the variable code of a variable
> > - Search by variable code or variable name. After a variable is selected,
> > the parameter transfer method is displayed
> > 3.Flink sql will use variables as placeholders, E.g:
> >
> > CREATE TABLE KafkaTableA (
> > `user_id` BIGINT,
> > `item_id` BIGINT,
> > `type` STRING
> > ) WITH (
> > 'connector' = 'kafka',
> > 'topic' = 'user_behaviorA',
> > 'properties.bootstrap.servers' = '{kafka_clusterA}',
> --
> > 'kafka_ ClusterA' is the variable code
> > 'properties.group.id' = 'testGroupA',
> > 'scan.startup.mode' = 'earliest-offset',
> >
> > CREATE TABLE KafkaTableB (
> > `user_id` BIGINT,
> > `item_id` BIGINT
> > ) WITH (
> > 'connector' = 'kafka',
> > 'topic' = 'user_behaviorB',
> > 'properties.bootstrap.servers' = '{kafka_clusterB}',
> --
> > 'kafka_ ClusterB' is the variable code
> > 'properties.group.id' = 'testGroupB',
> > 'scan.startup.mode' = 'earliest-offset',
> >
> > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM
> > KafkaTableB -- 'type' is the variable code
> >
> > 4.In the future, we can consider setting some system built-in variables
> >
> > Best,
> > ChunJin Mu
> >
> > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <[email protected]> wrote:
> >
> > > Hi 阿穆:
> > >
> > > It's a great proposal.
> > >
> > > For usage scenarios, I strongly recommend support for replacing
> > variables
> > > in sql. E.g:
> > >
> > > CREATE TABLE KafkaTable (
> > > `user_id` BIGINT,
> > > `item_id` BIGINT
> > > ) WITH (
> > > 'connector' = 'kafka',
> > > 'topic' = 'user_behavior',
> > > 'properties.bootstrap.servers' =
> '{kafka_clusterA}',
> > > 'properties.group.id' = 'testGroup',
> > > 'scan.startup.mode' = 'earliest-offset',
> > > 'format' = 'csv'
> > > )
> > >
> > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > >
> > > For variable management, I have some question:
> > >
> > > 1. Is the parameter used within the team? If all teams use a common
> > > parameter, it
> > > needs to be defined for each team, right?
> > > 2. Which users can modify variables? Can all users modify all
> > variables of
> > > their team?
> > > After the variable is modified incorrectly, many jobs may be
> > affected. So I
> > > suggest that
> > > only team admin or system admin have permission to modify, add and
> > delete
> > > variables.
> > >
> > > Best
> > > fanrui
> >
> >
> > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <[email protected]> wrote:
> >
> > > hi 阿穆:
> > >
> > >
> > > Thanks for starting a very useful discussion, I have a some
> questions:
> > >
> > >
> > > 1. The access boundary problem of variable management, is this all
> > > people in the platform can access these variables( no matter who
> > > defines) or that only team members can be access these variables.
> > >
> > > 2. There are two usage scenarios for variables that I know of, one
> is
> > > in the entry parameter of main, and the other is in flinksql. Are
> both
> > > supported, or only supported main entry ?
> > >
> > > about first question, my opinion:
> > >
> > > 1. The variable management is associated with the team. After the
> user
> > > logs in to the system, he must select a team (a user can belong to
> one
> > > or more teams), The variable belongs to a team
> > >
> > > 2. All resources between teams are isolated, which means that
> > > variables are also isolated. In short, members of a team can use and
> > > define these variables, which are invisible to other members, and
> > > other members cannot use these variables.
> > >
> > >
> > >
> > > Welcome to your views
> > >
> > >
> > > Best,
> > > Huajie Wang
> > >
> > >
> > >
> > >
> > > 陈卓宇 <[email protected]> 于2022年10月7日周五 14:18写道：
> >
> > > > Is it possible to do this variable support in sql as well
> > > >
> > > > 阿穆 <[email protected]> 于2022年10月7日周五 13:54写道：
> > > >
> > > > > Hi guys,
> > > > >
> > > > >
> > > > > I want to start a discussion about the variable management
> > module. Please
> > > > > see this issue issue-1477.
> > > > >
> > > > >
> > > > > Background:
> > > > >
> > > > >
> > > > > In the actual production environment, Flink jobs are
> > generally complex,
> > > > > and usually require multiple external components. For
> > example, Flink jobs
> > > > > consume data from Kafka, then connect external components
> > such as HBase
> > > > or
> > > > > Redis to obtain additional business information, and then
> > write it to the
> > > > > downstream external components. There are the following
> > problems:
> > > > >
> > > > >
> > > > > 1. The connection information of external components, such
> > as IP, port
> > > > and
> > > > > user password, needs to be configured in the application
> > args and
> > > > > transferred to the Flink job, so that the connection
> > information of
> > > > > external components is distributed in multiple
> > applications. Once the
> > > > > connection information of external components changes,
> many
> > application
> > > > > args parameters need to be modified, which will lead to
> > high operation
> > > > and
> > > > > maintenance costs.
> > > > > 2. There are many people in the team developing real-time
> > computing jobs.
> > > > > There is no uniform specification for the connection
> > information of the
> > > > > external components passed to the job, resulting in
> > different parameter
> > > > > names of the same component. This is difficult to count
> > which external
> > > > > components are dependent.
> > > > > 3. In production practice, there are usually multiple sets
> > of
> > > > > environments, such as test environment and production
> > environment. It is
> > > > > not intuitive to judge whether a component belongs to a
> > test environment
> > > > or
> > > > > a production environment simply through IP and ports. Even
> > if it can be
> > > > > judged, there are some omissions. In this way, the
> > connection information
> > > > > online to the production environment may be external
> > components of the
> > > > test
> > > > > environment, or components of the production environment
> are
> > > > inadvertently
> > > > > configured during testing, leading to production failures.
> > > > >
> > > > >
> > > > > Proposal:
> > > > >
> > > > >
> > > > > 1. Create a unified module to manage variables. Users can
> > define their
> > > > own
> > > > > variables in this module. Variable information includes
> > variable code
> > > > > (passed to Flink program as parameter name), variable
> > value, variable
> > > name,
> > > > > variable description, and which applications depend on
> > > > > 2. In the same Team module, the variable code or variable
> > name is unique
> > > > > - By using placeholders, such as adding -- kafka {mykafka}
> > to the args of
> > > > > the application, mykafka is the variable code of a
> variable
> > > > > - Search by variable code or variable name. After a
> > variable is selected,
> > > > > the parameter transfer method is displayed
> > > > >
> > > > >
> > > > >
> > > > > If you have any suggestions, welcome to participate in the
> > discussion on
> > > > > the mail list or issue, I look forward to your feedback.
> > > > >
> > > > >
> > > > > Best wishes
> > > >
>
