Hi 阿穆:
It's a great proposal.
For usage scenarios, I strongly recommend support for replacing variables
in sql. E.g:
CREATE TABLE KafkaTable (
`user_id` BIGINT,
`item_id` BIGINT
) WITH (
'connector' = 'kafka',
'topic' = 'user_behavior',
'properties.bootstrap.servers' = '{kafka_clusterA}',
'properties.group.id' = 'testGroup',
'scan.startup.mode' = 'earliest-offset',
'format' = 'csv'
)
Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
For variable management, I have some question:
1. Is the parameter used within the team? If all teams use a common
parameter, it
needs to be defined for each team, right?
2. Which users can modify variables? Can all users modify all variables of
their team?
After the variable is modified incorrectly, many jobs may be affected. So I
suggest that
only team admin or system admin have permission to modify, add and delete
variables.
Best
fanrui
On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <[email protected]> wrote:
> hi 阿穆:
>
>
> Thanks for starting a very useful discussion, I have a some questions:
>
>
> 1. The access boundary problem of variable management, is this all
> people in the platform can access these variables( no matter who
> defines) or that only team members can be access these variables.
>
> 2. There are two usage scenarios for variables that I know of, one is
> in the entry parameter of main, and the other is in flinksql. Are both
> supported, or only supported main entry ?
>
> about first question, my opinion:
>
> 1. The variable management is associated with the team. After the user
> logs in to the system, he must select a team (a user can belong to one
> or more teams), The variable belongs to a team
>
> 2. All resources between teams are isolated, which means that
> variables are also isolated. In short, members of a team can use and
> define these variables, which are invisible to other members, and
> other members cannot use these variables.
>
>
>
> Welcome to your views
>
>
> Best,
> Huajie Wang
>
>
>
>
> 陈卓宇 <[email protected]> 于2022年10月7日周五 14:18写道：
>
> > Is it possible to do this variable support in sql as well
> >
> > 阿穆 <[email protected]> 于2022年10月7日周五 13:54写道：
> >
> > > Hi guys,
> > >
> > >
> > > I want to start a discussion about the variable management module.
> Please
> > > see this issue issue-1477.
> > >
> > >
> > > Background:
> > >
> > >
> > > In the actual production environment, Flink jobs are generally complex,
> > > and usually require multiple external components. For example, Flink
> jobs
> > > consume data from Kafka, then connect external components such as HBase
> > or
> > > Redis to obtain additional business information, and then write it to
> the
> > > downstream external components. There are the following problems:
> > >
> > >
> > > 1. The connection information of external components, such as IP, port
> > and
> > > user password, needs to be configured in the application args and
> > > transferred to the Flink job, so that the connection information of
> > > external components is distributed in multiple applications. Once the
> > > connection information of external components changes, many application
> > > args parameters need to be modified, which will lead to high operation
> > and
> > > maintenance costs.
> > > 2. There are many people in the team developing real-time computing
> jobs.
> > > There is no uniform specification for the connection information of the
> > > external components passed to the job, resulting in different parameter
> > > names of the same component. This is difficult to count which external
> > > components are dependent.
> > > 3. In production practice, there are usually multiple sets of
> > > environments, such as test environment and production environment. It
> is
> > > not intuitive to judge whether a component belongs to a test
> environment
> > or
> > > a production environment simply through IP and ports. Even if it can be
> > > judged, there are some omissions. In this way, the connection
> information
> > > online to the production environment may be external components of the
> > test
> > > environment, or components of the production environment are
> > inadvertently
> > > configured during testing, leading to production failures.
> > >
> > >
> > > Proposal:
> > >
> > >
> > > 1. Create a unified module to manage variables. Users can define their
> > own
> > > variables in this module. Variable information includes variable code
> > > (passed to Flink program as parameter name), variable value, variable
> > name,
> > > variable description, and which applications depend on
> > > 2. In the same Team module, the variable code or variable name is
> unique
> > > - By using placeholders, such as adding -- kafka {mykafka} to the args
> of
> > > the application, mykafka is the variable code of a variable
> > > - Search by variable code or variable name. After a variable is
> selected,
> > > the parameter transfer method is displayed
> > >
> > >
> > >
> > > If you have any suggestions, welcome to participate in the discussion
> on
> > > the mail list or issue, I look forward to your feedback.
> > >
> > >
> > > Best wishes
> >
>
