Hi huajie,
Thanks for your great proposal.
I have 2 questions:
1. Why do you write the sql content in the config file?
2. For the table config, could we use `env.table-property` as the prefix?
If the prefix of flink table config isn't table, what can we do?
StreamPark should not be affected by flink parameter naming.
The prefix of some table configs are sql-client. For example:
sql-client.display.max-column-width [1]
My suggested format:
```
env:
option: #cli option args
target: yarn-application # yarn-application, yarn-perjob
shutdownOnAttachedExit:
jobmanager:
...
property:
${StreamExecutionEnvironment.key} : $value
...
table-property:
table.exec.mini-batch.enabled : true
```
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/config/#sql-client-display-max-column-width
Best
Rui Fan
On Thu, Oct 20, 2022 at 4:48 PM Huajie Wang <[email protected]> wrote:
> Hello everyone, this discussion is about the development of a unified
> specification for flink'job profiles in streampark, Welcome to join the
> discussion
>
>
>
>
> *background:*
> Streampark is positioned as a rapid development framework such as flink &
> spark. An important part of it is standardized configuration: put all the
> configurations hardcoded in the code into the configuration file. When the
> project starts, you only need to pass in the agreed configuration. The file
> can complete the initialization of the environment and the setting of
> parameters,
> Because the parameter specification customization in the current version is
> somewhat unreasonable, the specific performance is as follows: the format
> of the parameter is redefined, which is slightly different from the
> official configuration of flink. For this part, pr has already done related
> work [1], the specific method It is to put the parameter settings of env in
> flink under property.
> The key is the key of the standard parameter in flink, but this part only
> regulates the parameter configuration under the property, and does not
> regulate the global parameter setting.
>
>
> The current configuration rules is as follows:
>
> flink:
> deployment:
> option:
> ...
> property:
> ...
>
>
> For example: Now the flink'job is deployed in yarn-perjob mode, the job
> name is: test-job, the parallelism is 2, and the entity class is:
> org.apache.streampark.FlinkJob, so the configuration is as follows:
>
> flink:
> deployment:
> option:
> target: yarn-per-job
> property:
> $internal.application.main: org.apache.streampark.FlinkJob
> pipeline.name: test-job
> taskmanager.numberOfTaskSlots: 1
> parallelism.default: 2
>
>
> we can see, root prefix is `flink`, The `option` defined the parameters
> related to the deployment task,
> and the `property` defined the parameter configuration in flink. The
> configurable parameters is completely consistent with the standard
> parameters in flink [2], There are deficiencies in this design
> specification, which are manifested as follows:
>
> 1. The format of table-related parameter settings is not defined
> 2. The user's business parameters are not defined
> 3. The content of flinksql is not defined.
>
> Therefore, the purpose of this discussion is to solve this problem and
> further standardize the parameters. Since the design of this part of the
> specification is more important, it will directly affect the users
> developed with the streampark api, so it is necessary for us to conduct
> in-depth communication and discussion.
>
>
>
> *Proposal:*
> The improved format I initially proposed[3] is for example, the parameters
> are generally divided into three parts, env, app, sql, "env" defined
> deployment parameters and environment setting related parameters, and table
> parameters, "app" defined user-defined parameters, "sql" defined the
> content of flinksql.
>
> env:
> option: #cli opiton args
> target: yarn-application # yarn-application, yarn-perjob
> shutdownOnAttachedExit:
> jobmanager:
> ...
> property:
> ${StreamExecutionEnvironment.key} : $value
> ...
> table:
> ${TableEnvironment.key} : $value
> ...
> sql: # flinksql
> my_flinksql: |
> CREATE TABLE datagen (
> f_sequence INT,
> ts AS localtimestamp,
> WATERMARK FOR ts AS ts
> ) WITH (
> ....
> );
> ...
>
> app:
> kafka.bootstrap:
> kafka.topic: test
> ...
>
>
> Looking forward to your opinion.
>
>
>
> [1] : https://github.com/apache/incubator-streampark/issues/1762
> [2] :
>
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/config/
> [3] : https://github.com/apache/incubator-streampark/issues/1857
>
> Best,
> Huajie Wang
>
>
>
> Best,
> Huajie Wang
>
