Hi huajie, Thanks for your great proposal. for about ‘app: # user's parameter’，I have a question: ‘user's parameter’：such as database connection, have to start with app or support custom starting? Can this parameter be used only in the streampark development framework or can it be used by the streampark platform.
There are two suggestions for sql schema reading: 1. The option parameter is assigned to DeplymentOptions, which is consistent with flink. 2.table identifies table Api and SQL configuration with overlapping table prefixes and SQL Client Options parameter does not take effect. Can we specifically call it table-property? Because I understand that it is a unique parameter in a table scenario of flink property. Huajie Wang <[email protected]> 于2022年10月21日周五 16:22写道： > > hi all: > > Redefined the rules of the configuration file as follows: > > flink: > option: > ... > property: #@see: > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/config/ > ... > table: # @see > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/config/ > ... > > app: # user's parameter > ... > sql: > ... > > > > Looking forward to your opinion. > > > > Best, > Huajie Wang > > > > Huajie Wang <[email protected]> 于2022年10月20日周四 23:18写道： > > > > 2. For the table config, could we use `env.table-property` as the prefix? > > If the prefix of flink table config isn't table, what can we do? > > StreamPark should not be affected by flink parameter naming. > > > > AFAIK, flink table'property key all startwith "table", sql-client is a > > special case, sql-client is just a program that flink comes with to execute > > sql, in other words, I don't need sql-client to execute sql, so I don't > > need those parameters, There is an essential difference between the > > parameters defined by sql-client and the table'properties > > > > Best, > > Huajie Wang > > > > > > > > Rui Fan <[email protected]> 于2022年10月20日周四 22:10写道： > > > >> Hi huajie, > >> > >> Thanks for your great proposal. > >> > >> I have 2 questions: > >> 1. Why do you write the sql content in the config file? > >> 2. For the table config, could we use `env.table-property` as the prefix? > >> If the prefix of flink table config isn't table, what can we do? > >> StreamPark should not be affected by flink parameter naming. > >> > >> The prefix of some table configs are sql-client. For example: > >> sql-client.display.max-column-width [1] > >> > >> My suggested format: > >> > >> ``` > >> env: > >> option: #cli option args > >> target: yarn-application # yarn-application, yarn-perjob > >> shutdownOnAttachedExit: > >> jobmanager: > >> ... > >> property: > >> ${StreamExecutionEnvironment.key} : $value > >> ... > >> table-property: > >> table.exec.mini-batch.enabled : true > >> ``` > >> > >> > >> [1] > >> > >> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/config/#sql-client-display-max-column-width > >> > >> Best > >> Rui Fan > >> > >> On Thu, Oct 20, 2022 at 4:48 PM Huajie Wang <[email protected]> wrote: > >> > >> > Hello everyone, this discussion is about the development of a unified > >> > specification for flink'job profiles in streampark, Welcome to join the > >> > discussion > >> > > >> > > >> > > >> > > >> > *background:* > >> > Streampark is positioned as a rapid development framework such as flink > >> & > >> > spark. An important part of it is standardized configuration: put all > >> the > >> > configurations hardcoded in the code into the configuration file. When > >> the > >> > project starts, you only need to pass in the agreed configuration. The > >> file > >> > can complete the initialization of the environment and the setting of > >> > parameters, > >> > Because the parameter specification customization in the current > >> version is > >> > somewhat unreasonable, the specific performance is as follows: the > >> format > >> > of the parameter is redefined, which is slightly different from the > >> > official configuration of flink. For this part, pr has already done > >> related > >> > work [1], the specific method It is to put the parameter settings of > >> env in > >> > flink under property. > >> > The key is the key of the standard parameter in flink, but this part > >> only > >> > regulates the parameter configuration under the property, and does not > >> > regulate the global parameter setting. > >> > > >> > > >> > The current configuration rules is as follows: > >> > > >> > flink: > >> > deployment: > >> > option: > >> > ... > >> > property: > >> > ... > >> > > >> > > >> > For example: Now the flink'job is deployed in yarn-perjob mode, the job > >> > name is: test-job, the parallelism is 2, and the entity class is: > >> > org.apache.streampark.FlinkJob, so the configuration is as follows: > >> > > >> > flink: > >> > deployment: > >> > option: > >> > target: yarn-per-job > >> > property: > >> > $internal.application.main: org.apache.streampark.FlinkJob > >> > pipeline.name: test-job > >> > taskmanager.numberOfTaskSlots: 1 > >> > parallelism.default: 2 > >> > > >> > > >> > we can see, root prefix is `flink`, The `option` defined the parameters > >> > related to the deployment task, > >> > and the `property` defined the parameter configuration in flink. The > >> > configurable parameters is completely consistent with the standard > >> > parameters in flink [2], There are deficiencies in this design > >> > specification, which are manifested as follows: > >> > > >> > 1. The format of table-related parameter settings is not defined > >> > 2. The user's business parameters are not defined > >> > 3. The content of flinksql is not defined. > >> > > >> > Therefore, the purpose of this discussion is to solve this problem and > >> > further standardize the parameters. Since the design of this part of the > >> > specification is more important, it will directly affect the users > >> > developed with the streampark api, so it is necessary for us to conduct > >> > in-depth communication and discussion. > >> > > >> > > >> > > >> > *Proposal:* > >> > The improved format I initially proposed[3] is for example, the > >> parameters > >> > are generally divided into three parts, env, app, sql, "env" defined > >> > deployment parameters and environment setting related parameters, and > >> table > >> > parameters, "app" defined user-defined parameters, "sql" defined the > >> > content of flinksql. > >> > > >> > env: > >> > option: #cli opiton args > >> > target: yarn-application # yarn-application, yarn-perjob > >> > shutdownOnAttachedExit: > >> > jobmanager: > >> > ... > >> > property: > >> > ${StreamExecutionEnvironment.key} : $value > >> > ... > >> > table: > >> > ${TableEnvironment.key} : $value > >> > ... > >> > sql: # flinksql > >> > my_flinksql: | > >> > CREATE TABLE datagen ( > >> > f_sequence INT, > >> > ts AS localtimestamp, > >> > WATERMARK FOR ts AS ts > >> > ) WITH ( > >> > .... > >> > ); > >> > ... > >> > > >> > app: > >> > kafka.bootstrap: > >> > kafka.topic: test > >> > ... > >> > > >> > > >> > Looking forward to your opinion. > >> > > >> > > >> > > >> > [1] : https://github.com/apache/incubator-streampark/issues/1762 > >> > [2] : > >> > > >> > > >> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/config/ > >> > [3] : https://github.com/apache/incubator-streampark/issues/1857 > >> > > >> > Best, > >> > Huajie Wang > >> > > >> > > >> > > >> > Best, > >> > Huajie Wang > >> > > >> > >
