Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hi everyone,
Thanks for all the comments! Based on the discussion so far, I will proceed
to initiate the vote tomorrow if there are no further discussions or
objections.
Best regards,
Junrui
Junrui Lee 于2023年10月9日周一 11:52写道：
> Hi, Chesnay and David, sorry for the late reply.
>
> Thanks for your feedback about this FLIP.
>
> @Chesnay
>
> >> Personally I'd just name it "config.yaml"
> Thanks for your suggestions, I agree that "config.yaml" is a simpler and
> more intuitive name. I have updated the FLIP accordingly.
>
> >> In terms of scope, is the migration of existing e2e tests and the
> docker image to the new parser part of the FLIP?
> The migration of existing end-to-end tests and the docker image to the new
> parser is indeed included in the FLIP. Taking into account the difficulty
> of using shell scripts to modify configuration files that adhere to
> standard YAML syntax, I propose utilizing the
> org.apache.flink.runtime.util.bash.BashJavaUtils class as a unified utility
> for modifying the configuration file.
>
> In addition, considering that users may not only configure properties in
> the configuration file, but also use dynamic parameters like '-D' or
> configure the "with" clauses in FLINK SQL. When the default parser is
> changed to the standard YAML parser, the configOption whose value is List
> or Map types also needs to be modified accordingly. This migration cost is
> relatively high because users have to modify their code or command lines.
> To reduce migration costs while considering compatibility, I propose to
> add compatibility support in FLINK-1.X versions to parse old format List
> and Map values using the standard YAML parser. In case of an error while
> parsing List or Map configurations with the standard YAML parser, a WARNING
> log will be printed, and an attempt will be made to parse using the old
> parser. And this support will be removed in the FLINK-2.0 version.
>
> Do you think the aforementioned compatibility handling is necessary in
> order to reduce migration costs? If you agree, I will also update this
> support in the FLIP under the migration section.
>
> @David
>
> >> could we write a migration tool, that would convert the old config
> files to the new format,
> I agree that providing a more user-friendly guidance for the migration is
> important. In the FLIP, we can include a user migration manual to help
> users with the migration process. This migration manual will be presented
> on the official FLINK website, making it easily accessible for users. And
> the migrated configuration files will adhere to standard YAML syntax, and
> users can validate them using various third-party YAML tools.
>
> Regarding the automated migration tool, I think it can be considered as a
> follow-up task rather than a requirement for the initial version. We can
> focus on implementing the migration manually first and then explore the
> possibility of automating it in future updates.
>
> Best,
> Junrui
>
> David Radley 于2023年10月3日周二 18:56写道：
>
>> Hi,
>> I agree this is a standardising, simplifying change for read,
>> simplifying programmatically authoring the config file as well. As you
>> know the mapping for the old config form to the new form, could we write a
>> migration tool, that would convert the old config files to the new format,
>> with errors and warnings if the content is such that the user would need to
>> manually fix up the file.
>> If there are minimal user fixups, we should consider automatically
>> migrating the config file to the new format,
>> Kind regards, David.
>>
>>
>>
>>
>> From: Chesnay Schepler
>> Date: Tuesday, 3 October 2023 at 11:17
>> To: dev@flink.apache.org
>> Subject: [EXTERNAL] Re: [Discuss] FLIP-366: Support standard YAML for
>> FLINK configuration
>> It is a unfortunate that we'll need a separate config file but the FLIP
>> does a good job justifying it.
>>
>> Personally I'd just name it "config.yaml"; I never quite understood why
>> there was a flink prefix to begin with, and the current proposal
>> ("flink-configuration.yaml") seems unnecessarily long.
>>
>> For the deprecation process we could consider logging a warning if the
>> old parser is used.
>>
>> In terms of scope, is the migration of existing e2e tests and the docker
>> image to the new parser part of the FLIP?
>>
>> On 22/09/2023 09:32, Jane Chan wrote:
>> > Hi, Junrui,
>> >
>> > Sorry for the late reply. The update looks good to me and thanks for
>> your
>> > effort!
>> >
>> >
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hi, Chesnay and David, sorry for the late reply.
Thanks for your feedback about this FLIP.
@Chesnay
>> Personally I'd just name it "config.yaml"
Thanks for your suggestions, I agree that "config.yaml" is a simpler and
more intuitive name. I have updated the FLIP accordingly.
>> In terms of scope, is the migration of existing e2e tests and the docker
image to the new parser part of the FLIP?
The migration of existing end-to-end tests and the docker image to the new
parser is indeed included in the FLIP. Taking into account the difficulty
of using shell scripts to modify configuration files that adhere to
standard YAML syntax, I propose utilizing the
org.apache.flink.runtime.util.bash.BashJavaUtils class as a unified utility
for modifying the configuration file.
In addition, considering that users may not only configure properties in
the configuration file, but also use dynamic parameters like '-D' or
configure the "with" clauses in FLINK SQL. When the default parser is
changed to the standard YAML parser, the configOption whose value is List
or Map types also needs to be modified accordingly. This migration cost is
relatively high because users have to modify their code or command lines.
To reduce migration costs while considering compatibility, I propose to add
compatibility support in FLINK-1.X versions to parse old format List and
Map values using the standard YAML parser. In case of an error while
parsing List or Map configurations with the standard YAML parser, a WARNING
log will be printed, and an attempt will be made to parse using the old
parser. And this support will be removed in the FLINK-2.0 version.
Do you think the aforementioned compatibility handling is necessary in
order to reduce migration costs? If you agree, I will also update this
support in the FLIP under the migration section.
@David
>> could we write a migration tool, that would convert the old config files
to the new format,
I agree that providing a more user-friendly guidance for the migration is
important. In the FLIP, we can include a user migration manual to help
users with the migration process. This migration manual will be presented
on the official FLINK website, making it easily accessible for users. And
the migrated configuration files will adhere to standard YAML syntax, and
users can validate them using various third-party YAML tools.
Regarding the automated migration tool, I think it can be considered as a
follow-up task rather than a requirement for the initial version. We can
focus on implementing the migration manually first and then explore the
possibility of automating it in future updates.
Best,
Junrui
David Radley 于2023年10月3日周二 18:56写道：
> Hi,
> I agree this is a standardising, simplifying change for read, simplifying
> programmatically authoring the config file as well. As you know the
> mapping for the old config form to the new form, could we write a migration
> tool, that would convert the old config files to the new format, with
> errors and warnings if the content is such that the user would need to
> manually fix up the file.
> If there are minimal user fixups, we should consider automatically
> migrating the config file to the new format,
> Kind regards, David.
>
>
>
>
> From: Chesnay Schepler
> Date: Tuesday, 3 October 2023 at 11:17
> To: dev@flink.apache.org
> Subject: [EXTERNAL] Re: [Discuss] FLIP-366: Support standard YAML for
> FLINK configuration
> It is a unfortunate that we'll need a separate config file but the FLIP
> does a good job justifying it.
>
> Personally I'd just name it "config.yaml"; I never quite understood why
> there was a flink prefix to begin with, and the current proposal
> ("flink-configuration.yaml") seems unnecessarily long.
>
> For the deprecation process we could consider logging a warning if the
> old parser is used.
>
> In terms of scope, is the migration of existing e2e tests and the docker
> image to the new parser part of the FLIP?
>
> On 22/09/2023 09:32, Jane Chan wrote:
> > Hi, Junrui,
> >
> > Sorry for the late reply. The update looks good to me and thanks for your
> > effort!
> >
> > Best,
> > Jane
> >
> > On Fri, Sep 22, 2023 at 2:25 PM Yuxin Tan
> wrote:
> >
> >> Hi, Junrui
> >>
> >> +1 for the proposal.
> >> Thanks for your effort.
> >>
> >> Best,
> >> Yuxin
> >>
> >>
> >> Samrat Deb 于2023年9月22日周五 13:23写道：
> >>
> >>> Hello Junrui,
> >>>
> >>> +1 for the proposal.
> >>>
> >>>
> >>> Bests,
> >>> Samrat
> >>>
> >>> On Fri, Sep 22, 2023 at 10:18 AM Shammon FY wrote:
> >>>
> >>>>
RE: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hi,
I agree this is a standardising, simplifying change for read, simplifying
programmatically authoring the config file as well. As you know the mapping
for the old config form to the new form, could we write a migration tool, that
would convert the old config files to the new format, with errors and warnings
if the content is such that the user would need to manually fix up the file.
If there are minimal user fixups, we should consider automatically migrating
the config file to the new format,
Kind regards, David.
From: Chesnay Schepler
Date: Tuesday, 3 October 2023 at 11:17
To: dev@flink.apache.org
Subject: [EXTERNAL] Re: [Discuss] FLIP-366: Support standard YAML for FLINK
configuration
It is a unfortunate that we'll need a separate config file but the FLIP
does a good job justifying it.
Personally I'd just name it "config.yaml"; I never quite understood why
there was a flink prefix to begin with, and the current proposal
("flink-configuration.yaml") seems unnecessarily long.
For the deprecation process we could consider logging a warning if the
old parser is used.
In terms of scope, is the migration of existing e2e tests and the docker
image to the new parser part of the FLIP?
On 22/09/2023 09:32, Jane Chan wrote:
> Hi, Junrui,
>
> Sorry for the late reply. The update looks good to me and thanks for your
> effort!
>
> Best,
> Jane
>
> On Fri, Sep 22, 2023 at 2:25 PM Yuxin Tan wrote:
>
>> Hi, Junrui
>>
>> +1 for the proposal.
>> Thanks for your effort.
>>
>> Best,
>> Yuxin
>>
>>
>> Samrat Deb 于2023年9月22日周五 13:23写道：
>>
>>> Hello Junrui,
>>>
>>> +1 for the proposal.
>>>
>>>
>>> Bests,
>>> Samrat
>>>
>>> On Fri, Sep 22, 2023 at 10:18 AM Shammon FY wrote:
>>>
>>>> +1 for the proposal, thanks for driving.
>>>>
>>>> Bet,
>>>> Shammon FY
>>>>
>>>> On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo
>> wrote:
>>>>> Thanks for driving this, +1 for the proposal.
>>>>>
>>>>> Best,
>>>>> Yangze Guo
>>>>>
>>>>>
>>>>> On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang <
>> wangdachui9...@gmail.com>
>>>>> wrote:
>>>>>> Hi Junrui,
>>>>>>
>>>>>> +1 for this proposal, thanks for driving.
>>>>>>
>>>>>> Best,
>>>>>> Lijie
>>>>>>
>>>>>> ConradJam 于2023年9月22日周五 10:07写道：
>>>>>>
>>>>>>> +1 Support for standard YAML format facilitates specification
>>>>>>>
>>>>>>> Jing Ge 于2023年9月22日周五 02:23写道：
>>>>>>>
>>>>>>>> Hi Junrui,
>>>>>>>>
>>>>>>>> +1 for following the standard. Thanks for your effort!
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Jing
>>>>>>>>
>>>>>>>> On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee <
>> jrlee@gmail.com>
>>>>> wrote:
>>>>>>>>> Hi Jane,
>>>>>>>>>
>>>>>>>>> Thank you for your valuable feedback and suggestions.
>>>>>>>>> I agree with your point about differentiating between
>>>>>>> "flink-config.yaml"
>>>>>>>>> and "flink-conf.yaml" to determine the standard syntax at a
>>>> glance.
>>>>>>>>> While I understand your suggestion of using
>>>>> "flink-conf-default.yaml"
>>>>>>> to
>>>>>>>>> represent the default YAML file for Flink 1.x, I have been
>>>>> considering
>>>>>>>>> the option of using "flink-configuration.yaml" as the file
>> name
>>>>> for the
>>>>>>>>> new configuration file.
>>>>>>>>> This name "flink-configuration.yaml" provides a clear
>>> distinction
>>>>>>> between
>>>>>>>>> the new and old configuration files based on their names, and
>>> it
>>>>> does
>>>>>>> not
>>>>>>>>> introduce any additional semantics. Moreover, this name
>>>>>>>>> "flink-configuration.yaml" can continue to be used in future
>>
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
It is a unfortunate that we'll need a separate config file but the FLIP
does a good job justifying it.
Personally I'd just name it "config.yaml"; I never quite understood why
there was a flink prefix to begin with, and the current proposal
("flink-configuration.yaml") seems unnecessarily long.
For the deprecation process we could consider logging a warning if the
old parser is used.
In terms of scope, is the migration of existing e2e tests and the docker
image to the new parser part of the FLIP?
On 22/09/2023 09:32, Jane Chan wrote:
Hi, Junrui,
Sorry for the late reply. The update looks good to me and thanks for your
effort!
Best,
Jane
On Fri, Sep 22, 2023 at 2:25 PM Yuxin Tan wrote:
Hi, Junrui
+1 for the proposal.
Thanks for your effort.
Best,
Yuxin
Samrat Deb 于2023年9月22日周五 13:23写道：
Hello Junrui,
+1 for the proposal.
Bests,
Samrat
On Fri, Sep 22, 2023 at 10:18 AM Shammon FY wrote:
+1 for the proposal, thanks for driving.
Bet,
Shammon FY
On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo
wrote:
Thanks for driving this, +1 for the proposal.
Best,
Yangze Guo
On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang <
wangdachui9...@gmail.com>
wrote:
Hi Junrui,
+1 for this proposal, thanks for driving.
Best,
Lijie
ConradJam 于2023年9月22日周五 10:07写道：
+1 Support for standard YAML format facilitates specification
Jing Ge 于2023年9月22日周五 02:23写道：
Hi Junrui,
+1 for following the standard. Thanks for your effort!
Best regards,
Jing
On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee <
jrlee@gmail.com>
wrote:
Hi Jane,
Thank you for your valuable feedback and suggestions.
I agree with your point about differentiating between
"flink-config.yaml"
and "flink-conf.yaml" to determine the standard syntax at a
glance.
While I understand your suggestion of using
"flink-conf-default.yaml"
to
represent the default YAML file for Flink 1.x, I have been
considering
the option of using "flink-configuration.yaml" as the file
name
for the
new configuration file.
This name "flink-configuration.yaml" provides a clear
distinction
between
the new and old configuration files based on their names, and
it
does
not
introduce any additional semantics. Moreover, this name
"flink-configuration.yaml" can continue to be used in future
versions
FLINK-2.0.
WDYT? If we can reach a consensus on this, I will update the
FLIP
documentation
accordingly.
Best regards,
Junrui
Jane Chan 于2023年9月20日周三 23:38写道：
Hi Junrui,
Thanks for driving this FLIP. +1 for adoption of the
standard
YAML
syntax.
I just have one minor suggestion. It's a little bit
challenging
to
differentiate between `flink-config.yaml` and
`flink-conf.yaml`
to
determine which one uses the standard syntax at a glance.
How
about
using `flink-conf-default.yaml` to represent the default
yaml
file
for
Flink 1.x?
Best,
Jane
On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee <
jrlee@gmail.com
wrote:
Hi devs,
I would like to start a discussion about FLIP-366:
Support standard YAML for FLINK configuration[1]
The current flink-conf.yaml parser in FLINK is not a
standard
YAML
parser,
which has some shortcomings.
Firstly, it does not support nested structure
configuration
items
and
only
supports key-value pairs, resulting in poor readability.
Secondly,
if
the
value is a collection type, such as a List or Map, users
are
required
to
write the value in a FLINK-specific pattern, which is
inconvenient
to
use.
Additionally, the parser of FLINK has some differences in
syntax
compared
to the standard YAML parser, such as the syntax for
parsing
comments
and
null values. These inconsistencies can cause confusion
for
users,
as
seen
in FLINK-15358 and FLINK-32740.
By supporting standard YAML, these issues can be
resolved,
and
users
can
create a Flink configuration file using third-party tools
and
leverage
some advanced YAML features. Therefore, we propose to
support
standard
YAML for FLINK configuration.
You can find more details in the FLIP-366[1]. Looking
forward
to
your
feedback.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration
Best,
Junrui
--
Best
ConradJam
回复: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hi Junrui, Thanks for driving this, +1 for it Best, Zhanghao Chen 发件人: Junrui Lee 发送时间: 2023年9月20日 11:06 收件人: dev@flink.apache.org 主题: [Discuss] FLIP-366: Support standard YAML for FLINK configuration Hi devs, I would like to start a discussion about FLIP-366: Support standard YAML for FLINK configuration[1] The current flink-conf.yaml parser in FLINK is not a standard YAML parser, which has some shortcomings. Firstly, it does not support nested structure configuration items and only supports key-value pairs, resulting in poor readability. Secondly, if the value is a collection type, such as a List or Map, users are required to write the value in a FLINK-specific pattern, which is inconvenient to use. Additionally, the parser of FLINK has some differences in syntax compared to the standard YAML parser, such as the syntax for parsing comments and null values. These inconsistencies can cause confusion for users, as seen in FLINK-15358 and FLINK-32740. By supporting standard YAML, these issues can be resolved, and users can create a Flink configuration file using third-party tools and leverage some advanced YAML features. Therefore, we propose to support standard YAML for FLINK configuration. You can find more details in the FLIP-366[1]. Looking forward to your feedback. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration Best, Junrui
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hi, Junrui, Sorry for the late reply. The update looks good to me and thanks for your effort! Best, Jane On Fri, Sep 22, 2023 at 2:25 PM Yuxin Tan wrote: > Hi, Junrui > > +1 for the proposal. > Thanks for your effort. > > Best, > Yuxin > > > Samrat Deb 于2023年9月22日周五 13:23写道： > > > Hello Junrui, > > > > +1 for the proposal. > > > > > > Bests, > > Samrat > > > > On Fri, Sep 22, 2023 at 10:18 AM Shammon FY wrote: > > > > > +1 for the proposal, thanks for driving. > > > > > > Bet, > > > Shammon FY > > > > > > On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo > wrote: > > > > > > > Thanks for driving this, +1 for the proposal. > > > > > > > > Best, > > > > Yangze Guo > > > > > > > > > > > > On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang < > wangdachui9...@gmail.com> > > > > wrote: > > > > > > > > > > Hi Junrui, > > > > > > > > > > +1 for this proposal, thanks for driving. > > > > > > > > > > Best, > > > > > Lijie > > > > > > > > > > ConradJam 于2023年9月22日周五 10:07写道： > > > > > > > > > > > +1 Support for standard YAML format facilitates specification > > > > > > > > > > > > Jing Ge 于2023年9月22日周五 02:23写道： > > > > > > > > > > > > > Hi Junrui, > > > > > > > > > > > > > > +1 for following the standard. Thanks for your effort! > > > > > > > > > > > > > > Best regards, > > > > > > > Jing > > > > > > > > > > > > > > On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee < > jrlee@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > Hi Jane, > > > > > > > > > > > > > > > > Thank you for your valuable feedback and suggestions. > > > > > > > > I agree with your point about differentiating between > > > > > > "flink-config.yaml" > > > > > > > > and "flink-conf.yaml" to determine the standard syntax at a > > > glance. > > > > > > > > > > > > > > > > While I understand your suggestion of using > > > > "flink-conf-default.yaml" > > > > > > to > > > > > > > > represent the default YAML file for Flink 1.x, I have been > > > > considering > > > > > > > > the option of using "flink-configuration.yaml" as the file > name > > > > for the > > > > > > > > new configuration file. > > > > > > > > This name "flink-configuration.yaml" provides a clear > > distinction > > > > > > between > > > > > > > > the new and old configuration files based on their names, and > > it > > > > does > > > > > > not > > > > > > > > introduce any additional semantics. Moreover, this name > > > > > > > > "flink-configuration.yaml" can continue to be used in future > > > > versions > > > > > > > > FLINK-2.0. > > > > > > > > > > > > > > > > WDYT? If we can reach a consensus on this, I will update the > > FLIP > > > > > > > > documentation > > > > > > > > accordingly. > > > > > > > > > > > > > > > > Best regards, > > > > > > > > Junrui > > > > > > > > > > > > > > > > Jane Chan 于2023年9月20日周三 23:38写道： > > > > > > > > > > > > > > > > > Hi Junrui, > > > > > > > > > > > > > > > > > > Thanks for driving this FLIP. +1 for adoption of the > standard > > > > YAML > > > > > > > > syntax. > > > > > > > > > I just have one minor suggestion. It's a little bit > > challenging > > > > to > > > > > > > > > differentiate between `flink-config.yaml` and > > `flink-conf.yaml` > > > > to > > > > > > > > > determine which one uses the standard syntax at a glance. > How > > > > about > > > > > > > > > using `flink-conf-default.yaml` to represent the default > yaml > > > > file > > > > > > for > > > > > > > > > Flink 1.x? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Jane > > > > > > > > > > > > > > > > > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee < > > > jrlee@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > > > > > I would like to start a discussion about FLIP-366: > > > > > > > > > > Support standard YAML for FLINK configuration[1] > > > > > > > > > > > > > > > > > > > > The current flink-conf.yaml parser in FLINK is not a > > standard > > > > YAML > > > > > > > > > parser, > > > > > > > > > > which has some shortcomings. > > > > > > > > > > Firstly, it does not support nested structure > configuration > > > > items > > > > > > and > > > > > > > > > only > > > > > > > > > > supports key-value pairs, resulting in poor readability. > > > > Secondly, > > > > > > if > > > > > > > > the > > > > > > > > > > value is a collection type, such as a List or Map, users > > are > > > > > > required > > > > > > > > to > > > > > > > > > > write the value in a FLINK-specific pattern, which is > > > > inconvenient > > > > > > to > > > > > > > > > use. > > > > > > > > > > Additionally, the parser of FLINK has some differences in > > > > syntax > > > > > > > > compared > > > > > > > > > > to the standard YAML parser, such as the syntax for > parsing > > > > > > comments > > > > > > > > and > > > > > > > > > > null values. These inconsistencies can cause confusion > for > > > > users, > > > > > > as > > > > > > > > seen > > > > > > > > > > in FLINK-15358 and FLINK-32740. > > > > > > > > > > > > > > > > > > > > By
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hi, Junrui +1 for the proposal. Thanks for your effort. Best, Yuxin Samrat Deb 于2023年9月22日周五 13:23写道： > Hello Junrui, > > +1 for the proposal. > > > Bests, > Samrat > > On Fri, Sep 22, 2023 at 10:18 AM Shammon FY wrote: > > > +1 for the proposal, thanks for driving. > > > > Bet, > > Shammon FY > > > > On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo wrote: > > > > > Thanks for driving this, +1 for the proposal. > > > > > > Best, > > > Yangze Guo > > > > > > > > > On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang > > > wrote: > > > > > > > > Hi Junrui, > > > > > > > > +1 for this proposal, thanks for driving. > > > > > > > > Best, > > > > Lijie > > > > > > > > ConradJam 于2023年9月22日周五 10:07写道： > > > > > > > > > +1 Support for standard YAML format facilitates specification > > > > > > > > > > Jing Ge 于2023年9月22日周五 02:23写道： > > > > > > > > > > > Hi Junrui, > > > > > > > > > > > > +1 for following the standard. Thanks for your effort! > > > > > > > > > > > > Best regards, > > > > > > Jing > > > > > > > > > > > > On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee > > > wrote: > > > > > > > > > > > > > Hi Jane, > > > > > > > > > > > > > > Thank you for your valuable feedback and suggestions. > > > > > > > I agree with your point about differentiating between > > > > > "flink-config.yaml" > > > > > > > and "flink-conf.yaml" to determine the standard syntax at a > > glance. > > > > > > > > > > > > > > While I understand your suggestion of using > > > "flink-conf-default.yaml" > > > > > to > > > > > > > represent the default YAML file for Flink 1.x, I have been > > > considering > > > > > > > the option of using "flink-configuration.yaml" as the file name > > > for the > > > > > > > new configuration file. > > > > > > > This name "flink-configuration.yaml" provides a clear > distinction > > > > > between > > > > > > > the new and old configuration files based on their names, and > it > > > does > > > > > not > > > > > > > introduce any additional semantics. Moreover, this name > > > > > > > "flink-configuration.yaml" can continue to be used in future > > > versions > > > > > > > FLINK-2.0. > > > > > > > > > > > > > > WDYT? If we can reach a consensus on this, I will update the > FLIP > > > > > > > documentation > > > > > > > accordingly. > > > > > > > > > > > > > > Best regards, > > > > > > > Junrui > > > > > > > > > > > > > > Jane Chan 于2023年9月20日周三 23:38写道： > > > > > > > > > > > > > > > Hi Junrui, > > > > > > > > > > > > > > > > Thanks for driving this FLIP. +1 for adoption of the standard > > > YAML > > > > > > > syntax. > > > > > > > > I just have one minor suggestion. It's a little bit > challenging > > > to > > > > > > > > differentiate between `flink-config.yaml` and > `flink-conf.yaml` > > > to > > > > > > > > determine which one uses the standard syntax at a glance. How > > > about > > > > > > > > using `flink-conf-default.yaml` to represent the default yaml > > > file > > > > > for > > > > > > > > Flink 1.x? > > > > > > > > > > > > > > > > Best, > > > > > > > > Jane > > > > > > > > > > > > > > > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee < > > jrlee@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > > > I would like to start a discussion about FLIP-366: > > > > > > > > > Support standard YAML for FLINK configuration[1] > > > > > > > > > > > > > > > > > > The current flink-conf.yaml parser in FLINK is not a > standard > > > YAML > > > > > > > > parser, > > > > > > > > > which has some shortcomings. > > > > > > > > > Firstly, it does not support nested structure configuration > > > items > > > > > and > > > > > > > > only > > > > > > > > > supports key-value pairs, resulting in poor readability. > > > Secondly, > > > > > if > > > > > > > the > > > > > > > > > value is a collection type, such as a List or Map, users > are > > > > > required > > > > > > > to > > > > > > > > > write the value in a FLINK-specific pattern, which is > > > inconvenient > > > > > to > > > > > > > > use. > > > > > > > > > Additionally, the parser of FLINK has some differences in > > > syntax > > > > > > > compared > > > > > > > > > to the standard YAML parser, such as the syntax for parsing > > > > > comments > > > > > > > and > > > > > > > > > null values. These inconsistencies can cause confusion for > > > users, > > > > > as > > > > > > > seen > > > > > > > > > in FLINK-15358 and FLINK-32740. > > > > > > > > > > > > > > > > > > By supporting standard YAML, these issues can be resolved, > > and > > > > > users > > > > > > > can > > > > > > > > > create a Flink configuration file using third-party tools > and > > > > > > leverage > > > > > > > > > some advanced YAML features. Therefore, we propose to > support > > > > > > standard > > > > > > > > > YAML for FLINK configuration. > > > > > > > > > > > > > > > > > > You can find more details in the FLIP-366[1]. Looking > forward > > > to > > > > > your > > > > > > > > > feedback. > > > > > > > > > > > > > > > > > > [1] > >
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hello Junrui, +1 for the proposal. Bests, Samrat On Fri, Sep 22, 2023 at 10:18 AM Shammon FY wrote: > +1 for the proposal, thanks for driving. > > Bet, > Shammon FY > > On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo wrote: > > > Thanks for driving this, +1 for the proposal. > > > > Best, > > Yangze Guo > > > > > > On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang > > wrote: > > > > > > Hi Junrui, > > > > > > +1 for this proposal, thanks for driving. > > > > > > Best, > > > Lijie > > > > > > ConradJam 于2023年9月22日周五 10:07写道： > > > > > > > +1 Support for standard YAML format facilitates specification > > > > > > > > Jing Ge 于2023年9月22日周五 02:23写道： > > > > > > > > > Hi Junrui, > > > > > > > > > > +1 for following the standard. Thanks for your effort! > > > > > > > > > > Best regards, > > > > > Jing > > > > > > > > > > On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee > > wrote: > > > > > > > > > > > Hi Jane, > > > > > > > > > > > > Thank you for your valuable feedback and suggestions. > > > > > > I agree with your point about differentiating between > > > > "flink-config.yaml" > > > > > > and "flink-conf.yaml" to determine the standard syntax at a > glance. > > > > > > > > > > > > While I understand your suggestion of using > > "flink-conf-default.yaml" > > > > to > > > > > > represent the default YAML file for Flink 1.x, I have been > > considering > > > > > > the option of using "flink-configuration.yaml" as the file name > > for the > > > > > > new configuration file. > > > > > > This name "flink-configuration.yaml" provides a clear distinction > > > > between > > > > > > the new and old configuration files based on their names, and it > > does > > > > not > > > > > > introduce any additional semantics. Moreover, this name > > > > > > "flink-configuration.yaml" can continue to be used in future > > versions > > > > > > FLINK-2.0. > > > > > > > > > > > > WDYT? If we can reach a consensus on this, I will update the FLIP > > > > > > documentation > > > > > > accordingly. > > > > > > > > > > > > Best regards, > > > > > > Junrui > > > > > > > > > > > > Jane Chan 于2023年9月20日周三 23:38写道： > > > > > > > > > > > > > Hi Junrui, > > > > > > > > > > > > > > Thanks for driving this FLIP. +1 for adoption of the standard > > YAML > > > > > > syntax. > > > > > > > I just have one minor suggestion. It's a little bit challenging > > to > > > > > > > differentiate between `flink-config.yaml` and `flink-conf.yaml` > > to > > > > > > > determine which one uses the standard syntax at a glance. How > > about > > > > > > > using `flink-conf-default.yaml` to represent the default yaml > > file > > > > for > > > > > > > Flink 1.x? > > > > > > > > > > > > > > Best, > > > > > > > Jane > > > > > > > > > > > > > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee < > jrlee@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > I would like to start a discussion about FLIP-366: > > > > > > > > Support standard YAML for FLINK configuration[1] > > > > > > > > > > > > > > > > The current flink-conf.yaml parser in FLINK is not a standard > > YAML > > > > > > > parser, > > > > > > > > which has some shortcomings. > > > > > > > > Firstly, it does not support nested structure configuration > > items > > > > and > > > > > > > only > > > > > > > > supports key-value pairs, resulting in poor readability. > > Secondly, > > > > if > > > > > > the > > > > > > > > value is a collection type, such as a List or Map, users are > > > > required > > > > > > to > > > > > > > > write the value in a FLINK-specific pattern, which is > > inconvenient > > > > to > > > > > > > use. > > > > > > > > Additionally, the parser of FLINK has some differences in > > syntax > > > > > > compared > > > > > > > > to the standard YAML parser, such as the syntax for parsing > > > > comments > > > > > > and > > > > > > > > null values. These inconsistencies can cause confusion for > > users, > > > > as > > > > > > seen > > > > > > > > in FLINK-15358 and FLINK-32740. > > > > > > > > > > > > > > > > By supporting standard YAML, these issues can be resolved, > and > > > > users > > > > > > can > > > > > > > > create a Flink configuration file using third-party tools and > > > > > leverage > > > > > > > > some advanced YAML features. Therefore, we propose to support > > > > > standard > > > > > > > > YAML for FLINK configuration. > > > > > > > > > > > > > > > > You can find more details in the FLIP-366[1]. Looking forward > > to > > > > your > > > > > > > > feedback. > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration > > > > > > > > > > > > > > > > Best, > > > > > > > > Junrui > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Best > > > > > > > > ConradJam > > > > > > >
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
+1 for the proposal, thanks for driving. Bet, Shammon FY On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo wrote: > Thanks for driving this, +1 for the proposal. > > Best, > Yangze Guo > > > On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang > wrote: > > > > Hi Junrui, > > > > +1 for this proposal, thanks for driving. > > > > Best, > > Lijie > > > > ConradJam 于2023年9月22日周五 10:07写道： > > > > > +1 Support for standard YAML format facilitates specification > > > > > > Jing Ge 于2023年9月22日周五 02:23写道： > > > > > > > Hi Junrui, > > > > > > > > +1 for following the standard. Thanks for your effort! > > > > > > > > Best regards, > > > > Jing > > > > > > > > On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee > wrote: > > > > > > > > > Hi Jane, > > > > > > > > > > Thank you for your valuable feedback and suggestions. > > > > > I agree with your point about differentiating between > > > "flink-config.yaml" > > > > > and "flink-conf.yaml" to determine the standard syntax at a glance. > > > > > > > > > > While I understand your suggestion of using > "flink-conf-default.yaml" > > > to > > > > > represent the default YAML file for Flink 1.x, I have been > considering > > > > > the option of using "flink-configuration.yaml" as the file name > for the > > > > > new configuration file. > > > > > This name "flink-configuration.yaml" provides a clear distinction > > > between > > > > > the new and old configuration files based on their names, and it > does > > > not > > > > > introduce any additional semantics. Moreover, this name > > > > > "flink-configuration.yaml" can continue to be used in future > versions > > > > > FLINK-2.0. > > > > > > > > > > WDYT? If we can reach a consensus on this, I will update the FLIP > > > > > documentation > > > > > accordingly. > > > > > > > > > > Best regards, > > > > > Junrui > > > > > > > > > > Jane Chan 于2023年9月20日周三 23:38写道： > > > > > > > > > > > Hi Junrui, > > > > > > > > > > > > Thanks for driving this FLIP. +1 for adoption of the standard > YAML > > > > > syntax. > > > > > > I just have one minor suggestion. It's a little bit challenging > to > > > > > > differentiate between `flink-config.yaml` and `flink-conf.yaml` > to > > > > > > determine which one uses the standard syntax at a glance. How > about > > > > > > using `flink-conf-default.yaml` to represent the default yaml > file > > > for > > > > > > Flink 1.x? > > > > > > > > > > > > Best, > > > > > > Jane > > > > > > > > > > > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee > > > > > wrote: > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > I would like to start a discussion about FLIP-366: > > > > > > > Support standard YAML for FLINK configuration[1] > > > > > > > > > > > > > > The current flink-conf.yaml parser in FLINK is not a standard > YAML > > > > > > parser, > > > > > > > which has some shortcomings. > > > > > > > Firstly, it does not support nested structure configuration > items > > > and > > > > > > only > > > > > > > supports key-value pairs, resulting in poor readability. > Secondly, > > > if > > > > > the > > > > > > > value is a collection type, such as a List or Map, users are > > > required > > > > > to > > > > > > > write the value in a FLINK-specific pattern, which is > inconvenient > > > to > > > > > > use. > > > > > > > Additionally, the parser of FLINK has some differences in > syntax > > > > > compared > > > > > > > to the standard YAML parser, such as the syntax for parsing > > > comments > > > > > and > > > > > > > null values. These inconsistencies can cause confusion for > users, > > > as > > > > > seen > > > > > > > in FLINK-15358 and FLINK-32740. > > > > > > > > > > > > > > By supporting standard YAML, these issues can be resolved, and > > > users > > > > > can > > > > > > > create a Flink configuration file using third-party tools and > > > > leverage > > > > > > > some advanced YAML features. Therefore, we propose to support > > > > standard > > > > > > > YAML for FLINK configuration. > > > > > > > > > > > > > > You can find more details in the FLIP-366[1]. Looking forward > to > > > your > > > > > > > feedback. > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration > > > > > > > > > > > > > > Best, > > > > > > > Junrui > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best > > > > > > ConradJam > > > >
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Thanks for driving this, +1 for the proposal. Best, Yangze Guo On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang wrote: > > Hi Junrui, > > +1 for this proposal, thanks for driving. > > Best, > Lijie > > ConradJam 于2023年9月22日周五 10:07写道： > > > +1 Support for standard YAML format facilitates specification > > > > Jing Ge 于2023年9月22日周五 02:23写道： > > > > > Hi Junrui, > > > > > > +1 for following the standard. Thanks for your effort! > > > > > > Best regards, > > > Jing > > > > > > On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee wrote: > > > > > > > Hi Jane, > > > > > > > > Thank you for your valuable feedback and suggestions. > > > > I agree with your point about differentiating between > > "flink-config.yaml" > > > > and "flink-conf.yaml" to determine the standard syntax at a glance. > > > > > > > > While I understand your suggestion of using "flink-conf-default.yaml" > > to > > > > represent the default YAML file for Flink 1.x, I have been considering > > > > the option of using "flink-configuration.yaml" as the file name for the > > > > new configuration file. > > > > This name "flink-configuration.yaml" provides a clear distinction > > between > > > > the new and old configuration files based on their names, and it does > > not > > > > introduce any additional semantics. Moreover, this name > > > > "flink-configuration.yaml" can continue to be used in future versions > > > > FLINK-2.0. > > > > > > > > WDYT? If we can reach a consensus on this, I will update the FLIP > > > > documentation > > > > accordingly. > > > > > > > > Best regards, > > > > Junrui > > > > > > > > Jane Chan 于2023年9月20日周三 23:38写道： > > > > > > > > > Hi Junrui, > > > > > > > > > > Thanks for driving this FLIP. +1 for adoption of the standard YAML > > > > syntax. > > > > > I just have one minor suggestion. It's a little bit challenging to > > > > > differentiate between `flink-config.yaml` and `flink-conf.yaml` to > > > > > determine which one uses the standard syntax at a glance. How about > > > > > using `flink-conf-default.yaml` to represent the default yaml file > > for > > > > > Flink 1.x? > > > > > > > > > > Best, > > > > > Jane > > > > > > > > > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee > > > wrote: > > > > > > > > > > > Hi devs, > > > > > > > > > > > > I would like to start a discussion about FLIP-366: > > > > > > Support standard YAML for FLINK configuration[1] > > > > > > > > > > > > The current flink-conf.yaml parser in FLINK is not a standard YAML > > > > > parser, > > > > > > which has some shortcomings. > > > > > > Firstly, it does not support nested structure configuration items > > and > > > > > only > > > > > > supports key-value pairs, resulting in poor readability. Secondly, > > if > > > > the > > > > > > value is a collection type, such as a List or Map, users are > > required > > > > to > > > > > > write the value in a FLINK-specific pattern, which is inconvenient > > to > > > > > use. > > > > > > Additionally, the parser of FLINK has some differences in syntax > > > > compared > > > > > > to the standard YAML parser, such as the syntax for parsing > > comments > > > > and > > > > > > null values. These inconsistencies can cause confusion for users, > > as > > > > seen > > > > > > in FLINK-15358 and FLINK-32740. > > > > > > > > > > > > By supporting standard YAML, these issues can be resolved, and > > users > > > > can > > > > > > create a Flink configuration file using third-party tools and > > > leverage > > > > > > some advanced YAML features. Therefore, we propose to support > > > standard > > > > > > YAML for FLINK configuration. > > > > > > > > > > > > You can find more details in the FLIP-366[1]. Looking forward to > > your > > > > > > feedback. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration > > > > > > > > > > > > Best, > > > > > > Junrui > > > > > > > > > > > > > > > > > > > > > > > > -- > > Best > > > > ConradJam > >
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hi Junrui, +1 for this proposal, thanks for driving. Best, Lijie ConradJam 于2023年9月22日周五 10:07写道： > +1 Support for standard YAML format facilitates specification > > Jing Ge 于2023年9月22日周五 02:23写道： > > > Hi Junrui, > > > > +1 for following the standard. Thanks for your effort! > > > > Best regards, > > Jing > > > > On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee wrote: > > > > > Hi Jane, > > > > > > Thank you for your valuable feedback and suggestions. > > > I agree with your point about differentiating between > "flink-config.yaml" > > > and "flink-conf.yaml" to determine the standard syntax at a glance. > > > > > > While I understand your suggestion of using "flink-conf-default.yaml" > to > > > represent the default YAML file for Flink 1.x, I have been considering > > > the option of using "flink-configuration.yaml" as the file name for the > > > new configuration file. > > > This name "flink-configuration.yaml" provides a clear distinction > between > > > the new and old configuration files based on their names, and it does > not > > > introduce any additional semantics. Moreover, this name > > > "flink-configuration.yaml" can continue to be used in future versions > > > FLINK-2.0. > > > > > > WDYT? If we can reach a consensus on this, I will update the FLIP > > > documentation > > > accordingly. > > > > > > Best regards, > > > Junrui > > > > > > Jane Chan 于2023年9月20日周三 23:38写道： > > > > > > > Hi Junrui, > > > > > > > > Thanks for driving this FLIP. +1 for adoption of the standard YAML > > > syntax. > > > > I just have one minor suggestion. It's a little bit challenging to > > > > differentiate between `flink-config.yaml` and `flink-conf.yaml` to > > > > determine which one uses the standard syntax at a glance. How about > > > > using `flink-conf-default.yaml` to represent the default yaml file > for > > > > Flink 1.x? > > > > > > > > Best, > > > > Jane > > > > > > > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee > > wrote: > > > > > > > > > Hi devs, > > > > > > > > > > I would like to start a discussion about FLIP-366: > > > > > Support standard YAML for FLINK configuration[1] > > > > > > > > > > The current flink-conf.yaml parser in FLINK is not a standard YAML > > > > parser, > > > > > which has some shortcomings. > > > > > Firstly, it does not support nested structure configuration items > and > > > > only > > > > > supports key-value pairs, resulting in poor readability. Secondly, > if > > > the > > > > > value is a collection type, such as a List or Map, users are > required > > > to > > > > > write the value in a FLINK-specific pattern, which is inconvenient > to > > > > use. > > > > > Additionally, the parser of FLINK has some differences in syntax > > > compared > > > > > to the standard YAML parser, such as the syntax for parsing > comments > > > and > > > > > null values. These inconsistencies can cause confusion for users, > as > > > seen > > > > > in FLINK-15358 and FLINK-32740. > > > > > > > > > > By supporting standard YAML, these issues can be resolved, and > users > > > can > > > > > create a Flink configuration file using third-party tools and > > leverage > > > > > some advanced YAML features. Therefore, we propose to support > > standard > > > > > YAML for FLINK configuration. > > > > > > > > > > You can find more details in the FLIP-366[1]. Looking forward to > your > > > > > feedback. > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration > > > > > > > > > > Best, > > > > > Junrui > > > > > > > > > > > > > > > > > -- > Best > > ConradJam >
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
+1 Support for standard YAML format facilitates specification Jing Ge 于2023年9月22日周五 02:23写道： > Hi Junrui, > > +1 for following the standard. Thanks for your effort! > > Best regards, > Jing > > On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee wrote: > > > Hi Jane, > > > > Thank you for your valuable feedback and suggestions. > > I agree with your point about differentiating between "flink-config.yaml" > > and "flink-conf.yaml" to determine the standard syntax at a glance. > > > > While I understand your suggestion of using "flink-conf-default.yaml" to > > represent the default YAML file for Flink 1.x, I have been considering > > the option of using "flink-configuration.yaml" as the file name for the > > new configuration file. > > This name "flink-configuration.yaml" provides a clear distinction between > > the new and old configuration files based on their names, and it does not > > introduce any additional semantics. Moreover, this name > > "flink-configuration.yaml" can continue to be used in future versions > > FLINK-2.0. > > > > WDYT? If we can reach a consensus on this, I will update the FLIP > > documentation > > accordingly. > > > > Best regards, > > Junrui > > > > Jane Chan 于2023年9月20日周三 23:38写道： > > > > > Hi Junrui, > > > > > > Thanks for driving this FLIP. +1 for adoption of the standard YAML > > syntax. > > > I just have one minor suggestion. It's a little bit challenging to > > > differentiate between `flink-config.yaml` and `flink-conf.yaml` to > > > determine which one uses the standard syntax at a glance. How about > > > using `flink-conf-default.yaml` to represent the default yaml file for > > > Flink 1.x? > > > > > > Best, > > > Jane > > > > > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee > wrote: > > > > > > > Hi devs, > > > > > > > > I would like to start a discussion about FLIP-366: > > > > Support standard YAML for FLINK configuration[1] > > > > > > > > The current flink-conf.yaml parser in FLINK is not a standard YAML > > > parser, > > > > which has some shortcomings. > > > > Firstly, it does not support nested structure configuration items and > > > only > > > > supports key-value pairs, resulting in poor readability. Secondly, if > > the > > > > value is a collection type, such as a List or Map, users are required > > to > > > > write the value in a FLINK-specific pattern, which is inconvenient to > > > use. > > > > Additionally, the parser of FLINK has some differences in syntax > > compared > > > > to the standard YAML parser, such as the syntax for parsing comments > > and > > > > null values. These inconsistencies can cause confusion for users, as > > seen > > > > in FLINK-15358 and FLINK-32740. > > > > > > > > By supporting standard YAML, these issues can be resolved, and users > > can > > > > create a Flink configuration file using third-party tools and > leverage > > > > some advanced YAML features. Therefore, we propose to support > standard > > > > YAML for FLINK configuration. > > > > > > > > You can find more details in the FLIP-366[1]. Looking forward to your > > > > feedback. > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration > > > > > > > > Best, > > > > Junrui > > > > > > > > > > -- Best ConradJam
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hi Junrui, +1 for following the standard. Thanks for your effort! Best regards, Jing On Thu, Sep 21, 2023 at 5:09 AM Junrui Lee wrote: > Hi Jane, > > Thank you for your valuable feedback and suggestions. > I agree with your point about differentiating between "flink-config.yaml" > and "flink-conf.yaml" to determine the standard syntax at a glance. > > While I understand your suggestion of using "flink-conf-default.yaml" to > represent the default YAML file for Flink 1.x, I have been considering > the option of using "flink-configuration.yaml" as the file name for the > new configuration file. > This name "flink-configuration.yaml" provides a clear distinction between > the new and old configuration files based on their names, and it does not > introduce any additional semantics. Moreover, this name > "flink-configuration.yaml" can continue to be used in future versions > FLINK-2.0. > > WDYT? If we can reach a consensus on this, I will update the FLIP > documentation > accordingly. > > Best regards, > Junrui > > Jane Chan 于2023年9月20日周三 23:38写道： > > > Hi Junrui, > > > > Thanks for driving this FLIP. +1 for adoption of the standard YAML > syntax. > > I just have one minor suggestion. It's a little bit challenging to > > differentiate between `flink-config.yaml` and `flink-conf.yaml` to > > determine which one uses the standard syntax at a glance. How about > > using `flink-conf-default.yaml` to represent the default yaml file for > > Flink 1.x? > > > > Best, > > Jane > > > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee wrote: > > > > > Hi devs, > > > > > > I would like to start a discussion about FLIP-366: > > > Support standard YAML for FLINK configuration[1] > > > > > > The current flink-conf.yaml parser in FLINK is not a standard YAML > > parser, > > > which has some shortcomings. > > > Firstly, it does not support nested structure configuration items and > > only > > > supports key-value pairs, resulting in poor readability. Secondly, if > the > > > value is a collection type, such as a List or Map, users are required > to > > > write the value in a FLINK-specific pattern, which is inconvenient to > > use. > > > Additionally, the parser of FLINK has some differences in syntax > compared > > > to the standard YAML parser, such as the syntax for parsing comments > and > > > null values. These inconsistencies can cause confusion for users, as > seen > > > in FLINK-15358 and FLINK-32740. > > > > > > By supporting standard YAML, these issues can be resolved, and users > can > > > create a Flink configuration file using third-party tools and leverage > > > some advanced YAML features. Therefore, we propose to support standard > > > YAML for FLINK configuration. > > > > > > You can find more details in the FLIP-366[1]. Looking forward to your > > > feedback. > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration > > > > > > Best, > > > Junrui > > > > > >
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hi Jane, Thank you for your valuable feedback and suggestions. I agree with your point about differentiating between "flink-config.yaml" and "flink-conf.yaml" to determine the standard syntax at a glance. While I understand your suggestion of using "flink-conf-default.yaml" to represent the default YAML file for Flink 1.x, I have been considering the option of using "flink-configuration.yaml" as the file name for the new configuration file. This name "flink-configuration.yaml" provides a clear distinction between the new and old configuration files based on their names, and it does not introduce any additional semantics. Moreover, this name "flink-configuration.yaml" can continue to be used in future versions FLINK-2.0. WDYT? If we can reach a consensus on this, I will update the FLIP documentation accordingly. Best regards, Junrui Jane Chan 于2023年9月20日周三 23:38写道： > Hi Junrui, > > Thanks for driving this FLIP. +1 for adoption of the standard YAML syntax. > I just have one minor suggestion. It's a little bit challenging to > differentiate between `flink-config.yaml` and `flink-conf.yaml` to > determine which one uses the standard syntax at a glance. How about > using `flink-conf-default.yaml` to represent the default yaml file for > Flink 1.x? > > Best, > Jane > > On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee wrote: > > > Hi devs, > > > > I would like to start a discussion about FLIP-366: > > Support standard YAML for FLINK configuration[1] > > > > The current flink-conf.yaml parser in FLINK is not a standard YAML > parser, > > which has some shortcomings. > > Firstly, it does not support nested structure configuration items and > only > > supports key-value pairs, resulting in poor readability. Secondly, if the > > value is a collection type, such as a List or Map, users are required to > > write the value in a FLINK-specific pattern, which is inconvenient to > use. > > Additionally, the parser of FLINK has some differences in syntax compared > > to the standard YAML parser, such as the syntax for parsing comments and > > null values. These inconsistencies can cause confusion for users, as seen > > in FLINK-15358 and FLINK-32740. > > > > By supporting standard YAML, these issues can be resolved, and users can > > create a Flink configuration file using third-party tools and leverage > > some advanced YAML features. Therefore, we propose to support standard > > YAML for FLINK configuration. > > > > You can find more details in the FLIP-366[1]. Looking forward to your > > feedback. > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration > > > > Best, > > Junrui > > >
Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hi Junrui, Thanks for driving this FLIP. +1 for adoption of the standard YAML syntax. I just have one minor suggestion. It's a little bit challenging to differentiate between `flink-config.yaml` and `flink-conf.yaml` to determine which one uses the standard syntax at a glance. How about using `flink-conf-default.yaml` to represent the default yaml file for Flink 1.x? Best, Jane On Wed, Sep 20, 2023 at 11:06 AM Junrui Lee wrote: > Hi devs, > > I would like to start a discussion about FLIP-366: > Support standard YAML for FLINK configuration[1] > > The current flink-conf.yaml parser in FLINK is not a standard YAML parser, > which has some shortcomings. > Firstly, it does not support nested structure configuration items and only > supports key-value pairs, resulting in poor readability. Secondly, if the > value is a collection type, such as a List or Map, users are required to > write the value in a FLINK-specific pattern, which is inconvenient to use. > Additionally, the parser of FLINK has some differences in syntax compared > to the standard YAML parser, such as the syntax for parsing comments and > null values. These inconsistencies can cause confusion for users, as seen > in FLINK-15358 and FLINK-32740. > > By supporting standard YAML, these issues can be resolved, and users can > create a Flink configuration file using third-party tools and leverage > some advanced YAML features. Therefore, we propose to support standard > YAML for FLINK configuration. > > You can find more details in the FLIP-366[1]. Looking forward to your > feedback. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration > > Best, > Junrui >
[Discuss] FLIP-366: Support standard YAML for FLINK configuration
Hi devs, I would like to start a discussion about FLIP-366: Support standard YAML for FLINK configuration[1] The current flink-conf.yaml parser in FLINK is not a standard YAML parser, which has some shortcomings. Firstly, it does not support nested structure configuration items and only supports key-value pairs, resulting in poor readability. Secondly, if the value is a collection type, such as a List or Map, users are required to write the value in a FLINK-specific pattern, which is inconvenient to use. Additionally, the parser of FLINK has some differences in syntax compared to the standard YAML parser, such as the syntax for parsing comments and null values. These inconsistencies can cause confusion for users, as seen in FLINK-15358 and FLINK-32740. By supporting standard YAML, these issues can be resolved, and users can create a Flink configuration file using third-party tools and leverage some advanced YAML features. Therefore, we propose to support standard YAML for FLINK configuration. You can find more details in the FLIP-366[1]. Looking forward to your feedback. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration Best, Junrui
