I don't think this is a good idea because it breaks the jar run api, this
is just a job submit parameter, now a non-job parameter
(overwriteFlinkConfiguration) is introduced. flinkConfiguration is just an
extra extension, users can keep their submit job by original way
Jing Ge <j...@ververica.com> 于2022年8月19日周五 07:00写道：
> +1 for the FLIP. I like this FLIP. It is very useful.
>
> I'd like to ask one more question:
>
> "
>
> - The internal parameters passed in by the user contains the external
> parameter. At this point, we can see that the user configures the same
> two
> parameters in the *external parameter* and *internal parameters*, in
> this case the Outer parameters is valid ("allowNonRestoredState": true)
>
> "
>
> Should we enable one more feature to let users decide which parameter will
> win if there is a conflict between the external parameter and internal
> parameter? By default, like you described, the external parameter will win.
> Beyond that, users could pass an additional parameter to tell Flink that
> all parameters in flinkConfiguration have higher priorities and will
> overwrite the value of the external parameter with the same name.
>
> This additional parameter e.g. "overwriteExternalParameter" (more feasible
> name could be reconsidered) could be defined within the flinkConfiguration
> like:
> {
> "entryClass":
> "org.apache.flink.streaming.examples.statemachine.StateMachineExample",
> "parallelism": "1",
> "programArgs": null,
> "savepointPath": "hdfs://flink/jobs/stateMachineExample/savepoints",
> "allowNonRestoredState": true,
> "flinkConfiguration": {
> *"overwriteExternalParameter" : true, // default value is false for
> backward compatibility*
> "allowNonRestoredState": false,
> "state.checkpoints.dir":
> "hdfs://flink/jobs/stateMachineExample/checkpoints"
> }
> }
> Or as a new external parameter e.g. "overwriteFlinkConfiguration"
>
> {
> "entryClass":
> "org.apache.flink.streaming.examples.statemachine.StateMachineExample",
> "parallelism": "1",
> "programArgs": null,
> "savepointPath": "hdfs://flink/jobs/stateMachineExample/savepoints",
> "allowNonRestoredState": true,
> *"overwriteFlinkConfiguration" : false, // default value is true for
> backward compatibility*
> "flinkConfiguration": {
> "allowNonRestoredState": false,
> "state.checkpoints.dir":
> "hdfs://flink/jobs/stateMachineExample/checkpoints"
> }
> }
> In this way, we will give even more flexibility to users. WDYT?
>
> Best regards,
> Jing
>
>
>
>
>
>
>
>
> On Thu, Aug 18, 2022 at 12:52 PM Teoh, Hong <lian...@amazon.co.uk.invalid>
> wrote:
>
> > @Gyula: I was thinking of the use case where the managed platform also
> > handles the job submission.
> >
> > Setup #1
> > External user : Creating jar
> > ------------------------------------
> > Managed platform : Submit job
> > Managed platform : Manage Flink cluster
> >
> > As compared to
> >
> > Setup #2
> > External user : Creating jar
> > External user : Submit job
> > ------------------------------------
> > Managed platform : Manage Flink cluster
> >
> >
> > But I take your point that a REST API toggle doesn't make sense at all in
> > Setup #2, so it would be more inclusive to include the cluster
> > configuration since that works for both setups
> >
> >
> > Regards,
> > Hong
> >
> >
> >
> > On 18/08/2022, 11:38, "Gyula Fóra" <gyula.f...@gmail.com> wrote:
> >
> > CAUTION: This email originated from outside of the organization. Do
> > not click links or open attachments unless you can confirm the sender and
> > know the content is safe.
> >
> >
> >
> > +1 for the proposal.
> >
> > @Hong : I feel that the inverted override flag should be a cluster
> > setting
> > and not something the user can override at will. I fear that this
> might
> > defeat the purpose of the feature itself.
> > So I think we should add a config for this but not expose it on the
> > rest api
> >
> > Gyula
> >
> > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
> > <lian...@amazon.co.uk.invalid>
> > wrote:
> >
> > > +1 to this FLIP.
> > > This is very useful for teams building a Flink platform to run jobs
> > from
> > > an external user
> > >
> > > +1 on Danny's comment on adding a configuration to allow inverted
> > order of
> > > overrides.
> > > However, might it be better to include an "override" toggle in the
> > REST
> > > API itself? That way we can change the flink configuration override
> > > behaviour without restarting the Flink cluster. This would make
> > sense if we
> > > are thinking of a Session cluster and deploying multiple Flink jobs
> > to the
> > > same cluster.
> > >
> > > Regards,
> > > Hong
> > >
> > >
> > > On 18/08/2022, 10:46, "Danny Cranmer" <dannycran...@apache.org>
> > wrote:
> > >
> > > CAUTION: This email originated from outside of the
> organization.
> > Do
> > > not click links or open attachments unless you can confirm the
> > sender and
> > > know the content is safe.
> > >
> > >
> > >
> > > +1 thanks for driving this FLIP.
> > >
> > > We actually have an internally forked equivalent of this, which
> > is on
> > > our
> > > list to try to upstream. Your proposal would *almost* work "off
> > the
> > > shelf"
> > > for us. We have an inverted order or priority:
> > > - Rest API / Flink CLI > User Code > Cluster Config
> > >
> > > This is because our end users do not submit their jobs
> directly,
> > our
> > > service does it on their behalf. For our usecase we do not want
> > to
> > > allow
> > > users to override certain values we set, since some are managed
> > from
> > > our
> > > service configuration, example: checkpointing interval.
> > >
> > > How would you feel about including a cluster configuration to
> > invert
> > > the
> > > order? This could be decoupled to a follow-up FLIP.
> > >
> > > Thanks,
> > > Danny Cranmer
> > >
> > >
> > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <
> > jam.gz...@gmail.com>
> > > wrote:
> > >
> > > > This is a good suggestion, we can start this work after we
> > finish the
> > > > discussion and vote
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > > >
> > > > <zhanghao.c...@outlook.com> 于2022年8月15日周一 21:01写道：
> > > >
> > > > > I've created a new ticket [FLINK-28973] Extending
> > > /jars/:jarid/plan API
> > > > to
> > > > > support setting Flink configs - ASF JIRA (apache.org)<
> > > > > https://issues.apache.org/jira/browse/FLINK-28973>. for
> the
> > job
> > > plan
> > > > API.
> > > > > Maybe we can create a new umbrella ticket for FLIP-256 and
> > put
> > > > FLINK-27060
> > > > > & FLINK-28973 as subtasks. WDYT?
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > ________________________________
> > > > > From: zhengyu chen <jam.gz...@gmail.com>
> > > > > Sent: Monday, August 15, 2022 18:06
> > > > > To: dev@flink.apache.org <dev@flink.apache.org>
> > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter
> > With
> > > Flink Rest
> > > > > Api
> > > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > We would like to start a discussion thread on FLIP-256
> > Support Job
> > > > Dynamic
> > > > > Parameter With Flink Rest Api
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > >
> > > > > [1]
> > > > >
> > > > > After the user submits the jar package, running a job
> through
> > > restapi
> > > > > (/jars/:jarid/run) [2] can only pass in
> > (allowNonRestoredState,
> > > > > savepointPath, programArg, entry-class, parallelism)
> > parameters,
> > > which is
> > > > > obvious with the diversification of job parameters (eg
> > Checkpoint
> > > > address)
> > > > >
> > > > > This solves the problem that the user can pass in other
> > parameters
> > > when
> > > > > submitting a job, avoiding the user to define these job
> > parameters
> > > in the
> > > > > code, resulting in the need to repackage the job for each
> > > modification
> > > > >
> > > > > There was some interest from users [3] from a meetup and
> the
> > > mailing
> > > > list.
> > > > > Looking forward to comments and feedback, thanks!
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > >
> > > > >
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > > >
> > > > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > >
> > >
> > >
> >
> >
>
--
Best
ConradJam
