+1 for the proposal. @Hong : I feel that the inverted override flag should be a cluster setting and not something the user can override at will. I fear that this might defeat the purpose of the feature itself. So I think we should add a config for this but not expose it on the rest api
Gyula On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong <lian...@amazon.co.uk.invalid> wrote: > +1 to this FLIP. > This is very useful for teams building a Flink platform to run jobs from > an external user > > +1 on Danny's comment on adding a configuration to allow inverted order of > overrides. > However, might it be better to include an "override" toggle in the REST > API itself? That way we can change the flink configuration override > behaviour without restarting the Flink cluster. This would make sense if we > are thinking of a Session cluster and deploying multiple Flink jobs to the > same cluster. > > Regards, > Hong > > > On 18/08/2022, 10:46, "Danny Cranmer" <dannycran...@apache.org> wrote: > > CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you can confirm the sender and > know the content is safe. > > > > +1 thanks for driving this FLIP. > > We actually have an internally forked equivalent of this, which is on > our > list to try to upstream. Your proposal would *almost* work "off the > shelf" > for us. We have an inverted order or priority: > - Rest API / Flink CLI > User Code > Cluster Config > > This is because our end users do not submit their jobs directly, our > service does it on their behalf. For our usecase we do not want to > allow > users to override certain values we set, since some are managed from > our > service configuration, example: checkpointing interval. > > How would you feel about including a cluster configuration to invert > the > order? This could be decoupled to a follow-up FLIP. > > Thanks, > Danny Cranmer > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <jam.gz...@gmail.com> > wrote: > > > This is a good suggestion, we can start this work after we finish the > > discussion and vote > > > > -- > > Best > > > > ConradJam > > > > > > <zhanghao.c...@outlook.com> 于2022年8月15日周一 21:01写道： > > > > > I've created a new ticket [FLINK-28973] Extending > /jars/:jarid/plan API > > to > > > support setting Flink configs - ASF JIRA (apache.org)< > > > https://issues.apache.org/jira/browse/FLINK-28973>. for the job > plan > > API. > > > Maybe we can create a new umbrella ticket for FLIP-256 and put > > FLINK-27060 > > > & FLINK-28973 as subtasks. WDYT? > > > > > > Best, > > > Zhanghao Chen > > > ________________________________ > > > From: zhengyu chen <jam.gz...@gmail.com> > > > Sent: Monday, August 15, 2022 18:06 > > > To: dev@flink.apache.org <dev@flink.apache.org> > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With > Flink Rest > > > Api > > > > > > Hi all, > > > > > > > > > We would like to start a discussion thread on FLIP-256 Support Job > > Dynamic > > > Parameter With Flink Rest Api > > > < > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api > > > > > > > [1] > > > > > > After the user submits the jar package, running a job through > restapi > > > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState, > > > savepointPath, programArg, entry-class, parallelism) parameters, > which is > > > obvious with the diversification of job parameters (eg Checkpoint > > address) > > > > > > This solves the problem that the user can pass in other parameters > when > > > submitting a job, avoiding the user to define these job parameters > in the > > > code, resulting in the need to repackage the job for each > modification > > > > > > There was some interest from users [3] from a meetup and the > mailing > > list. > > > Looking forward to comments and feedback, thanks! > > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api > > > > > > > > > [2] > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run > > > > > > [3] https://issues.apache.org/jira/browse/FLINK-27060 > > > > > > > > > > > > > > > -- > > > Best > > > > > > ConradJam > > > > > > >
