Hi Venkata krishnan, Thanks for starting a discussion on this topic. I completely agree with you on that, this behavior can create confusion and cause debugging sessions that could be spared with aligning how Flink parses external properties.
Personally, I find the Yarn props prefixing more intuitive, but I do not have strong opinions other than prefixing configs for external systems should follow the same semantics and behavior. It would make sense to align these in Flink 2.0 IMO, but I would be curious about other opinions. On Saturday, February 24th, 2024 at 07:36, Venkatakrishnan Sowrirajan <vsowr...@asu.edu> wrote: > > > Gentle ping on the ^^ question to surface this back up again. Any thoughts? > > Regards > Venkata krishnan > > > On Fri, Feb 16, 2024 at 7:32 PM Venkatakrishnan Sowrirajan vsowr...@asu.edu > > wrote: > > > Hi Flink devs, > > > > Flink supports overriding "hadoop" and "yarn" configuration. As part of > > the override mechanism, users have to prefix `hadoop` configs with " > > flink.hadoop." and the prefix will be removed, while with `yarn` configs > > users have to prefix it with "flink.yarn." but "flink." only is removed, > > not "flink.yarn.". > > > > Following is an example: > > > > 1. "Hadoop" config > > > > Hadoop config key = hadoop.tmp.dir => Flink config = > > flink.hadoop.hadoop.tmp.dir => Hadoop's configuration object would have > > hadoop.tmp.dir*.* > > > > 2. "YARN" config > > > > YARN config key = yarn.application.classpath => Flink config = > > flink.yarn.yarn.application.classpath => YARN's configuration object > > would have yarn.yarn.application.classpath*.* > > > > Although this is documented > > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/config/#flink-yarn-<key> > > properly, it feels unintuitive and it tripped me, took quite a while to > > understand why the above YARN configuration override was not working as > > expected. Is this something that should be fixed? The problem with fixing > > it is, it will become backwards incompatible. Therefore, can this be > > addressed as part of Flink-2.0? > > > > Any thoughts? > > > > Regards > > Venkata krishnan
