kl0u opened a new pull request #12170: URL: https://github.com/apache/flink/pull/12170
This PR builds on top of https://github.com/apache/flink/pull/12143 and adds the functionality described in the JIRA. It makes mandatory for users of the `yarn.provided.lib.dirs` option to also include the `lib/`, `plugin/` and `flink-dist jar`. If these are not included among the shared resources, Flink will throw an exception and will inform the user to not use this feature, but ship all the dependencies of the application, as it was done previously. If these dirs are provided, then these are going to be used to execute the application and **NOT** what the user may have locally (e.g. different flink or log4j versions). The reason for this requirement is to avoid unpleasant surprises with classloading issues. Alternatively we could say to go case-by-case and say if the file to be shipped has the same name as one in the shared directories, then do not ship it and use the remote. But this way of doing things transparently to the user oftentimes leads to unpleasant surprises that are difficult to debug. ## Brief change log More of the changes are in the `YarnApplicationFileUploader`. ## Verifying this change Partially covered by tests in the `YarnITCase` and also verified manually by me and @wangyang0918 . ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (**yes** / no / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (**yes** / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
