cameronlee314 opened a new pull request #1293: SAMZA-2472: Use runtime-framework-resources-pathing.jar to specify part of the runtime classpath and leverage it in IsolatingClassLoaderFactory URL: https://github.com/apache/samza/pull/1293 Symptom: When running in YARN and job coordinator dependency isolation is enabled, the framework infrastructure is unable to access the correct YARN configurations. This means that when it tries to interact with YARN, it is not using the correct configurations, and that results in issues. The observed issue was that the YARN client from the Samza side did not follow the RM HA config correctly, and so there was an issue in connecting to the RM. Cause: The YARN client reads configurations from certain resource files on the classpath. In `run-class.sh`, the `HADOOP_CONF_DIR` is directly used and added to the classpath. In job coordinator dependency isolation mode, the classpath in `run-class.sh` is not used for the framework infrastructure, since `IsolatingClassLoaderFactory` builds its own classpath. Therefore, the `HADOOP_CONF_DIR` is not on the framework infrastructure classpath. Changes: Refactored `run-class.sh` to create an additional "runtime framework resources" pathing JAR. The `HADOOP_CONF_DIR` is included in this new pathing JAR. The pathing JAR is added to the regular classpath in order to keep backwards compatibility, and it is also detected and used in `IsolatingClassLoaderFactory` when isolation is enabled. Tests: Deployed a job with isolation enabled in `samza-hello-samza` and verified using a log that `yarn.resourcemanager.scheduler.class` is properly set (the default of CapacityScheduler is overridden to FifoScheduler in the `samza-hello-samza` set-up). I also deployed the job without the fix and verified that the `yarn.resourcemanager.scheduler.class` property was not properly overridden. API changes: None Upgrade/usage instructions: None
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
