I've been tweaking the startup scripts and how we test them to make sure that client-side configurations get through to the AM; this deals with the problem that the cluster's core-site.xml may set an option (e.g. hadoop.authorization.enabled) that does not work unless the AM has matching configurations.
1. the directory SLIDER_CONF_DIR is now uploaded to HDFS for localization into the AM container 2. this directory is placed on the classpath, ahead of the cluster classpath and slider's JARs 3. the file slider-client.xml is loaded by the AM -if it is present 4. if present, the file slider-server.xml is loaded by the AM —overwriting any of the client options if there is a conflict. 5. the log4j.properties file in SLIDER_CONF_DIR now takes precedence over anything in /etc/hadoop/conf 6. functional tests can be switched into using the .py scripts on linux, for fuller coverage of the launcher options. (SLIDER-533) (and yes, this did find a bug) This lets you control logging, and do some late binding options to the AM. Some thoughts 1. Should we allow the appconf.json to set properties too? I can see that being a good place to put options that you don't want specific to the machine doing the launching. 2. With a shared client and server log, how can the AM support rolling and aggregate logs? I think for the logs, we could 1. allow for an optional log4j-server.properties file, which, if set, would get picked up 2. support a log4j.properties file in HDFS, with the appconf pointing the AM at it. If present, it would override the client-side option. Again, I think #2 would be better, as it would be consistent -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
