I need to configure the value of hadoop.service.shutdown.timeout due to some shutdown hooks triggering a timeout when our MR jobs stop:
2023-08-25 08:44:39,566 [WARN] [Thread-0] [org.apache.hadoop.util.ShutdownHookManager] - ShutdownHook '' timeout, java.util.concurrent.TimeoutException java.util.concurrent.TimeoutException at java.util.concurrent.FutureTask.get(FutureTask.java:205) at org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:124) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95) The problem is that the value takes effect only when the value is set in core-site.xml. For example, if we add it as a property to the env var YARN_OPTS (-D hadoop.service.shutdown.timeout ) or set it in the code (an instance of Configuration class which is passed to the ToolRunner), even though the value changes(we checked this through logging), the timeout is triggered based on the value from core-site.xml or default value if not specified in the file. Is this property configurable only from the core-site.xml config file ?
