GitHub user sarutak opened a pull request:
https://github.com/apache/spark/pull/2755
[SPARK-3900] ApplicationMaster's shutdown hook fails and
IllegalStateException is thrown.
ApplicationMaster registers a shutdown hook and it calls
ApplicationMaster#cleanupStagingDir.
cleanupStagingDir invokes FileSystem.get(yarnConf) and it invokes
FileSystem.getInternal. FileSystem.getInternal also registers shutdown hook.
In FileSystem of hadoop 0.23, the shutdown hook registration does not
consider whether shutdown is in progress or not (In 2.2, it's considered).
// 0.23
if (map.isEmpty() ) {
ShutdownHookManager.get().addShutdownHook(clientFinalizer,
SHUTDOWN_HOOK_PRIORITY);
}
// 2.2
if (map.isEmpty()
&& !ShutdownHookManager.get().isShutdownInProgress()) {
ShutdownHookManager.get().addShutdownHook(clientFinalizer,
SHUTDOWN_HOOK_PRIORITY);
}
Thus, in 0.23, another shutdown hook can be registered when
ApplicationMaster's shutdown hook run.
This issue cause IllegalStateException as follows.
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2306)
at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2278)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:316)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:162)
at
org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$cleanupStagingDir(ApplicationMaster.scala:307)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:118)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sarutak/spark SPARK-3900
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2755.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2755
----
commit 1fe76a1e3720d73623a357c6068646009192a7f7
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-01T09:13:06Z
Modified EOL character from LF to CRLF in *.cmd
commit 91fb0fdf1a3c4492853c0ecc9ffa9d723d369cd4
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-03T21:33:33Z
Fixed conflicts
commit c7f32d727b861f8723a8265f45dcee7bfd510388
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-05T03:46:20Z
Added dev/lint-windows-cmd
commit c2b5de1382b9b5c7d907637593744547240efc5f
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-05T03:53:31Z
Fixed mode of dev/lint-windows.cmd
commit 7ced891507a26058df18d45c1c53d1017670e51f
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-05T05:16:45Z
Changed lint-windows-cmd to print pass/error
commit 33376b181a361b04fea7d6f02565fa9914c43350
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-06T14:15:59Z
Merge branch 'master' of git://git.apache.org/spark into SPARK-3758
commit 9f12e607f9cd618b9242df1fa6906e8cf2d943b1
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-06T21:32:02Z
Merge branch 'master' of git://git.apache.org/spark into SPARK-3758
commit b0585da796aeb91957956f61d97fa98953d1c5e5
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-07T15:56:35Z
Merge branch 'master' of git://git.apache.org/spark into SPARK-3758
commit cfaa176a299b4c7b3f02e7dc8bf35627997021c5
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-07T16:25:41Z
Fixed style
commit fdb4d85f7e690d0a6b10c175bac13c2fce49b7be
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-07T23:38:51Z
Fixed style of dev/run-tests
commit 8c418e5e75ab2913247c96cef078e287d86646d7
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-07T23:39:00Z
Merge branch 'master' of git://git.apache.org/spark into SPARK-3758
commit 6b411d59f4f33e4cc3d2b733d5c2686a0b32a505
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-08T00:16:55Z
Improved dev/lint-windows-cmd
commit b7f4b1526190b14434c0d1034820406b04085b5a
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-08T08:29:50Z
Replaced 4-white-spaces with 2-white-spaces in github_jira_sync.py
Replaced tabs with white-spaces in github_jira_sync.py
commit d6f015e2908883f2d51a498fa23272e8912abf2e
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-08T08:30:32Z
Replaced tabs with white-spaces in sbin/spark-daemons.sh
commit f43b531e157d5ccbf17ff9ef11e9240dc35c5ea2
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-08T08:48:10Z
Removed extra white-spaces before EOL from some scripts
commit 44469abbb1e9dc693be771234885a405776134be
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-08T09:52:51Z
Replaced dev/lint-windows-cmd with dev/lint-scripts
Added some style rules to lint-scripts
commit cf95f5b7e6bb2d675aab68d84d0ab74f4ea04ac2
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-08T16:06:44Z
Added .gitattributes
commit cbd8dc4c487218fe17383bf88ceba1db6f233771
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-08T16:50:34Z
Minor modifiation of dev/lint-scripts
commit 504e5f3334cb6868c0c61f170d93178a675410b9
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-08T17:45:10Z
Added an entry ".gitattributes" to .rat-excludes
Normalize EOL in some files
commit ef9beec901a5abdfeeac0ab8d7e5c62190d7a795
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-08T17:57:04Z
Fixed broken code
commit b759f290ffbc330df5dd98536ba54d682a7a3d20
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-08T18:07:01Z
Fixed style
commit d825d6b61ce21b50c5f49ea91827399527a7da62
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-08T18:29:05Z
Modified error handling of lint-scripts
commit 3151f593734152fb582a63c0ae67520a76aaa1a4
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-10T15:58:28Z
Merge branch 'master' of git://git.apache.org/spark into origin/master
commit 240418816bbe840c42f911ab1d68f043f521553e
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-10T16:17:38Z
Fixed IllegalStateException caused by shutdown hook registration in another
shutdown hook
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
