GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/6618
[SPARK-8062] Fix NullPointerException in
SparkHadoopUtil.getFileSystemThreadStatistics (branch-1.2)
This patch adds a regression test for an extremely rare bug where
`SparkHadoopUtil.getFileSystemThreadStatistics` would fail with a
`NullPointerException` if the Hadoop `FileSystem.statisticsTable` contained a
`Statistics` entry without a schema. I'm not sure exactly how Hadoop gets into
such a state, but this patch's regression test forces that state in order to
reproduce this bug.
The fix is to add additional null-checking. I debated adding an additional
try-catch block around this entire metrics code to just ignore exceptions and
keep going in the case of errors, but decided against that approach for now
because it seemed overly conservative and might mask other bugs. We can revisit
this in followup patches.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-8062-branch-1.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6618.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6618
----
commit b6430f04ee70f39ab1a6e65790f4eeb9c2c46e91
Author: Josh Rosen <[email protected]>
Date: 2015-06-03T20:41:37Z
Add failing regression test for SPARK-8062
commit 1d8d1250daf12e7ac5826f575b10e25cde55237c
Author: Josh Rosen <[email protected]>
Date: 2015-06-03T20:42:12Z
Fix SPARK-8062 with additional null checks
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
