Thomas Graves created SPARK-14963:
-------------------------------------
Summary: YarnShuffleService should use YARN getRecoveryPath() for
leveldb location
Key: SPARK-14963
URL: https://issues.apache.org/jira/browse/SPARK-14963
Project: Spark
Issue Type: Bug
Components: Shuffle, YARN
Affects Versions: 1.6.1
Reporter: Thomas Graves
The YarnShuffleService, currently just picks a directly in the yarn local dirs
to store the leveldb file. YARN added an interface in hadoop 2.5
getRecoverPath() to get the location where it should be storing this.
We should change to use getRecoveryPath(). This does mean we will have to use
reflection or similar to check for its existence though since it doesn't exist
before hadoop 2.5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
