GitHub user wyaron opened a pull request:
https://github.com/apache/spark/pull/10456
add default RDD name for one created via sc.textFile
The feature was first added at commit:
7b877b27053bfb7092e250e01a3b887e1b50a109 but was later removed (probably by
mistake) at commit: fc8b58195afa67fbb75b4c8303e022f703cbf007.
This change sets the default path of RDDs created via sc.textFile(...) to
the path argument.
Here is the symptom:
* Using spark-1.5.2-bin-hadoop2.6:
scala> sc.textFile("/home/root/.bashrc").name
res5: String = null
scala> sc.binaryFiles("/home/root/.bashrc").name
res6: String = /home/root/.bashrc
* while using Spark 1.3.1:
scala> sc.textFile("/home/root/.bashrc").name
res0: String = /home/root/.bashrc
scala> sc.binaryFiles("/home/root/.bashrc").name
res1: String = /home/root/.bashrc
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wyaron/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10456.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10456
----
commit 86efdade980f63cce9f87696716c5d0c74eef245
Author: yaron <[email protected]>
Date: 2015-12-23T19:32:57Z
add default RDD name for one create via sc.textFile
The feature was first added at commit:
7b877b27053bfb7092e250e01a3b887e1b50a109 but was later removed (probably by
mistake) at at commit: fc8b58195afa67fbb75b4c8303e022f703cbf007.
here is the symptom:
using spark-1.5.2-bin-hadoop2.6 I get:
=================================
scala> sc.textFile("/home/root/.bashrc").name
res5: String = null
scala> sc.binaryFiles("/home/root/.bashrc").name
res6: String = /home/root/.bashrc
while using Spark 1.3.1:
=================================
scala> sc.textFile("/home/root/.bashrc").name
res0: String = /home/root/.bashrc
scala> sc.binaryFiles("/home/root/.bashrc").name
res1: String = /home/root/.bashrc
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
