Github user wyaron commented on the pull request:
https://github.com/apache/spark/pull/10456#issuecomment-167151653
Thanks for the quick CR !
Apparently, the new unit test has just failed for sc.wholeTextFiles(...) :-)
My guess is that it was broken in commit
297048ff730c022397e03afaf5ec9c91c811102a.
The change called setName on the WholeTextFileRDD and then called map on it
which yields a new RDD without the default name set (see code snippet below).
It seems that setName should be called AFTER the map is invoked, hence on
the *returned* RDD.
Note that wholeTextFiles provides the file name for each data item in the
RDD so the defaul path name is less useful... still, for consistency purposes
it should be fixed.
here is the original change:
new WholeTextFileRDD(
this,
classOf[WholeTextFileInputFormat],
- classOf[String],
- classOf[String],
+ classOf[Text],
+ classOf[Text],
updateConf,
- minPartitions).setName(path)
+ minPartitions).setName(path).map(record => (record._1.toString,
record._2.toString))
}
I have committed the proposed change which moves setName after the map.
I've tested locally and now the name field is valid.
does it make sense ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
