GitHub user jurriaan opened a pull request:
https://github.com/apache/spark/pull/13104
Fix reading of partitioned format=text datasets
I was using this in Spark 1.6.1 but it broke in Spark 2.0.0.
It would be logical if you could also write those,
but not entirely sure how to solve this with the new DataSet implementation.
Also it doesn't work using `sqlContext.read.text`, since that method
returns a `DataSet[String]`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jurriaan/spark fix-partitioned-text-reads
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13104.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13104
----
commit 4b6ad2b8586d8740041792b7d88f0e1666dd46a3
Author: Jurriaan Pruis <[email protected]>
Date: 2016-05-13T18:52:21Z
Fix reading of partitioned format=text datasets
I was using this in Spark 1.6.1 but it broke in Spark 2.0.0
It would be logical if you could also write those,
but not entirely sure how to solve this with the new DataSet implementation.
Also it doesn't work using `sqlContext.read.text`, since that method
returns a DataSet[String]
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
