GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/11016
[SPARK-13108][SQL] Validate ascii compatible encodings
https://issues.apache.org/jira/browse/SPARK-13108
CSV datasource currently does not support non-ascii compatible encodings.
I tested this in Max OS. I converted `cars_iso-8859-1.csv` into
`cars_utf-16.csv` as below:
```bash
iconv -f iso-8859-1 -t utf-16 < cars_iso-8859-1.csv > cars_utf-16.csv
```
and run the codes below:
```scala
val cars = "cars_utf-16.csv"
sqlContext.read
.format("csv")
.option("charset", "utf-16")
.option("delimiter", 'þ')
.load(cars)
.show()
```
This produces a wrong results below:
```
+----+-----+-----+--------------------+------+
|year| make|model| comment|blank�|
+----+-----+-----+--------------------+------+
|2012|Tesla| S| No comment| �|
| �| null| null| null| null|
|1997| Ford| E350|Go get one now th...| �|
|2015|Chevy|Volt�| null| null|
| �| null| null| null| null|
+----+-----+-----+--------------------+------+
```
(For more details, please check the Jira issue ticket).
So, this PR let this datasource checks the encodings to accept only
ascii-compatible encodings..
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark
SPARK-13108-non-ascii-encodings
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11016.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11016
----
commit d3cddb228f20b738b76c0fd74f6891dc9beaedb8
Author: hyukjinkwon <[email protected]>
Date: 2016-02-02T03:52:20Z
Validate ascii compatible encodings
commit 9f3735ca31f323fa460fd2f386fcef99725d890b
Author: hyukjinkwon <[email protected]>
Date: 2016-02-02T03:53:33Z
Remove a heading extra whitespace
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
