GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/22654
[SPARK-25660][SQL] Fix for the backward slash as CSV fields delimiter
## What changes were proposed in this pull request?
The PR addresses the exception raised on accessing chars out of delimiter
string. In particular, the backward slash `\` as the CSV fields delimiter
causes the following exception on reading `abc\1`:
```Scala
String index out of range: 1
java.lang.StringIndexOutOfBoundsException: String index out of range: 1
at java.lang.String.charAt(String.java:658)
```
because `str.charAt(1)` tries to access a char out of `str` in
`CSVUtils.toChar`
## How was this patch tested?
Added tests for empty string and string containing the backward slash to
`CSVUtilsSuite`. Besides of that I added an end-to-end test to check how the
backward slash is handled in reading CSV string with it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 csv-slash-delim
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22654.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22654
----
commit dd16ca349dd20626b6a8ae00f6dd445c269104b8
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-10-06T09:12:33Z
Test for backward slash as the delimiter
commit 7bf453a5a5ddedda4319d114bbbb6b8c125fa0d4
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-10-06T09:55:15Z
Bug fix + tests
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
