GitHub user ep1804 opened a pull request:
https://github.com/apache/spark/pull/20004
[Spark 22818][SQL] csv escape of quote escape
## What changes were proposed in this pull request?
Escape of escape should be considered when using the UniVocity csv
encoding/decoding library.
Ref:
https://github.com/uniVocity/univocity-parsers#escaping-quote-escape-characters
One option is added for reading and writing CSV: `escapeQuoteEscaping`
## How was this patch tested?
Unit test added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ep1804/spark SPARK-22818
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20004.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20004
----
commit 9dc513cceb29f0989f9e0f00ec731c24b8139d76
Author: soonmok-kwon <[email protected]>
Date: 2017-03-06T11:30:20Z
SPARK-19384 csv escape of escape
commit 931c31a1aa13534866e494106a7c180c39e4fa7a
Author: soonmok-kwon <[email protected]>
Date: 2017-03-07T03:18:37Z
csv escapeEscape option
commit 7d832939857c75bb3f04af0d0cc434e481ff33b4
Author: soonmok-kwon <[email protected]>
Date: 2017-03-07T03:20:21Z
csv comment fix
commit d54be65e7db060f68ca80adabb755cf41f43307b
Author: soonmok-kwon <[email protected]>
Date: 2017-03-10T05:03:25Z
escapeQuoteEscaping
commit e961e9bd8e9a2d5b0a4eb136c9eb50fd9f0ac560
Author: soonmok-kwon <[email protected]>
Date: 2017-03-10T05:04:40Z
escapeQuoteEscaping again
commit bafa26835e88979ac8d1b3fe9c8af05c7a64c751
Author: soonmok-kwon <[email protected]>
Date: 2017-03-10T05:45:49Z
doc for escapeQuoteEscaping and python
commit 8d20da9f8d7d48f95153c61eb06a45636d04d99f
Author: soonmok-kwon <[email protected]>
Date: 2017-03-10T11:04:28Z
python add argument fix
commit 734a2fa43f690713cbf80308bdf3abcfb82f0f53
Author: soonmok-kwon <[email protected]>
Date: 2017-03-10T11:07:07Z
python add argument fix 2
commit 3d8d734e847d04518183d3479490e439cf501ab1
Author: soonmok-kwon <[email protected]>
Date: 2017-03-10T11:24:24Z
more test data
commit 008eaa4348fe17e2c0fbc5369aad6e75091e4d9c
Author: soonmok-kwon <[email protected]>
Date: 2017-03-12T09:18:16Z
csv unescapedQuoteHandling option
commit e844b1dec70c7f3aaaf6c58c9985ba88e2dd5a6f
Author: soonmok-kwon <[email protected]>
Date: 2017-03-12T11:12:54Z
csv escapeUnquotedValues
commit b3d86767a270d817277712d9f1391d45f9cc8bf3
Author: soonmok-kwon <[email protected]>
Date: 2017-03-12T11:16:27Z
code done
commit c67e7306297c00a1fc55c6205fc4d502dede539a
Author: soonmok-kwon <[email protected]>
Date: 2017-03-12T11:59:39Z
escapeUnquotedValues docs
commit 73a10b4342321be73a535dacc6db5e2233f0e0bb
Author: soonmok-kwon <[email protected]>
Date: 2017-03-15T01:10:42Z
resolve conflicts
commit 743f0087af42a2e24f43ef8028222cc0dce97964
Author: soonmok-kwon <[email protected]>
Date: 2017-03-15T02:02:34Z
bugfix
commit c0e21674190181a2ceed45e84d56b6529ec3ed62
Author: soonmok-kwon <[email protected]>
Date: 2017-04-09T14:33:23Z
upstream merged
commit d1806d08daff16efb97b3e86294e26ac509ab5c8
Author: soonmok-kwon <[email protected]>
Date: 2017-12-17T10:35:37Z
merge upstream
commit 626dff7389285c15e361bebbd5b34e9bd1f073ab
Author: soonmok-kwon <[email protected]>
Date: 2017-12-17T11:31:47Z
escapeUnquotedValues option removed
commit 68fec23beac93fb307f5e91f23791c5965117cee
Author: soonmok-kwon <[email protected]>
Date: 2017-12-17T11:35:24Z
minor fix
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
