GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/13808
[SPARK-14480][SQL] Remove meaningless StringIteratorReader for CSV data
source.
## What changes were proposed in this pull request?
This PR removes meaningless `StringIteratorReader` for CSV data source.
In `CSVParser.scala`, there is an `Reader` wrapping `Iterator` but there
are two problems by this.
Firstly, it was actually not faster than processing line by line with
Iterator due to additional logics to wrap `Iterator` to `Reader`.
Secondly, this brought a bit of complexity because it needs additional
logics to allow every line to be read bytes by bytes. So, it was pretty
difficult to figure out issues about parsing, (eg. SPARK-14103).
A benchmark was performed manually and the results were below:
- Original codes with Reader wrapping Iterator
|End-to-end (ns) | Parse Time (ns) |
|-----------------------|------------------------|
|14116265034 |2008277960 |
- New codes with Iterator
|End-to-end (ns) | Parse Time (ns) |
|-----------------------|------------------------|
|13451699644 | 1549050564 |
For the details for the environment, dataset and methods, please refer the
JIRA ticket.
## How was this patch tested?
Existing tests should cover this.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-14480-small
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13808.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13808
----
commit 978a7286255fb16f3ad4270d59cb58fe18c967c9
Author: hyukjinkwon <[email protected]>
Date: 2016-06-21T11:39:09Z
Remove StringIteratorReader
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
