[
https://issues.apache.org/jira/browse/ARROW-16682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Buhrmann updated ARROW-16682:
------------------------------------
Description:
When trying to read arbitrary CSV files, it is not possible to infer/guess the
correct encoding 100% of the time. The Arrow CSV reader will currently fail if
any byte cannot be decoded given the specified encoding (see example below).
With pandas.read_csv(), I can often get a result that is 99.9% correct by
passing it a text stream decoded in Python with
[errors="replace"|https://docs.python.org/3/library/codecs.html#error-handlers]
(or "ignore" etc.).
Pyarrow's csv.read_csv() on the other hand neither accepts an already decoded
text stream (TypeError: binary file expected, got text file), nor a parameter
to configure what to do with decoding errors. As a result the parser simply
fails.
The simplest solution would probably be to expose Python's error handling in
pyarrow.csv.ReadOptions (e.g. encoding_errors: "strict" | "ignore" | "replace"
...).
It would also be useful to document the behaviour of the CSV reader. E.g. that
it only accepts binary streams, and how encoding errors are handled. In
particular it is unclear what "Columns that cannot decode using this encoding
can still be read as Binary" means, since the parser will currently fail if any
bytes cannot be decoded.
Toy example:
{code:java}
txt = """
col_😀_1, col2
0,a
1,b
"""
buffer = io.BytesIO(txt.encode("utf-8"))
pa.csv.read_csv(buffer, pa.csv.ReadOptions(encoding="ascii")){code}
{noformat}
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 5: ordinal
not in range(128){noformat}
whereas "with pandas":
{code:java}
buffer = io.BytesIO(txt.encode("utf-8"))
text = io.TextIOWrapper(buffer, encoding="ascii", errors="replace")
pd.read_csv(text){code}
{noformat}
col_����_1 col2
0 0 a
1 1 b
{noformat}
was:
When trying to read arbitrary CSV files, it is not possible to infer/guess the
correct encoding 100% of the time. The Arrow CSV reader will currently fail if
any byte cannot be decoded given the specified encoding (see example below).
With pandas.read_csv(), I can often get a result that is 99.9% correct by
passing it a text stream decoded in Python with
[errors="replace"|https://docs.python.org/3/library/codecs.html#error-handlers]
(or "ignore" etc.).
Pyarrow's csv.read_csv() on the other hand neither accepts an already decoded
text stream (TypeError: binary file expected, got text file), nor a parameter
to configure what to do with decoding errors. As a result the parser simply
fails.
The simplest solution would probably be to expose Python's error handling in
pyarrow.csv.ReadOptions (e.g. encoding_errors: "strict" | "ignore" | "replace"
...).
It would also be useful to document the behaviour of the CSV reader. E.g. that
it only accepts binary streams, and how encoding errors are handled. In
particular it is unclear what "Columns that cannot decode using this encoding
can still be read as Binary" means, since the parser will currently fail if any
bytes cannot be decoded.
Toy example:
{code:java}
import io
txt = """
col_😀_1, col2
0,a
1,b
"""
buffer = io.BytesIO(txt.encode("utf-8"))
pa.csv.read_csv(buffer, pa.csv.ReadOptions(encoding="ascii")){code}
{noformat}
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 5: ordinal
not in range(128){noformat}
whereas "with pandas":
{code:java}
buffer = io.BytesIO(txt.encode("utf-8"))
text = io.TextIOWrapper(buffer, encoding="ascii", errors="replace")
pd.read_csv(text){code}
{noformat}
col_����_1 col2
0 0 a
1 1 b
{noformat}
> [Python] CSV reader: allow parsing without encoding errors
> ----------------------------------------------------------
>
> Key: ARROW-16682
> URL: https://issues.apache.org/jira/browse/ARROW-16682
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Affects Versions: 8.0.0
> Reporter: Thomas Buhrmann
> Priority: Major
>
> When trying to read arbitrary CSV files, it is not possible to infer/guess
> the correct encoding 100% of the time. The Arrow CSV reader will currently
> fail if any byte cannot be decoded given the specified encoding (see example
> below).
> With pandas.read_csv(), I can often get a result that is 99.9% correct by
> passing it a text stream decoded in Python with
> [errors="replace"|https://docs.python.org/3/library/codecs.html#error-handlers]
> (or "ignore" etc.).
> Pyarrow's csv.read_csv() on the other hand neither accepts an already decoded
> text stream (TypeError: binary file expected, got text file), nor a parameter
> to configure what to do with decoding errors. As a result the parser simply
> fails.
> The simplest solution would probably be to expose Python's error handling in
> pyarrow.csv.ReadOptions (e.g. encoding_errors: "strict" | "ignore" |
> "replace" ...).
> It would also be useful to document the behaviour of the CSV reader. E.g.
> that it only accepts binary streams, and how encoding errors are handled. In
> particular it is unclear what "Columns that cannot decode using this encoding
> can still be read as Binary" means, since the parser will currently fail if
> any bytes cannot be decoded.
> Toy example:
>
> {code:java}
> txt = """
> col_😀_1, col2
> 0,a
> 1,b
> """
> buffer = io.BytesIO(txt.encode("utf-8"))
> pa.csv.read_csv(buffer, pa.csv.ReadOptions(encoding="ascii")){code}
> {noformat}
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 5:
> ordinal not in range(128){noformat}
> whereas "with pandas":
> {code:java}
> buffer = io.BytesIO(txt.encode("utf-8"))
> text = io.TextIOWrapper(buffer, encoding="ascii", errors="replace")
> pd.read_csv(text){code}
> {noformat}
> col_����_1 col2
> 0 0 a
> 1 1 b
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
