[
https://issues.apache.org/jira/browse/ARROW-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231699#comment-17231699
]
ran commented on ARROW-10579:
-----------------------------
Yes. I did a drill down it seems when the file is bigger than 1GB it starts.
How can I share this file? can you send me your email?
> read_csv from a large file with long string columns failed to parse the input
> correctly
> ---------------------------------------------------------------------------------------
>
> Key: ARROW-10579
> URL: https://issues.apache.org/jira/browse/ARROW-10579
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 2.0.0
> Environment: python 3.8 pyarrow 2.0.0
> Reporter: ran
> Priority: Major
>
> import pyarrow.csv as pac # PyArrow is installed with `datasets`
> read_options = pac.ReadOptions(block_size=1e9)
> parse_options = pac.ParseOptions(delimiter=',')
> table = csv.read_csv('0.csv',
> read_options=read_options,parse_options=parse_options)
> I get the following error:
> ~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/_csv.pyx in
> pyarrow._csv.read_csv()
> ~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in
> pyarrow.lib.pyarrow_internal_check_status()
> ~/anaconda2/envs/nlp/lib/python3.8/site-packages/pyarrow/error.pxi in
> pyarrow.lib.check_status() ArrowInvalid: CSV parse error: Expected 5 columns,
> got 187
> I validate that the CSV is good and I am able to read it using Pandas and
> write+read it to hdf5.
> Thanks for the help :)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
