westonpace commented on issue #11427: URL: https://github.com/apache/arrow/issues/11427#issuecomment-948928059
If I understand correctly you were not able to create a minimal test file. Is this true? Can you try creating a minimal file with something other than pandas. For example, on Linux you can do `cat large.csv | head -n 10 | tail -n 10 > small.csv`? From the error message it seems there is one line that does not have enough fields. For example: ``` model_number,width,length xyz,100,200 abc, def,200,400 ``` The line `abc,` would generate an error like the one you are seeing. Can you maybe check for lines like this. I realize it is a large file so this may be difficult. Pyarrow version 6.0.0 should provide a bit more information with the error (it will print the first 100 characters of the offending line). Can you try rerunning this when version 6.0.0 releases? If you want to try sooner you could try installing a [nightly wheel](https://arrow.apache.org/docs/python/install.html#installing-nightly-packages). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
