vkhodygo commented on issue #37655:
URL: https://github.com/apache/arrow/issues/37655#issuecomment-1755258086
It's the latter. I know how you feel, dealing with TBs of data can be pretty
annoying. However, resolving this issue might take some time whereas many
people would benefit from a fix right now.
I did have another workaround for some of my data:
- group by keys
- save as `parquet`
- load and merge in batches
This is a very crude version of what devs suggested and it seems to be
working nicely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
