johnseekins commented on issue #34688: URL: https://github.com/apache/arrow/issues/34688#issuecomment-1489473730
As far as I've been able to find, the correct answer is to basically not use datasets to try and load large amounts of data. And generally, using partitions with pyarrow seems to simply remove those columns from the actual data, which somewhat defeats the purpose of them. Closing this, but maybe some day we'll come back to it with renewed vigor! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
