lidavidm commented on issue #44944: URL: https://github.com/apache/arrow/issues/44944#issuecomment-2518809918
So the 2GB limit is on a single Array, but the columns of a Table are not Arrays. Instead they're ChunkedArrays which you can think of as a list of Arrays. Each individual chunk in a ChunkedArray has the 2GB limit, but the total of the ChunkedArray can surpass that limit. It's possible Polars converted to LargeString and combined the chunks into a single array, which then on the Python side can't be casted back into a String anymore. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
