thisisnic commented on issue #41890: URL: https://github.com/apache/arrow/issues/41890#issuecomment-7038930082
The reprex above no longer fails with current pandas: since pandas 2.2.0 the `string[pyarrow]` dtype is stored as `large_string`, so the offsets can't overflow. The underlying Arrow call does still fail on 26.0.0: ```python import numpy as np import pyarrow as pa chunk = pa.array(["x" * 1000] * 250_000) ca = pa.chunked_array([chunk] * 11) # 2.57 GiB of string data ca.take(pa.array([0, len(ca) - 1])) # ArrowInvalid: offset overflow while concatenating arrays, consider casting input from `string` to `large_string` first. ``` `take` on a chunked array concatenates the chunks into a single array first, which overflows the 32-bit offsets of `string` once there is more than 2 GiB of data. That is tracked in #25822, so I'm closing this as a duplicate of that issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
