thisisnic commented on issue #41890:
URL: https://github.com/apache/arrow/issues/41890#issuecomment-7038930082

   The reprex above no longer fails with current pandas: since pandas 2.2.0 the 
`string[pyarrow]` dtype is stored as `large_string`, so the offsets can't 
overflow.
   
   The underlying Arrow call does still fail on 26.0.0:
   
   ```python
   import numpy as np
   import pyarrow as pa
   
   chunk = pa.array(["x" * 1000] * 250_000)
   ca = pa.chunked_array([chunk] * 11)  # 2.57 GiB of string data
   ca.take(pa.array([0, len(ca) - 1]))
   # ArrowInvalid: offset overflow while concatenating arrays, consider casting 
input from `string` to `large_string` first.
   ```
   
   `take` on a chunked array concatenates the chunks into a single array first, 
which overflows the 32-bit offsets of `string` once there is more than 2 GiB of 
data. That is tracked in #25822, so I'm closing this as a duplicate of that 
issue.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to