ysj1173886760 commented on issue #14229:
URL: https://github.com/apache/arrow/issues/14229#issuecomment-2562101475
> Would `iter_batches()` as a workaround ok?
i found that iter_batches() actually works.
i've met the same issue while reading 800k vectors with dimension 2560.
using pyarrow.parquet read_parquet will trigger same issue: OSError: List index
overflow.
And i found using iter_batches to read parquet file while i was trying to
take the subset of the data, it works.
here is corresponding code:
```
for batch in parquet_file.iter_batches(batch_size=batch_size):
if total_rows >= rows_to_read:
break
table = pa.Table.from_batches([batch])
result_tables.append(table.slice(0, min(batch_size, rows_to_read -
total_rows)))
total_rows += batch.num_rows
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
