el-hult commented on issue #32723: URL: https://github.com/apache/arrow/issues/32723#issuecomment-6520505026
Hi! I'm interested in seeing this resolved. I haven't contributed code to Arrow before, but I've read the contributor guidelines. I'm not a strong C++ developer, though I can read it and follow the code. With help from Claude (an AI assistant), I went through the earlier attempts and discussion here and put together a proposal. Before opening a PR, I'd like to check whether the direction makes sense to you. **My understanding of the problem** Nested readers (list, map, struct) fail with `Nested data conversions not implemented for chunked array outputs` whenever a leaf returns more than one chunk. I found two ways this happens: 1. **Dictionary leaves.** Each row group has its own dictionary, so a `list<dictionary<string>>` with several row groups fails however small it is (#45087). With pyarrow 26.0.0, `ParquetFile.read()`, `read_row_groups()` and `iter_batches()` fail; `pq.read_table()` works because the dataset API reads row group by row group. 2. **String data over 2 GiB.** The decoder starts a new chunk when the 32-bit offsets would overflow. This is the original report here, reproduced by `large_string_map.brotli.parquet` in parquet-testing. That file fails through every pyarrow read API I tried, including `binary_type=pa.large_binary()`: the stored Arrow schema keeps the map keys as `string`. **Previous work, as I understand it** - #35825 (closed by its author, not merged) proposed an option to read as large types. As I read it, reviewers objected to the duplicated code and questioned whether a new API was justified. It did contribute `large_string_map.brotli.parquet` to parquet-testing (apache/parquet-testing#38, merged). - #41104 (issue, closed as stale) proposed building LargeBinary in the dictionary reader. - #46532 (merged) later added `ArrowReaderProperties::binary_type`, which avoids case 2 when it applies. The default still chunks, and it doesn't address case 1. **Proposal** Let nested readers do what flat readers already do: return several chunks instead of refusing. Nested columns could then come back as several chunks, for example one per row group for nested dictionary columns. I have a draft that's about 120 lines in `reader.cc` plus tests, and I can open it as a PR if the direction sounds reasonable. **Questions before I open a PR** 1. Is splitting into several chunks an acceptable direction, or would you prefer another approach? 2. Should the dictionary case and the 2 GiB case be one PR or two? Happy to adjust or drop this if it doesn't fit. Thanks for any guidance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
