viirya opened a new pull request, #50430: URL: https://github.com/apache/arrow/pull/50430
### Rationale for this change Follow-up of #50326; **stacked on #50327 — only the last commit is new, and this will be rebased once #50327 merges.** GH-50327 converts arrays to Python objects without per-element Scalars, but the `maps_as_pydicts` option still routes to the Scalar-based path: every map row allocates a `MapScalar`, converts its keys via per-element `as_py`, and builds the dict in Python. Map→dict is the natural consumption pattern for engines whose map values are Python dicts (e.g. Spark's Arrow-serialized Python UDFs currently receive association lists and rebuild a dict per row in pure Python — one of the dominant remaining costs in that path). ### What changes are included in this PR? Thread `maps_as_pydicts` through the scalar-free `_getitem_py` mechanism: - `MapArray._getitem_py` builds the dict directly from the flattened keys/items children. When the resulting dict size reveals duplicate keys, the row is redone with the careful per-key loop, so the `'lossy'` warnings and `'strict'` `KeyError` match `MapScalar.as_py` exactly (including messages and warning-per-duplicate behavior). - Invalid option values raise the same `ValueError` when a map value is converted — including null map rows, matching the Scalar path — while non-map arrays keep ignoring the option. - The option propagates through nested types (list/struct children, map values) as before, and unspecialized types keep the exact Scalar fallback, which now receives the option. Benchmark (macOS arm64, M4 Max; 1M rows of 2-entry `map<string,int64>`, 10% nulls): | conversion | before | after | speedup | |---|---|---|---| | `to_pylist(maps_as_pydicts='lossy')` | 2.20 s | 0.10 s | ~21x | Notably the dict form is now also faster than the default association-list form (0.78 s), which allocates a 2-tuple per entry. ### Are these changes tested? New `test_to_pylist_maps_as_pydicts` compares against the per-scalar conversion for flat maps, `list<map>`, `map<string, map<...>>` and `struct<map>` (plain and sliced) in both modes, and asserts the duplicate-key semantics (`'lossy'` warns and keeps the last value; `'strict'` raises `KeyError`), the invalid-value `ValueError`, and that non-map arrays ignore the option. Randomized differential tests against the Scalar path (exact type equality) and `pytest test_array.py test_scalars.py test_convert_builtin.py test_table.py` (1210 passed) also pass. ### Are there any user-facing changes? No behavior changes, only performance. * GitHub Issue: #50429 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
