Yicong-Huang opened a new pull request, #54172: URL: https://github.com/apache/spark/pull/54172
### What changes were proposed in this pull request? Consolidate the `SQL_MAP_ARROW_ITER_UDF` (`mapInArrow`) execution path so that all input transformation, UDF invocation, result verification, and output transformation live in a single mapper block inside `read_udfs`. - Remove `wrap_arrow_batch_iter_udf`; inline its logic into the mapper as an explicit `flatten → UDF → verify → wrap` pipeline with type annotations. - Introduce `ArrowStreamGroupSerializer`, a configurable serializer that extends `ArrowStreamSerializer` with grouped-loading and `START_ARROW_STREAM` support. This is intended as the single serializer base that all UDF eval types will gradually migrate to, replacing the current per-eval-type serializer subclasses. - Extract a reusable `verify_iterator_result` helper for iterable/element-type checking. ### Why are the changes needed? Part of [SPARK-55389](https://issues.apache.org/jira/browse/SPARK-55389). The existing logic for this eval type is scattered across a wrapper function, a shared mapper branch, and the serializer. After this change the full data flow is visible in one place. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing `mapInArrow` tests (`test_pandas_map.py`, `test_arrow_map.py`). ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
