ivoson commented on PR #59122: URL: https://github.com/apache/spark/pull/59122#issuecomment-6011896464
@Yicong-Huang Thanks for raising the nested-column case. Current status: - This PR is intentionally scoped to top-level `StringType` and `BinaryType` inputs; the PR title now makes that scope explicit. - The limitation is documented in both the SQL config description (`Nested inputs are not estimated`) and the PR description, which calls out arrays, maps, and structs as excluded. - Nested support is deferred because it requires a recursive, size-only traversal before conversion. That needs separate design and performance evaluation to avoid allocations and excessive per-row overhead, and to handle arrays, maps, structs, nulls, deeply nested values, and UDTs correctly. The existing `PickledSizeAccumulator` path runs during conversion, so it cannot directly provide the required pre-conversion protection. - The follow-up work is tracked in [SPARK-59996](https://issues.apache.org/jira/browse/SPARK-59996). The test-coverage feedback has also been addressed: the exact `maxRowSize`/`actualRowSize` parameters are now checked with `checkError`, and a SQLConf-driven test covers the config-to-evaluator wiring. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
