srielau commented on PR #58549: URL: https://github.com/apache/spark/pull/58549#issuecomment-5802195799
Ownership pass complete in 69ef259f915. Scope assessment: - The PR did experience scope creep while the shared Arrow mapper exposed UDTF, Python DataSource, TransformWithState, UDT storage, and non-scalar eval-type consumers. - The current head has already narrowed those paths to deterministic rejection rather than positive support. I do not recommend splitting those rejection guards from this PR: they are required to keep the shared mapper from silently widening unsupported APIs. - The PR description now contains an explicit fence. Positive CHAR/VARCHAR assignment support for UDTFs, Python DataSource, TransformWithState, UDT storage, and grouped/cogrouped/aggregate UDFs belongs in separate follow-up PRs. This pass also: - restored `_has_type` physical UDT recursion and separated logical capability checks; - prevented transpilation from discarding checked CHAR/VARCHAR UDF semantics; - restored constructor-time Connect UDTF rejection; - corrected the Python DataSource UDT rejection test action; - addressed and resolved all four open review threads. Verification: - `ConvertToCatalystSuite`: passed (full suite). - `DataTypeTests`: passed (full class). - Python compile checks and `git diff --check`: passed. - Focused Connect UDTF and Python DataSource integration tests were selected but skipped locally because pandas/PyArrow are unavailable; CI will exercise them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
