ivoson opened a new pull request, #59637:
URL: https://github.com/apache/spark/pull/59637
### What changes were proposed in this pull request?
This is a follow-up to #59122, which added a row-size guard for
pickle-serialized (non-Arrow) Python UDFs. That guard sizes only top-level
`StringType` and `BinaryType` arguments. This PR also sizes top-level
`GeometryType` and `GeographyType` arguments.
- `BatchEvalPythonExec.projectedInputPayloadSizeEstimator` now picks its
supported argument types through a shared `isSupportedProjectedInputType`
predicate. It sizes geometry and geography values by their `BinaryView` byte
count (`row.getBinaryView(ordinal).numBytes()`).
- The docs of `spark.sql.execution.python.udf.rowSizeGuard.enabled` and
`spark.sql.execution.python.udf.rowSizeGuard.maxRowHeapFraction` now list
geometry and geography among the guarded types.
### Why are the changes needed?
Geometry and geography values are stored as `BinaryView`
(`PhysicalBinaryViewType`), and their payloads can be arbitrarily large.
`EvaluatePython` already converts them for pickled UDFs, and its pickled
batch-size accounting already counts their byte length. The row-size guard
skipped them, so an oversized geometry or geography argument got past the guard
and was converted and pickled anyway.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Updated `BatchEvalPythonExecSuite`:
- The estimator test now also checks that `GeometryType` and `GeographyType`
arguments are sized by their `BinaryView` byte count.
- A new test checks that the estimator counts UTF-8 bytes rather than
characters.
- The SQLConf test is rewritten to check that the plan forwards the guard
flag to the evaluator, with the flag both off (the row gets pickled) and on
(the row is rejected with `UDF_LIMITS.ROW_SIZE`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
