Yicong-Huang commented on code in PR #56649:
URL: https://github.com/apache/spark/pull/56649#discussion_r3460990908
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -4777,6 +4777,25 @@ object SQLConf {
"must be positive.")
.createWithDefault(100)
+ val PYTHON_UDF_MAX_BYTES_PER_BATCH =
+ buildConf("spark.sql.execution.python.udf.maxBytesPerBatch")
+ .internal()
+ .doc("Byte-size cap for a batch of rows sent to a worker for regular
(pickle-serialized, " +
+ "non-Arrow) Python UDF evaluation. Without it, BatchEvalPythonExec
batches purely by " +
+ "row count (spark.sql.execution.python.udf.maxRecordsPerBatch), so one
oversized batch " +
+ "can OOM the executor; a finite value caps the batch at the min of the
row-count and " +
+ "byte limits. The size is a best-effort per-row estimate of the
pickled size of the " +
+ "converted row (its accuracy is observable via the
pythonEstimatedInputBytes vs " +
+ "pythonDataSent metrics); a row larger than the cap still yields a
one-row batch. " +
+ "-1 (the default) means no limit.")
+ .version("5.0.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .bytesConf(ByteUnit.BYTE)
+ .checkValue(x => x == -1 || (x > 0 && x <= Int.MaxValue),
Review Comment:
that makes sense, we can keep Int.MaxValue in this case. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
