zhengruifeng opened a new pull request, #57744:
URL: https://github.com/apache/spark/pull/57744
### What changes were proposed in this pull request?
Add a test to ScalarArrowUDFTestsMixin that verifies TaskContext is
initialized in both scalar and scalar-iterator Arrow UDFs. The test checks the
partition IDs returned by TaskContext and is also inherited by the Spark
Connect parity suite.
### Why are the changes needed?
TaskContext is covered for Pandas UDFs and mapInArrow, but scalar Arrow UDFs
did not have equivalent coverage. This test closes that gap and guards classic
and Spark Connect execution against regressions.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
build/sbt -Phive package
conda run -n spark-dev-313 --no-capture-output python/run-tests
--testnames "pyspark.sql.tests.arrow.test_arrow_udf_scalar
ScalarArrowUDFTests.test_arrow_udf_task_context"
conda run -n spark-dev-313 --no-capture-output python/run-tests
--testnames "pyspark.sql.tests.connect.arrow.test_parity_arrow_udf_scalar
ScalarArrowPythonUDFParityTests.test_arrow_udf_task_context"
conda run -n spark-dev-313 ruff check
python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
conda run -n spark-dev-313 ruff format --check
python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
