Spenserrrr commented on code in PR #59019: URL: https://github.com/apache/spark/pull/59019#discussion_r4137318045
########## python/pyspark/tests/test_worker.py: ########## @@ -31,9 +34,37 @@ from py4j.protocol import Py4JJavaError from pyspark import SparkConf, SparkContext +from pyspark.serializers import read_int from pyspark.testing.utils import PySparkTestCase, QuietTest, ReusedPySparkTestCase, eventually +class WorkerProtocolTests(unittest.TestCase): Review Comment: Thanks for pointing this out. I've removed the direct Python protocol test. There are already tests that check metrics from the driver side. For example, the [SPARK-55046 test in `PythonUDFSuite`](https://github.com/apache/spark/blob/9f6dfaa04bf5f3e6cd9879eeda777515b61886e6/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala#L197) runs a real Python UDF and calls `collect()`. The worker sends results and its JSON metrics report over the socket, and the JVM parses the report and updates the SQL metrics. The test then reads `pythonProcessingTime` and `pythonTotalTime` from the executed plan and checks that execution time is positive and total time is at least as large. This covers the successful worker-to-JVM communication path end to end. I also ran `bench_eval_type.ArrowBatchedUDFTimeBench` locally with ASV `--quick`: all 24 parameter combinations passed. The benchmark supplies input through `BytesIO` and does not parse the output metrics, so it requires no changes for the JSON report. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
