Spenserrrr commented on code in PR #59019: URL: https://github.com/apache/spark/pull/59019#discussion_r4137318045
########## python/pyspark/tests/test_worker.py: ########## @@ -31,9 +34,37 @@ from py4j.protocol import Py4JJavaError from pyspark import SparkConf, SparkContext +from pyspark.serializers import read_int from pyspark.testing.utils import PySparkTestCase, QuietTest, ReusedPySparkTestCase, eventually +class WorkerProtocolTests(unittest.TestCase): Review Comment: Removed the direct Python protocol test. For example, the existing `PythonUDFSuite` test `SPARK-55046: pythonProcessingTime reflects actual UDF computation time` runs a real Spark query with a Python UDF and calls `collect()`. The worker sends results and its JSON metrics report over the socket; the JVM parses the report and updates the SQL metrics. The test then reads `pythonProcessingTime` and `pythonTotalTime` from the executed plan and checks that execution time is positive and total time is at least as large. This exercises the complete successful worker-to-JVM communication path. The focused Scala reader tests cover invalid reports and stream alignment separately. The retained `test_worker_metrics_include_spills` supplies known spill counters inside real worker tasks, then reads the driver's status store to check the reported memory and disk spill totals. It also checks task results and accumulator updates, covering both normal output and the accumulator section after the metrics report. I also ran `bench_eval_type.ArrowBatchedUDFTimeBench` locally with ASV `--quick`: all 24 parameter combinations passed. The benchmark supplies input through `BytesIO` and does not parse the output metrics, so it requires no changes for the JSON report. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
