Spenserrrr commented on PR #59019: URL: https://github.com/apache/spark/pull/59019#issuecomment-5878249079
Thanks, @gaogaotiantian. I simplified this to reuse `-3` (`METRICS_DATA`) for one length-prefixed JSON object containing the existing timing and spill values. The Python worker dumps the values directly, and Scala parses and validates them before updating the existing SQL and task metrics. I removed the version negotiation and Python metrics container. The measurement boundaries and SQL metric keys are preserved; collection wrappers, additional measurements, and clock changes will be separate follow-ups. I also rebased onto master. Focused Python/Scala and SQL UDF tests pass locally; the Scala runs used LZ4 1.11.3 because master's 1.12.0 dependency was unavailable in the test environment. Could you take another look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
