marin-ma commented on PR #10119:
URL:
https://github.com/apache/incubator-gluten/pull/10119#issuecomment-3063102313
@FelixYBW The task stats is now printed in json format, but it's represented
as a string in the event log. The metric name is "velox task stats" in
"SparkListenerTaskEnd" event. Here's an example output
```
{
"Event": "SparkListenerTaskEnd",
"Stage ID": 15,
"Stage Attempt ID": 0,
"Task Type": "ResultTask",
"Task End Reason": {
"Reason": "Success"
},
"Task Info": {
"Task ID": 22,
"Index": 0,
"Attempt": 0,
"Partition ID": 0,
"Launch Time": 1752251466838,
"Executor ID": "driver",
"Host": "localhost",
"Locality": "NODE_LOCAL",
"Speculative": false,
"Getting Result Time": 0,
"Finish Time": 1752251466855,
"Failed": false,
"Killed": false,
"Accumulables": [
...
{
"ID": 2005,
"Name": "velox task stats",
"Value": "[{\"spilledFiles\":0,\"cpuWallTiming\":\"count: 15,
wallTime: 90.29us, cpuTime: 86.88us\",\"getOutputTiming\":\"count: 4, wallTime:
12.29us, cpuTime:
11.67us\",\"outputRows\":4,\"outputBytes\":704,\"addInputTiming\":\"count: 1,
wallTime: 18.54us, cpuTime:
18.33us\",\"operatorType\":\"OrderBy\",\"planNodeId\":\"1\",\"customStats\":{\"runningGetOutputWallNanos\":\"sum:12.29us,
count:1, min:12.29us, max:12.29us, avg:
12.29us\",\"runningAddInputWallNanos\":\"sum:18.54us, count:1, min:18.54us,
max:18.54us, avg: 18.54us\",\"queuedWallNanos\":\"sum:0ns, count:1, min:0ns,
max:0ns, avg: 0ns\",\"runningFinishWallNanos\":\"sum:55.71us, count:1,
min:55.71us, max:55.71us, avg:
55.71us\"},\"numMemoryAllocations\":12,\"numSplits\":0,\"blockedWallNanos\":0,\"finishTiming\":\"count:
4, wallTime: 55.71us, cpuTime:
55.04us\",\"rawInputRows\":0,\"inputVectors\":1,\"inputBytes\":584,\"inputRows\":4,\"peakMemoryBytes\":67264,\"spilledBytes\":0,\"spilledInputBytes\":0,\"isBlockedTiming
\":\"count: 6, wallTime: 3.75us, cpuTime:
1.83us\",\"numDrivers\":1,\"spilledRows\":0,\"physicalWrittenBytes\":0,\"outputVectors\":1,\"rawInputBytes\":0},{\"spilledFiles\":0,\"cpuWallTiming\":\"count:
6, wallTime: 337.33us, cpuTime:
330.42us\",\"outputBytes\":584,\"getOutputTiming\":\"count: 2, wallTime:
335.67us, cpuTime: 329.46us\",\"addInputTiming\":\"count: 0, wallTime: 0ns,
cpuTime:
0ns\",\"outputRows\":4,\"planNodeId\":\"0\",\"operatorType\":\"ValueStream\",\"customStats\":{\"runningFinishWallNanos\":\"sum:417ns,
count:1, min:417ns, max:417ns, avg:
417ns\",\"runningAddInputWallNanos\":\"sum:0ns, count:1, min:0ns, max:0ns, avg:
0ns\",\"runningGetOutputWallNanos\":\"sum:335.67us, count:1, min:335.67us,
max:335.67us, avg:
335.67us\"},\"blockedWallNanos\":0,\"numMemoryAllocations\":0,\"numSplits\":0,\"finishTiming\":\"count:
1, wallTime: 417ns, cpuTime:
208ns\",\"rawInputRows\":0,\"inputBytes\":0,\"inputVectors\":0,\"inputRows\":0,\"peakMemoryBytes\":0,\"spilledBytes\":0,\"spilled
InputBytes\":0,\"isBlockedTiming\":\"count: 3, wallTime: 1.25us, cpuTime:
751ns\",\"spilledRows\":0,\"numDrivers\":1,\"physicalWrittenBytes\":0,\"rawInputBytes\":0,\"outputVectors\":1}]",
"Internal": false,
"Count Failed Values": false
},
...
]
},
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org
