[GitHub] [spark] otterc commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

GitBox Mon, 12 Dec 2022 12:13:08 -0800


otterc commented on code in PR #36165:
URL: https://github.com/apache/spark/pull/36165#discussion_r1046319200



##########
core/src/main/scala/org/apache/spark/util/JsonProtocol.scala:
##########
@@ -1105,11 +1133,17 @@ private[spark] object JsonProtocol {
         case None => Seq.empty[AccumulableInfo]
       }
     }
+    val isPushBasedShuffleEnabled =

Review Comment:
   Nit: rename to `isShufflePushEnabled`?



##########
core/src/test/resources/HistoryServerExpectations/stage_with_summaries_expectation.json:
##########
@@ -1094,5 +1308,7 @@
       "MajorGCTime" : [ 0.0, 0.0, 339.0, 339.0, 339.0 ],
       "TotalGCTime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ]
     }
-  }
-}
+  },
+  "isShufflePushEnabled" : false,
+  "shuffleMergersCount" : 0
+}

Review Comment:
   Missing new line at the end of file



##########
core/src/test/resources/HistoryServerExpectations/stage_list_with_peak_metrics_expectation.json:
##########
@@ -203,5 +237,7 @@
     "MajorGCCount" : 3,
     "MajorGCTime" : 110,
     "TotalGCTime" : 0
-  }
-} ]
+  },
+  "isShufflePushEnabled" : false,
+  "shuffleMergersCount" : 0
+} ]

Review Comment:
   Missing new line at the end of file



##########
project/MimaExcludes.scala:
##########
@@ -87,6 +87,11 @@ object MimaExcludes {
     // [SPARK-36511][MINOR][SQL] Remove ColumnIOUtil
     
ProblemFilters.exclude[MissingClassProblem]("org.apache.parquet.io.ColumnIOUtil"),
 
+    // [SPARK-36620] [SHUFFLE] Expose push based shuffle metrics

Review Comment:
   Reword to `Expose shuffle push metrics`



##########
core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala:
##########
@@ -1653,7 +1663,19 @@ private[spark] object JsonProtocolSuite extends 
Assertions {
       |      "Remote Bytes Read": 1000,
       |      "Remote Bytes Read To Disk": 400,
       |      "Local Bytes Read": 1100,
-      |      "Total Records Read": 10
+      |      "Total Records Read": 10,
+      |      "Remote Requests Duration": 900,
+      |      "Push Based Shuffle": {
+      |         "Corrupt Merged Block Chunks" : 100,
+      |         "Merged Fetch Fallback Count" : 100,
+      |         "Merged Remote Blocks Fetched" : 0,
+      |         "Merged Local Blocks Fetched" : 0,
+      |         "Merged Remote Chunks Fetched" : 0,
+      |         "Merged Local Chunks Fetched" : 0,
+      |         "Merged Remote Bytes Read" : 0,
+      |         "Merged Local Bytes Read" : 0,
+      |         "Merged Remote Requests Duration": 900

Review Comment:
   If merged remote blocks are 0 then this can't be > 0



##########
core/src/main/scala/org/apache/spark/util/JsonProtocol.scala:
##########
@@ -1225,6 +1259,29 @@ private[spark] object JsonProtocol {
       readMetrics.incFetchWaitTime(readJson.get("Fetch Wait Time").extractLong)
       readMetrics.incRecordsRead(
         jsonOption(readJson.get("Total Records 
Read")).map(_.extractLong).getOrElse(0L))
+      readMetrics.incRemoteReqsDuration(jsonOption(readJson.get("Remote 
Requests Duration"))
+        .map(_.extractLong).getOrElse(0L))
+      jsonOption(readJson.get("Shuffle Push Read Metrics")).foreach { 
pushBasedShuffleReadJson =>

Review Comment:
   Nit: Rename to shufflePushReadJson?



##########
core/src/test/resources/HistoryServerExpectations/stage_with_summaries_expectation.json:
##########
@@ -33,6 +33,16 @@
   "shuffleLocalBytesRead" : 0,
   "shuffleReadBytes" : 0,
   "shuffleReadRecords" : 0,
+  "shuffleCorruptMergedBlockChunks" : 160,
+  "shuffleMergedFetchFallbackCount" : 32,
+  "shuffleMergedRemoteBlocksFetched" : 160,
+  "shuffleMergedLocalBlocksFetched" : 240,
+  "shuffleMergedRemoteChunksFetched" : 192,
+  "shuffleMergedLocalChunksFetched" : 160,
+  "shuffleMergedRemoteBytesRead" : 1600,
+  "shuffleMergedLocalBytesRead" : 1600,
+  "shuffleRemoteReqsDuration" : 0,

Review Comment:
   Same here



##########
core/src/test/resources/HistoryServerExpectations/stage_with_peak_metrics_expectation.json:
##########
@@ -33,6 +33,16 @@
   "shuffleLocalBytesRead" : 0,
   "shuffleReadBytes" : 0,
   "shuffleReadRecords" : 0,
+  "shuffleCorruptMergedBlockChunks" : 160,
+  "shuffleMergedFetchFallbackCount" : 32,
+  "shuffleMergedRemoteBlocksFetched" : 160,
+  "shuffleMergedLocalBlocksFetched" : 240,
+  "shuffleMergedRemoteChunksFetched" : 192,
+  "shuffleMergedLocalChunksFetched" : 160,
+  "shuffleMergedRemoteBytesRead" : 1600,
+  "shuffleMergedLocalBytesRead" : 1600,
+  "shuffleRemoteReqsDuration" : 0,

Review Comment:
   if mergedRemoteReqDuration > 0, then shuffleRemoteReqsDuration should be 
greater than mergedRemoteReqDuration. Can we update this?
   The existing values for shuffleLocalBytesRead, and shuffleReadBytes  also 
don't make sense now because if we are saying that there were merged local 
blocks, then these value should be > 0. 
   We can either change or ignore. I am good with either.
   
   



##########
core/src/test/resources/HistoryServerExpectations/stage_list_json_expectation.json:
##########
@@ -270,5 +316,7 @@
     "MajorGCCount" : 0,
     "MajorGCTime" : 0,
     "TotalGCTime" : 0
-  }
-} ]
+  },
+  "isShufflePushEnabled" : false,
+  "shuffleMergersCount" : 0
+} ]

Review Comment:
   Missing newline at the end of file



##########
core/src/test/resources/spark-events/app-20200706201101-0003:
##########
@@ -100,25 +100,25 @@
 {"Event":"SparkListenerTaskStart","Stage ID":2,"Stage Attempt ID":0,"Task 
Info":{"Task ID":45,"Index":13,"Attempt":0,"Launch 
Time":1594091481129,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":0,"Failed":false,"Killed":false,"Accumulables":[]}}
 {"Event":"SparkListenerTaskStart","Stage ID":2,"Stage Attempt ID":0,"Task 
Info":{"Task ID":46,"Index":14,"Attempt":0,"Launch 
Time":1594091481132,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":0,"Failed":false,"Killed":false,"Accumulables":[]}}
 {"Event":"SparkListenerTaskStart","Stage ID":2,"Stage Attempt ID":0,"Task 
Info":{"Task ID":47,"Index":15,"Attempt":0,"Launch 
Time":1594091481136,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":0,"Failed":false,"Killed":false,"Accumulables":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":38,"Index":6,"Attempt":0,"Launch Time":1594091481104,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091482939,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1467","Value":"1467","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"375000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"6250","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":24040,"Internal":true,"Count
 Failed 
Values":true},{"ID":71,"Name":"internal.metrics.resultSerializationTime","Update":11,"Value":11,"Internal":tru
 e,"Count Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":62,"Internal":true,"Count
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2158,"Value":2158,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":414110000,"Value":414110000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1498,"Value":1498,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":60358000,"Value":60358000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":255,"Value":255,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":
 
0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":255,"Executor Deserialize CPU 
Time":60358000,"Executor Run Time":1498,"Executor CPU Time":414110000,"Peak 
Execution Memory":24040,"Result Size":2158,"JVM GC Time":62,"Result 
Serialization Time":11,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":47,"Index":15,"Attempt":0,"Launch Time":1594091481136,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483014,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1563","Value":"3030","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"750000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"12500","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":48080,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":52,"Value":114,"Internal":true,"Count
 Fa
 iled 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":4273,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":1324251000,"Value":1738361000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1594,"Value":3092,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":47496000,"Value":107854000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":213,"Value":468,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMem
 
ory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":213,"Executor Deserialize CPU 
Time":47496000,"Executor Run Time":1594,"Executor CPU Time":1324251000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":52,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":37,"Index":5,"Attempt":0,"Launch Time":1594091481100,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483015,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1563","Value":"4593","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"1125000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"18750","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":72120,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":176,"Internal":true,"Count
 Fa
 iled 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":6388,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":507192000,"Value":2245553000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1596,"Value":4688,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":60890000,"Value":168744000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":256,"Value":724,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMemo
 
ry":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":256,"Executor Deserialize CPU 
Time":60890000,"Executor Run Time":1596,"Executor CPU Time":507192000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":36,"Index":4,"Attempt":0,"Launch Time":1594091481095,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483015,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1564","Value":"6157","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"1500000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"25000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":96160,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":238,"Internal":true,"Count
 Fa
 iled 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":8503,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":503010000,"Value":2748563000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1596,"Value":6284,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":112849000,"Value":281593000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":260,"Value":984,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMem
 
ory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":260,"Executor Deserialize CPU 
Time":112849000,"Executor Run Time":1596,"Executor CPU Time":503010000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":35,"Index":3,"Attempt":0,"Launch Time":1594091481091,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483016,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1566","Value":"7723","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"1875000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"31250","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":120200,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":300,"Internal":true,"Count
 F
 ailed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":10618,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":502908000,"Value":3251471000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1598,"Value":7882,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":62944000,"Value":344537000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":263,"Value":1247,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVM
 
emory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":263,"Executor Deserialize CPU 
Time":62944000,"Executor Run Time":1598,"Executor CPU Time":502908000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":40,"Index":8,"Attempt":0,"Launch Time":1594091481112,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483016,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1562","Value":"9285","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"2250000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"37500","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":144240,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":362,"Internal":true,"Count
 F
 ailed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":12733,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":510597000,"Value":3762068000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1595,"Value":9477,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":69760000,"Value":414297000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":246,"Value":1493,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVM
 
emory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":246,"Executor Deserialize CPU 
Time":69760000,"Executor Run Time":1595,"Executor CPU Time":510597000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":45,"Index":13,"Attempt":0,"Launch Time":1594091481129,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483024,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1564","Value":"10849","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"2625000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"43750","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":168280,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":424,"Internal":true,"Count
  Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":14848,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":495138000,"Value":4257206000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1595,"Value":11072,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":54222000,"Value":468519000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":221,"Value":1714,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePytho
 
nVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":221,"Executor Deserialize CPU 
Time":54222000,"Executor Run Time":1595,"Executor CPU Time":495138000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":39,"Index":7,"Attempt":0,"Launch Time":1594091481109,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483024,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1566","Value":"12415","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"3000000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"50000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":192320,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":486,"Internal":true,"Count
 
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":16963,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":539451000,"Value":4796657000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1596,"Value":12668,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":64380000,"Value":532899000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":254,"Value":1968,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePython
 
VMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":254,"Executor Deserialize CPU 
Time":64380000,"Executor Run Time":1596,"Executor CPU Time":539451000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":33,"Index":1,"Attempt":0,"Launch Time":1594091481082,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483025,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1566","Value":"13981","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"3375000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"56250","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":216360,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":548,"Internal":true,"Count
 
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":19078,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":519178000,"Value":5315835000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1597,"Value":14265,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":54442000,"Value":587341000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":267,"Value":2235,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePython
 
VMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":267,"Executor Deserialize CPU 
Time":54442000,"Executor Run Time":1597,"Executor CPU Time":519178000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":34,"Index":2,"Attempt":0,"Launch Time":1594091481087,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483026,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1574","Value":"15555","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"3750000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"62500","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":240400,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":614,"Internal":true,"Count
 
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":21193,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":508433000,"Value":5824268000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1606,"Value":15871,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":69492000,"Value":656833000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":265,"Value":2500,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePython
 
VMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":265,"Executor Deserialize CPU 
Time":69492000,"Executor Run Time":1606,"Executor CPU Time":508433000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":43,"Index":11,"Attempt":0,"Launch Time":1594091481123,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483029,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1578","Value":"17133","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"4125000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"68750","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":264440,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":680,"Internal":true,"Count
  Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":23308,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":502120000,"Value":6326388000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1609,"Value":17480,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":48849000,"Value":705682000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":225,"Value":2725,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePytho
 
nVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":225,"Executor Deserialize CPU 
Time":48849000,"Executor Run Time":1609,"Executor CPU Time":502120000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":41,"Index":9,"Attempt":0,"Launch Time":1594091481116,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483032,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1583","Value":"18716","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"4500000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"75000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":288480,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":746,"Internal":true,"Count
 
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":25423,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":489923000,"Value":6816311000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1614,"Value":19094,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":55787000,"Value":761469000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":240,"Value":2965,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePython
 
VMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":240,"Executor Deserialize CPU 
Time":55787000,"Executor Run Time":1614,"Executor CPU Time":489923000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":46,"Index":14,"Attempt":0,"Launch Time":1594091481132,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483037,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1587","Value":"20303","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"4875000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"81250","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":312520,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":812,"Internal":true,"Count
  Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":27538,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":490927000,"Value":7307238000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1618,"Value":20712,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":51464000,"Value":812933000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":218,"Value":3183,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePytho
 
nVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":218,"Executor Deserialize CPU 
Time":51464000,"Executor Run Time":1618,"Executor CPU Time":490927000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":32,"Index":0,"Attempt":0,"Launch Time":1594091481077,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483037,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1587","Value":"21890","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"5250000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"87500","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":336560,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":878,"Internal":true,"Count
 
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":29653,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":496683000,"Value":7803921000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1619,"Value":22331,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":56827000,"Value":869760000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":271,"Value":3454,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePython
 
VMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":271,"Executor Deserialize CPU 
Time":56827000,"Executor Run Time":1619,"Executor CPU Time":496683000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":42,"Index":10,"Attempt":0,"Launch Time":1594091481120,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483043,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1593","Value":"23483","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"5625000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"93750","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":360600,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":944,"Internal":true,"Count
  Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":31768,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":508230000,"Value":8312151000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1624,"Value":23955,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":58152000,"Value":927912000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":229,"Value":3683,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePytho
 
nVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":229,"Executor Deserialize CPU 
Time":58152000,"Executor Run Time":1624,"Executor CPU Time":508230000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
-{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":44,"Index":12,"Attempt":0,"Launch Time":1594091481126,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483043,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1593","Value":"25076","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"6000000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"100000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":384640,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":1010,"Internal":true,"Cou
 nt Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":33883,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":498187000,"Value":8810338000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1624,"Value":25579,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":51988000,"Value":979900000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":222,"Value":3905,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePyt
 
honVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":222,"Executor Deserialize CPU 
Time":51988000,"Executor Run Time":1624,"Executor CPU Time":498187000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes 
Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input 
Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes 
Written":0,"Records Written":0},"Updated Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":38,"Index":6,"Attempt":0,"Launch Time":1594091481104,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091482939,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1467","Value":"1467","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"375000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"6250","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":24040,"Internal":true,"Count
 Failed 
Values":true},{"ID":71,"Name":"internal.metrics.resultSerializationTime","Update":11,"Value":11,"Internal":tru
 e,"Count Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":62,"Internal":true,"Count
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2158,"Value":2158,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":414110000,"Value":414110000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1498,"Value":1498,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":60358000,"Value":60358000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":255,"Value":255,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":
 
0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":255,"Executor Deserialize CPU 
Time":60358000,"Executor Run Time":1498,"Executor CPU Time":414110000,"Peak 
Execution Memory":24040,"Result Size":2158,"JVM GC Time":62,"Result 
Serialization Time":11,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remo
 te Bytes Read":100,"Merged Local Bytes Read":100}},"Shuffle Write 
Metrics":{"Shuffle Bytes Written":0,"Shuffle Write Time":0,"Shuffle Records 
Written":0},"Input Metrics":{"Bytes Read":0,"Records Read":0},"Output 
Metrics":{"Bytes Written":0,"Records Written":0},"Updated Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":47,"Index":15,"Attempt":0,"Launch Time":1594091481136,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483014,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1563","Value":"3030","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"750000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"12500","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":48080,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":52,"Value":114,"Internal":true,"Count
 Fa
 iled 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":4273,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":1324251000,"Value":1738361000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1594,"Value":3092,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":47496000,"Value":107854000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":213,"Value":468,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMem
 
ory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":213,"Executor Deserialize CPU 
Time":47496000,"Executor Run Time":1594,"Executor CPU Time":1324251000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":52,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write Tim
 e":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":37,"Index":5,"Attempt":0,"Launch Time":1594091481100,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483015,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1563","Value":"4593","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"1125000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"18750","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":72120,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":176,"Internal":true,"Count
 Fa
 iled 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":6388,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":507192000,"Value":2245553000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1596,"Value":4688,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":60890000,"Value":168744000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":256,"Value":724,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMemo
 
ry":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":256,"Executor Deserialize CPU 
Time":60890000,"Executor Run Time":1596,"Executor CPU Time":507192000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write Time"
 :0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":36,"Index":4,"Attempt":0,"Launch Time":1594091481095,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483015,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1564","Value":"6157","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"1500000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"25000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":96160,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":238,"Internal":true,"Count
 Fa
 iled 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":8503,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":503010000,"Value":2748563000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1596,"Value":6284,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":112849000,"Value":281593000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":260,"Value":984,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMem
 
ory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":260,"Executor Deserialize CPU 
Time":112849000,"Executor Run Time":1596,"Executor CPU Time":503010000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write Tim
 e":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":35,"Index":3,"Attempt":0,"Launch Time":1594091481091,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483016,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1566","Value":"7723","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"1875000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"31250","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":120200,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":300,"Internal":true,"Count
 F
 ailed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":10618,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":502908000,"Value":3251471000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1598,"Value":7882,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":62944000,"Value":344537000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":263,"Value":1247,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVM
 
emory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":263,"Executor Deserialize CPU 
Time":62944000,"Executor Run Time":1598,"Executor CPU Time":502908000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write Ti
 me":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":40,"Index":8,"Attempt":0,"Launch Time":1594091481112,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483016,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1562","Value":"9285","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"2250000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"37500","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":144240,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":362,"Internal":true,"Count
 F
 ailed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":12733,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":510597000,"Value":3762068000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1595,"Value":9477,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":69760000,"Value":414297000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":246,"Value":1493,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVM
 
emory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":246,"Executor Deserialize CPU 
Time":69760000,"Executor Run Time":1595,"Executor CPU Time":510597000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write Ti
 me":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":45,"Index":13,"Attempt":0,"Launch Time":1594091481129,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483024,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1564","Value":"10849","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"2625000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"43750","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":168280,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":424,"Internal":true,"Count
  Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":14848,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":495138000,"Value":4257206000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1595,"Value":11072,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":54222000,"Value":468519000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":221,"Value":1714,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePytho
 
nVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":221,"Executor Deserialize CPU 
Time":54222000,"Executor Run Time":1595,"Executor CPU Time":495138000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write
  Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":39,"Index":7,"Attempt":0,"Launch Time":1594091481109,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483024,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1566","Value":"12415","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"3000000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"50000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":192320,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":486,"Internal":true,"Count
 
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":16963,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":539451000,"Value":4796657000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1596,"Value":12668,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":64380000,"Value":532899000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":254,"Value":1968,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePython
 
VMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":254,"Executor Deserialize CPU 
Time":64380000,"Executor Run Time":1596,"Executor CPU Time":539451000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write 
 Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":33,"Index":1,"Attempt":0,"Launch Time":1594091481082,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483025,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1566","Value":"13981","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"3375000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"56250","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":216360,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":62,"Value":548,"Internal":true,"Count
 
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":19078,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":519178000,"Value":5315835000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1597,"Value":14265,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":54442000,"Value":587341000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":267,"Value":2235,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePython
 
VMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":267,"Executor Deserialize CPU 
Time":54442000,"Executor Run Time":1597,"Executor CPU Time":519178000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":62,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write 
 Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":34,"Index":2,"Attempt":0,"Launch Time":1594091481087,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483026,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1574","Value":"15555","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"3750000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"62500","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":240400,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":614,"Internal":true,"Count
 
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":21193,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":508433000,"Value":5824268000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1606,"Value":15871,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":69492000,"Value":656833000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":265,"Value":2500,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePython
 
VMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":265,"Executor Deserialize CPU 
Time":69492000,"Executor Run Time":1606,"Executor CPU Time":508433000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write 
 Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":43,"Index":11,"Attempt":0,"Launch Time":1594091481123,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483029,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1578","Value":"17133","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"4125000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"68750","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":264440,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":680,"Internal":true,"Count
  Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":23308,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":502120000,"Value":6326388000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1609,"Value":17480,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":48849000,"Value":705682000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":225,"Value":2725,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePytho
 
nVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":225,"Executor Deserialize CPU 
Time":48849000,"Executor Run Time":1609,"Executor CPU Time":502120000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write
  Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":41,"Index":9,"Attempt":0,"Launch Time":1594091481116,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483032,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1583","Value":"18716","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"4500000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"75000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":288480,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":746,"Internal":true,"Count
 
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":25423,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":489923000,"Value":6816311000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1614,"Value":19094,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":55787000,"Value":761469000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":240,"Value":2965,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePython
 
VMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":240,"Executor Deserialize CPU 
Time":55787000,"Executor Run Time":1614,"Executor CPU Time":489923000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write 
 Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":46,"Index":14,"Attempt":0,"Launch Time":1594091481132,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483037,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1587","Value":"20303","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"4875000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"81250","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":312520,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":812,"Internal":true,"Count
  Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":27538,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":490927000,"Value":7307238000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1618,"Value":20712,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":51464000,"Value":812933000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":218,"Value":3183,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePytho
 
nVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":218,"Executor Deserialize CPU 
Time":51464000,"Executor Run Time":1618,"Executor CPU Time":490927000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write
  Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":32,"Index":0,"Attempt":0,"Launch Time":1594091481077,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483037,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1587","Value":"21890","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"5250000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"87500","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":336560,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":878,"Internal":true,"Count
 
 Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":29653,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":496683000,"Value":7803921000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1619,"Value":22331,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":56827000,"Value":869760000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":271,"Value":3454,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePython
 
VMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":271,"Executor Deserialize CPU 
Time":56827000,"Executor Run Time":1619,"Executor CPU Time":496683000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write 
 Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":42,"Index":10,"Attempt":0,"Launch Time":1594091481120,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483043,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1593","Value":"23483","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"5625000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"93750","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":360600,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":944,"Internal":true,"Count
  Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":31768,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":508230000,"Value":8312151000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1624,"Value":23955,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":58152000,"Value":927912000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":229,"Value":3683,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePytho
 
nVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":229,"Executor Deserialize CPU 
Time":58152000,"Executor Run Time":1624,"Executor CPU Time":508230000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Write
  Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records 
Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated 
Blocks":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":2,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":44,"Index":12,"Attempt":0,"Launch Time":1594091481126,"Executor 
ID":"0","Host":"127.0.0.1","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1594091483043,"Failed":false,"Killed":false,"Accumulables":[{"ID":0,"Name":"duration","Update":"1593","Value":"25076","Internal":true,"Count
 Failed Values":true,"Metadata":"sql"},{"ID":1,"Name":"number of output 
rows","Update":"375000","Value":"6000000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":14,"Name":"number of output 
rows","Update":"6250","Value":"100000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Update":24040,"Value":384640,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Update":66,"Value":1010,"Internal":true,"Cou
 nt Failed 
Values":true},{"ID":69,"Name":"internal.metrics.resultSize","Update":2115,"Value":33883,"Internal":true,"Count
 Failed 
Values":true},{"ID":68,"Name":"internal.metrics.executorCpuTime","Update":498187000,"Value":8810338000,"Internal":true,"Count
 Failed 
Values":true},{"ID":67,"Name":"internal.metrics.executorRunTime","Update":1624,"Value":25579,"Internal":true,"Count
 Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Update":51988000,"Value":979900000,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Update":222,"Value":3905,"Internal":true,"Count
 Failed Values":true}]},"Task Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePyt
 
honVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task
 Metrics":{"Executor Deserialize Time":222,"Executor Deserialize CPU 
Time":51988000,"Executor Run Time":1624,"Executor CPU Time":498187000,"Peak 
Execution Memory":24040,"Result Size":2115,"JVM GC Time":66,"Result 
Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle 
Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait 
Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes 
Read":0,"Total Records Read":0,"Shuffle Push Read Metrics":{"Corrupt Merged 
Block Chunks":10,"Merged Fallback Count":2,"Merged Remote Blocks 
Fetched":10,"Merged Local Blocks Fetched":15,"Merged Remote Chunks 
Fetched":12,"Merged Local Chunks Fetched":10,"Merged Remote Bytes 
Read":100,"Merged Local Bytes Read":100}},"Shuffle Write Metrics":{"Shuffle 
Bytes Written":0,"Shuffle Wri
 te Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes 
Read":0,"Records Read":0},"Output Metrics":{"Bytes Written":0,"Records 
Written":0},"Updated Blocks":[]}}
 {"Event":"SparkListenerStageExecutorMetrics","Executor ID":"driver","Stage 
ID":2,"Stage Attempt ID":0,"Executor 
Metrics":{"JVMHeapMemory":213367864,"JVMOffHeapMemory":189011656,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":2133349,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":2133349,"OffHeapUnifiedMemory":0,"DirectPoolMemory":282024,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":13,"MinorGCTime":115,"MajorGCCount":4,"MajorGCTime":339}}
 {"Event":"SparkListenerStageExecutorMetrics","Executor ID":"0","Stage 
ID":2,"Stage Attempt ID":0,"Executor 
Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0}}
 {"Event":"SparkListenerStageCompleted","Stage Info":{"Stage ID":2,"Stage 
Attempt ID":0,"Stage Name":"foreach at <console>:26","Number of Tasks":16,"RDD 
Info":[{"RDD 
ID":10,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"18\",\"name\":\"mapPartitions\"}","Callsite":"foreach
 at <console>:26","Parent IDs":[9],"Storage Level":{"Use Disk":false,"Use 
Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of 
Partitions":16,"Number of Cached Partitions":0,"Memory Size":0,"Disk 
Size":0},{"RDD 
ID":8,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"0\",\"name\":\"DeserializeToObject\"}","Callsite":"foreach
 at <console>:26","Parent IDs":[7],"Storage Level":{"Use Disk":false,"Use 
Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of 
Partitions":16,"Number of Cached Partitions":0,"Memory Size":0,"Disk 
Size":0},{"RDD 
ID":6,"Name":"ParallelCollectionRDD","Scope":"{\"id\":\"1\",\"name\":\"WholeStageCodegen
 (2)\"}","Callsite":"foreach at <console>:26","Parent ID
 s":[],"Storage Level":{"Use Disk":false,"Use 
Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of 
Partitions":16,"Number of Cached Partitions":0,"Memory Size":0,"Disk 
Size":0},{"RDD 
ID":7,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"1\",\"name\":\"WholeStageCodegen
 (2)\"}","Callsite":"foreach at <console>:26","Parent IDs":[6],"Storage 
Level":{"Use Disk":false,"Use 
Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of 
Partitions":16,"Number of Cached Partitions":0,"Memory Size":0,"Disk 
Size":0},{"RDD ID":9,"Name":"SQLExecutionRDD","Callsite":"foreach at 
<console>:26","Parent IDs":[8],"Storage Level":{"Use Disk":false,"Use 
Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of 
Partitions":16,"Number of Cached Partitions":0,"Memory Size":0,"Disk 
Size":0}],"Parent 
IDs":[],"Details":"org.apache.spark.sql.Dataset.foreach(Dataset.scala:2862)\n$line19.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:26)\n$line1
 
9.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:30)\n$line19.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:32)\n$line19.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:34)\n$line19.$read$$iw$$iw$$iw$$iw.<init>(<console>:36)\n$line19.$read$$iw$$iw$$iw.<init>(<console>:38)\n$line19.$read$$iw$$iw.<init>(<console>:40)\n$line19.$read$$iw.<init>(<console>:42)\n$line19.$read.<init>(<console>:44)\n$line19.$read$.<init>(<console>:48)\n$line19.$read$.<clinit>(<console>)\n$line19.$eval$.$print$lzycompute(<console>:7)\n$line19.$eval$.$print(<console>:6)\n$line19.$eval.$print(<console>)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native
 
Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\nscala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)","Submission
 Time":1594091481040,"Completion Time":1594091483044,"A
 
ccumulables":[{"ID":68,"Name":"internal.metrics.executorCpuTime","Value":8810338000,"Internal":true,"Count
 Failed 
Values":true},{"ID":71,"Name":"internal.metrics.resultSerializationTime","Value":11,"Internal":true,"Count
 Failed 
Values":true},{"ID":74,"Name":"internal.metrics.peakExecutionMemory","Value":384640,"Internal":true,"Count
 Failed 
Values":true},{"ID":65,"Name":"internal.metrics.executorDeserializeTime","Value":3905,"Internal":true,"Count
 Failed Values":true},{"ID":14,"Name":"number of output 
rows","Value":"100000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":67,"Name":"internal.metrics.executorRunTime","Value":25579,"Internal":true,"Count
 Failed 
Values":true},{"ID":70,"Name":"internal.metrics.jvmGCTime","Value":1010,"Internal":true,"Count
 Failed Values":true},{"ID":1,"Name":"number of output 
rows","Value":"6000000","Internal":true,"Count Failed 
Values":true,"Metadata":"sql"},{"ID":69,"Name":"internal.metrics.resultSize","Value":33883,"Internal":true,"C
 ount Failed 
Values":true},{"ID":66,"Name":"internal.metrics.executorDeserializeCpuTime","Value":979900000,"Internal":true,"Count
 Failed 
Values":true},{"ID":0,"Name":"duration","Value":"25076","Internal":true,"Count 
Failed Values":true,"Metadata":"sql"}],"Resource Profile Id":0}}
 {"Event":"SparkListenerJobEnd","Job ID":2,"Completion Time":1594091483044,"Job 
Result":{"Result":"JobSucceeded"}}
 
{"Event":"org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionEnd","executionId":0,"time":1594091483045}
-{"Event":"SparkListenerApplicationEnd","Timestamp":1594091824231}
+{"Event":"SparkListenerApplicationEnd","Timestamp":1594091824231}

Review Comment:
   Missing new line at the end of file



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

[GitHub] [spark] otterc commented on a diff in pull request #36165: [SPARK-36620][SHUFFLE] Add Push Based Shuffle client side read metrics

Reply via email to