[GitHub] [spark] DenineLu commented on pull request #35806: [SPARK-38505][SQL] Make partial aggregation adaptive

[GitBox]

DenineLu commented on PR #35806:
URL: https://github.com/apache/spark/pull/35806#issuecomment-1340951402

   Thanks for your reply, I have provided two graphs which show that the 
HashAgg node shortens the execution time, but the whole stage becomes slower. I 
guess maybe because of this code, HashAgg's time statistics are not accurate 
anymore.
   ```  
   |if (!$initAgg) {
   |  $initAgg = true;
   |  $createFastHashMap
   |  $addHookToCloseFastHashMap
   |  $hashMapTerm = $thisPlan.createHashMap();
   |  long $beforeAgg = System.nanoTime();
   |  $doAggFuncName();
   |  $aggTime.add((System.nanoTime() - $beforeAgg) / $NANOS_PER_MILLIS);
   |  $shouldStopCheckCode
   |}
   |// This function cannot be counted in time, but will be executed frequently
   |$childDoAgg
   ```
   
   <img width="426" alt="not_refine" 
src="https://user-images.githubusercontent.com/37650442/206184902-af8a794a-8529-4527-8c9c-da24aac734ef.png";>
   The top picture does not add this patch, the bottom picture adds this patch.
   <img width="419" alt="refine" 
src="https://user-images.githubusercontent.com/37650442/206185581-7f692721-6176-4709-b7f3-d1b63fa04068.png";>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org