Re: [PR] chore: Reimplement ShuffleWriterExec using interleave_record_batch [datafusion-comet]

[via GitHub]

Kontinuation commented on PR #1511:
URL: 
https://github.com/apache/datafusion-comet/pull/1511#issuecomment-2755868919

   Reran TPC-H SF=100 on an m7i.4xlarge instances with `master = local[8]`, 
Most of the disk accesses hit the OS cache so the slow EBS didn't affect the 
query performance too much. The unstable reallocation performance problem we've 
seen on macOS does not show up and the performance of Q18 is quite stable.
   
   | On-heap size | Off-heap size | Spark 3.5.4 | Comet main | Comet PR | Bar 
plot |
   |--|--|--|--|--|--|
   | 3g | 3g | 1157 s | 579 s | 555 s | 
![tpch_queries_compare_3g](https://github.com/user-attachments/assets/d676a0b4-5084-46a0-9b6f-b439eaa8c8b1)
 |
   | 3g | 5g | 1150s | 573s | 556s | 
![tpch_queries_compare_5g](https://github.com/user-attachments/assets/efce3acd-fc22-4ea7-977e-9afcf405d125)
 |
   | 3g | 8g | 1158s | 552s | 549s | 
![tpch_queries_compare_8g](https://github.com/user-attachments/assets/204051d0-d6af-4c9f-b626-ab567425dd63)
 |
   | 3g | 16g | 1153s | 535s | 535s | 
![tpch_queries_compare_16g](https://github.com/user-attachments/assets/c6c94779-de47-4a2b-ae05-c679dba3523a)
 |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org