Robin-Everaars commented on issue #27609:
URL: https://github.com/apache/datafusion/issues/27609#issuecomment-7044248757

   Could #24891 be backported to branch-55 for the next 55.x patch release? It 
fixes #24883 and is still listed as Backport: TBD in #25758 and not yet in 
55.3.0. The current branch-55 SpillPoolReader still waits on the head file.
   
   A Sail source build using DataFusion 55.2.0 reproduced a spill stall in 
50/50 fresh processes: greedy 4 MiB pool, two partitions, batch size 64 and 
grouped COUNT(DISTINCT md5(...)) over 40,000 generated rows. Each run reached 
query collection, spilled and timed out at a 60-second process deadline. 
Changing only max_spill_file_size_bytes to 1 completed 50/50 with correct 
results while every run still spilled. Four unfinished spill files stayed 
unchanged between the 45-second and 55-second samples in every stalled run.
   
   This corroborates #24883 in a downstream runtime. We have not tested #24891 
in that runtime yet. A branch-55 backport would let consumers take the fix 
without retaining a diagnostic one-byte spill-file limit.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to