Robin-Everaars commented on issue #27609: URL: https://github.com/apache/datafusion/issues/27609#issuecomment-7044248757
Could #24891 be backported to branch-55 for the next 55.x patch release? It fixes #24883 and is still listed as Backport: TBD in #25758 and not yet in 55.3.0. The current branch-55 SpillPoolReader still waits on the head file. A Sail source build using DataFusion 55.2.0 reproduced a spill stall in 50/50 fresh processes: greedy 4 MiB pool, two partitions, batch size 64 and grouped COUNT(DISTINCT md5(...)) over 40,000 generated rows. Each run reached query collection, spilled and timed out at a 60-second process deadline. Changing only max_spill_file_size_bytes to 1 completed 50/50 with correct results while every run still spilled. Four unfinished spill files stayed unchanged between the 45-second and 55-second samples in every stalled run. This corroborates #24883 in a downstream runtime. We have not tested #24891 in that runtime yet. A branch-55 backport would let consumers take the fix without retaining a diagnostic one-byte spill-file limit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
