milenkovicm opened a new pull request, #1461:
URL: https://github.com/apache/datafusion-ballista/pull/1461
# Which issue does this PR close?
Closes #.
# Rationale for this change
Pull based scheduling strategy has been use as default since version 43,
without any valid reason (probably as it was the first one in the enum).
Running some test its apparent that Push based strategy provides much lover
latency on TPCH queries (1.5-4x with 2.5x being average)
| PULL (ms) | PUSH (ms) | Difference
-- | -- | -- | --
Q1 | 953 | 549 | 1.7
Q2 | 2513.9 | 1061.3 | 2.4
Q3 | 2189 | 745.5 | 2.9
Q4 | 1779 | 528 | 3.4
Q5 | 4341.3 | 1687.9 | 2.6
Q6 | 420 | 212.5 | 2
Q7 | 4556.7 | 1989.4 | 2.3
Q8 | 5075.8 | 2225.9 | 2.3
Q9 | 5277.4 | 2407.6 | 2.2
Q10 | 1886 | 1157.1 | 1.6
Q11 | 1674 | 638.1 | 2.6
Q12 | 1553.2 | 627.9 | 2.5
Q13 | 1456.5 | 535.8 | 2.7
Q14 | 939 | 319.8 | 2.9
Q15 | 1470.8 | 636.2 | 2.3
Q16 | 1601.3 | 548.1 | 2.9
Q17 | 4549.3 | 1156.2 | 3.9
Q18 | 4065.9 | 1271.1 | 3.2
Q19 | 1073.5 | 547.4 | 2
Q20 | 1371.5 | 841.5 | 1.6
Q21 | 7436.2 | 2093.6 | 3.6
Q22 | 1268.6 | 444.4 | 2.9
<img width="610" height="379" alt="image"
src="https://github.com/user-attachments/assets/bf924211-2373-47ce-8127-0e9692163741";
/>
with external benchmarks not changing default policy, I would suggest
changing default to push based.
# What changes are included in this PR?
Change default scheduling policy to push based
- changing default policy to push based
- update executor logging to show policy at startup
# Are there any user-facing changes?
Yes, users which rely on Pull based policy (for any reason) should configure
scheduler and executors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
