Ted-Jiang opened a new pull request #1337:
URL: https://github.com/apache/arrow-datafusion/pull/1337
…r to improve locality
# Which issue does this PR close?
#1336
# Rationale for this change
``
plan1 -> shuffle wirte -> plan2
``
Make p1 and p2 run on the same executor may avoid sending intermediate
results.
Describe the solution you'd like
# What changes are included in this PR?
1. executors need record which jobs have been run
2. executors ask tasks with prefer job id
3. If prefer job already done, need send back to executor and executor
remove the record
Question 1:
if executor can not get prefer task ? will the task still hang there or
request other Job
only a draft, look forward for your opinions? 😊😊😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
