sandugood opened a new pull request, #1853:
URL: https://github.com/apache/datafusion-ballista/pull/1853
# Which issue does this PR close?
Closes #1851.
# Rationale for this change
In the current implementation there is a problem - in the
`PartitionLocation` that is used in each shuffle operation (for each partition
in the previous stage) there was `executor_metadata` field (which is of
`ExecutorMetadata` type) filled with unnecessary info, because it did not add
up any information that could be used by `ShuffleReaderExec` to extract
partition (or i.e resume the execution from a failed stage)
# What changes are included in this PR?
`ExecutorMetadata` was decoupled from `PartitionLocation` and now it is used
as a separate struct for:
- registering executor
- in heartbeats
- REST API info fetch
`PartitionLocation` now is only exposed to the `executor_id`, host and
`port`, which is sufficient for fetching needed partitions by the
`ShuffleReaderExec`
This way we can save up a lot of space and data-transfer during each shuffle
operation (potentially removing possibility of `Scheduler`'s OOM errors and
improving speed of queries with lots of partitions)
# Are there any user-facing changes?
Yes.
In the REST API interface `ExecutorResponse` doesnt contain nested struct
with `Executor`'s hardware and OS info. It was flattened.
# Potential follow-up:
- check the TUI REST API integration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
