andygrove commented on code in PR #2813: URL: https://github.com/apache/datafusion-ballista/pull/2813#discussion_r4519569617
########## docs/source/upgrading/56.0.0.md: ########## @@ -196,6 +196,28 @@ cannot re-run another job's output, so a caller must fail them (the scheduler posts `JobRunningFailed`). A caller that only needs the tasks to cancel can take the first element. +#### Partition locations hold an `ExecutorEndpoint` + +A partition location only has to say where to fetch from, so +`PartitionLocation::executor_meta` and +`PipelinedPartitionLocation::executor_meta` are now `Arc<ExecutorEndpoint>`, a +new type with the executor's `id`, `host` and `port`, shared between locations +rather than copied into each one. `ExecutorMetadata` is unchanged, and is still +what the scheduler records about a registered executor. To build an endpoint +from one, use `Arc::new(ExecutorEndpoint::from(&metadata))`. Code that read +`grpc_port`, `specification` or `os_info` from a location should look up the +`ExecutorMetadata` by `id` instead. + +Shuffle reader plan nodes now list each executor once and their locations refer +to it by position. This changes the wire format between schedulers and +executors, so `BALLISTA_PROTOCOL_VERSION` is now `4` and they must be upgraded +together. + +In the protobuf definitions, the `ExecutorMetadata` message is renamed +`ExecutorEndpoint` and loses `grpc_port`, `specification` and `os_info`. Job +statuses still set `executor_meta` on every location, so clients that read its +`id`, `host` or `port` do not need to change. Review Comment: Thanks @milenkovicm! I think it's worth keeping. The paragraph above covers the Rust API and the wire version, but this one is for anyone using the generated proto types directly, like `ballista_pb2.ExecutorMetadata` in Python or a custom gRPC client polling job status. The last sentence is mostly there to tell those folks they probably don't need to change anything. Happy to trim it in a follow-up if you think it's noise though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
