andygrove opened a new pull request, #2795: URL: https://github.com/apache/datafusion-ballista/pull/2795
# Which issue does this PR close? No single issue. Split out of #2792. Related: #2520 (performance review tracking). # Rationale for this change Under pull scheduling the scheduler binds tasks, marking them running, before tonic encodes the `PollWorkResult`. When the response exceeds the gRPC encoding limit (many tasks with large plans, such as broadcast readers listing every upstream location at 128 partitions), the executor gets an error, the tasks never run, and the job hangs. # What changes are included in this PR? The scheduler's poll handler sends only the tasks that fit under the gRPC encoding limit (`--grpc-server-max-encoding-message-size`) and reports the rest as killed, which re-queues their partitions without counting a failure. A single task larger than the limit fails its job with an error naming the setting to raise. Unit tests cover `split_tasks_to_fit`: tasks that fit are kept and the rest returned, everything under the limit is kept, and a task larger than the limit is returned. # Are there any user-facing changes? No config or API changes. A job that previously hung now runs, or fails with an error that names the gRPC message-size setting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
