andygrove commented on code in PR #2867:
URL:
https://github.com/apache/datafusion-ballista/pull/2867#discussion_r4579320693
##########
docs/source/upgrading/56.0.0.md:
##########
@@ -218,9 +218,9 @@ that holds data per partition from one that does not, such
as a `FilterExec`. An
interior node of your own that holds data per partition still returns wrong
results in a split stage unless a resolver restricts it.
-The error names the node and what to implement. Under push-staged scheduling
-the job's status says only that a task could not be prepared; the reason is in
-the scheduler's log. To make nodes of your own restrictable, see
+The error, which the job's status carries under both push-staged and
+pull-staged scheduling, names the node and what to implement. To make nodes of
+your own restrictable, see
[Custom Physical Plan
Nodes](../user-guide/extending-components.md#custom-physical-plan-nodes).
Review Comment:
Following up on the earlier note about the old message. Could you add a
sentence after this line along the lines of "A job whose task an executor
rejects now fails with `task was rejected by executor <id>` instead of `task
could not be prepared or was rejected by executor`, so update anything that
matches on the old text"? That text shipped in 55.0.0, so someone may be
alerting on it.
##########
ballista/scheduler/src/state/task_manager.rs:
##########
@@ -1064,40 +1064,40 @@ impl<T: 'static + AsLogicalPlan, U: 'static +
AsExecutionPlan> TaskManager<T, U>
/// Launch the given tasks on the specified executor.
///
- /// Returns the jobs that cannot run: those the executor rejected, and
- /// those whose task definitions could not be prepared here. Tasks of the
- /// latter were never sent anywhere and will never report a status, so the
- /// caller has to fail the job and refund the tasks' slots, exactly as for
a
- /// rejection. `Err` is only for the launch RPC itself failing, which says
- /// the executor is sick, not the plan.
+ /// Returns the jobs that cannot run: those whose task definitions could
not
+ /// be prepared here, each with the preparation error, and those the
+ /// executor rejected. Tasks of the former were never sent anywhere and
will
+ /// never report a status, so the caller has to fail the job and refund the
+ /// tasks' slots, exactly as for a rejection. `Err` is only for the launch
+ /// RPC itself failing, which says the executor is sick, not the plan.
pub(crate) async fn launch_multi_task(
&self,
executor: &ExecutorMetadata,
tasks: Vec<Vec<TaskDescription>>,
executor_manager: &ExecutorManager,
- ) -> Result<HashSet<JobId>> {
+ ) -> Result<(HashMap<JobId, String>, HashSet<JobId>)> {
let mut multi_tasks = vec![];
- let mut failed_jobs = HashSet::new();
+ let mut unprepared_jobs = HashMap::new();
for stage_tasks in tasks {
let job_id = stage_tasks.first().map(|task|
task.key.job_id.clone());
match self.prepare_multi_task_definition(stage_tasks) {
Ok(stage_tasks) => multi_tasks.extend(stage_tasks),
Err(e) => {
error!("Fail to prepare task definition: {e:?}");
- failed_jobs.extend(job_id);
+ unprepared_jobs.extend(job_id.map(|job_id| (job_id,
e.to_string())));
}
}
}
- if !multi_tasks.is_empty() {
- failed_jobs.extend(
- self.launcher
- .launch_tasks(executor, multi_tasks, executor_manager)
- .await?,
- );
- }
+ let rejected_jobs = if multi_tasks.is_empty() {
+ HashSet::new()
+ } else {
Review Comment:
Not something this PR introduced, just noticed while reading. If the launch
RPC fails, `launch_multi_task` returns early through `?` and the
`unprepared_jobs` it already collected are dropped. The executor-lost path
should reschedule them, so it seems harmless. Would it be worth a follow-up
issue to carry them out on the error path too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]