LiaCastaneda commented on issue #22758:
URL: https://github.com/apache/datafusion/issues/22758#issuecomment-4958607962
> I think that's what's in my OomGuard PR. It's definitely part of what we
run at Coralogix. If the tokio task that gets killed is one we spawned, we know
the query ID. Unfortunately operators like RepartitionExec spawn their own
tasks with their own default panic handlers and those sadly just die.
I think it uses a `thread_local` that doesn't keep an ID (the first PR does
keep a `context_id`, but it's thread local too). Given that multiple threads
can run the tasks of a given query, I was imagining the ID granularity would
need to be per task, all tasks of a given query carrying the same ID? Then you
can use a global map like the one in your first PR, but keyed by a query ID
instead of a context ID/thread ID. With per query IDs you could then rank
queries by memory and cancel the perpetrator.
> Unfortunately operators like RepartitionExec spawn their own tasks with
their own default panic handlers and those sadly just die.
If a task from `RepartitionExec` panics, the error would get
[propagated](https://github.com/apache/datafusion/blob/65f9726f753bea6a13ca7f7356eede0127aa4b87/datafusion/physical-plan/src/repartition/mod.rs#L1836)
to the rest of tasks and the query will fail with a generic error, but it's
true it would not be possible to know if it errored because of an `alloc`
failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
