andygrove commented on code in PR #2889:
URL:
https://github.com/apache/datafusion-ballista/pull/2889#discussion_r4607345316
##########
ballista/scheduler/src/state/cache_linker.rs:
##########
@@ -694,6 +756,51 @@ impl<T: 'static + AsLogicalPlan, U: 'static +
AsExecutionPlan> SchedulerState<T,
}
}
+ /// Drops the `DataFrame::cache()` data of `session_id`, which is being
+ /// removed. Its built caches are gone at once for new queries, and their
+ /// data is cleaned up once no running query reads it. Its builds still in
+ /// flight are cancelled, and what waits on them fails. Using the session
+ /// again builds its caches anew.
+ pub(crate) async fn remove_session_caches(
Review Comment:
One case I traced: if a session is removed while a query is still reading
its cache, and that query then loses the cache data, `rebuild_lost_inputs`
builds it again under the removed session. `new_context` succeeds for a removed
session, and `record_materialized_cache` writes a fresh entry for it. Nothing
removes that entry afterwards, so its stages stay pinned. Would it make sense
to fail that reader with `CACHE_DATA_LOST` instead, so a retry on the session
rebuilds as usual? Or drop the rebuilt entry once `provide_input` has handed it
over? `orphaned` alone can't tell the two cases apart, since `invalidate_build`
in the rebuild path orphans too. A test along the lines of
`removing_a_session_spares_running_reads_and_rebuilds_on_reuse` plus a
`lose_executor` would pin this down.
##########
ballista/scheduler/src/state/cache_linker.rs:
##########
@@ -694,6 +756,51 @@ impl<T: 'static + AsLogicalPlan, U: 'static +
AsExecutionPlan> SchedulerState<T,
}
}
+ /// Drops the `DataFrame::cache()` data of `session_id`, which is being
+ /// removed. Its built caches are gone at once for new queries, and their
+ /// data is cleaned up once no running query reads it. Its builds still in
+ /// flight are cancelled, and what waits on them fails. Using the session
+ /// again builds its caches anew.
+ pub(crate) async fn remove_session_caches(
+ &self,
+ session_id: &str,
+ sender: &EventSender<QueryStageSchedulerEvent>,
+ ) -> Result<()> {
+ // A build fails with what waits on it, held builds included, so look
+ // again each time rather than walk a list taken up front.
+ while let Some((build, key)) =
self.cache_linker.inflight_of(session_id) {
+ match self.cache_linker.cancel(&build) {
+ Some(held) => self.fail_pending(&held, SESSION_CLOSED,
sender).await,
Review Comment:
The tests cover a running build nicely. Could we add one with a nested
cache, like `cache_over("c2", cache_over("c1", aggregation_logical_plan(2)))`,
removed before `c1` finishes? That exercises this `Some(held)` branch, where
`c2`'s build is still held. It also checks that the loop ends with both keys
gone.
##########
docs/source/user-guide/tuning-guide.md:
##########
@@ -460,8 +460,10 @@ cached.collect().await?; // reads it
a cache it was built from, another way (a self-join of a cached frame, say),
fails instead with "lost DataFrame::cache() data; retry the query", and the
retry rebuilds the cache.
-- Cached data is not evicted yet: it is kept until it is found lost, so cache
- only what you reuse.
+- Cached data is not evicted yet: it is kept until it is found lost, or its
+ session is removed with a `RemoveSession` call, so cache only what you reuse.
Review Comment:
Could we say `RemoveSession` is a scheduler gRPC call that the client
libraries don't expose yet? Otherwise users may look for a method on
`SessionContext` that isn't there.
##########
ballista/scheduler/src/state/cache_linker.rs:
##########
@@ -694,6 +756,51 @@ impl<T: 'static + AsLogicalPlan, U: 'static +
AsExecutionPlan> SchedulerState<T,
}
}
+ /// Drops the `DataFrame::cache()` data of `session_id`, which is being
+ /// removed. Its built caches are gone at once for new queries, and their
+ /// data is cleaned up once no running query reads it. Its builds still in
+ /// flight are cancelled, and what waits on them fails. Using the session
+ /// again builds its caches anew.
+ pub(crate) async fn remove_session_caches(
+ &self,
+ session_id: &str,
+ sender: &EventSender<QueryStageSchedulerEvent>,
+ ) -> Result<()> {
+ // A build fails with what waits on it, held builds included, so look
+ // again each time rather than walk a list taken up front.
+ while let Some((build, key)) =
self.cache_linker.inflight_of(session_id) {
+ match self.cache_linker.cancel(&build) {
+ Some(held) => self.fail_pending(&held, SESSION_CLOSED,
sender).await,
+ None => {
+ self.fail_cache_job(&build, Some(&key), SESSION_CLOSED,
sender)
Review Comment:
Small question here. If another submission on the same session is between
`track` and `register` when this loop runs, the build isn't held yet. So we
abort its claim and post `JobCancel` for a job that hasn't been queued. The
submitter then registers and submits the build anyway, and its read fails with
a cache-miss error rather than "its session was closed". It's a narrow race, so
I'm fine leaving it. Maybe worth a comment, or a check in `register` that the
build is still in flight?
##########
ballista/scheduler/src/scheduler_server/grpc.rs:
##########
@@ -444,9 +444,12 @@ impl<T: 'static + AsLogicalPlan, U: 'static +
AsExecutionPlan> SchedulerGrpc
request: Request<RemoveSessionParams>,
) -> Result<Response<RemoveSessionResult>, Status> {
let session_params = request.into_inner();
+ let sender = self
+ .query_stage_event_loop
+ .get_sender()
Review Comment:
Tiny thing. `get_sender()` errors out before the session is removed, which
slightly undercuts the "removed even if dropping its caches fails" doc on
`SchedulerState::remove_session`. Same in `flight_sql.rs`. It's unlikely in
practice, so feel free to ignore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]