akshaychitneni commented on code in PR #2889:
URL: 
https://github.com/apache/datafusion-ballista/pull/2889#discussion_r4634119997


##########
ballista/scheduler/src/state/cache_linker.rs:
##########
@@ -694,6 +756,51 @@ impl<T: 'static + AsLogicalPlan, U: 'static + 
AsExecutionPlan> SchedulerState<T,
         }
     }
 
+    /// Drops the `DataFrame::cache()` data of `session_id`, which is being
+    /// removed. Its built caches are gone at once for new queries, and their
+    /// data is cleaned up once no running query reads it. Its builds still in
+    /// flight are cancelled, and what waits on them fails. Using the session
+    /// again builds its caches anew.
+    pub(crate) async fn remove_session_caches(

Review Comment:
   Good catch, thanks. Fixed by failing the reader: removing a session now 
marks the queries reading its caches, and a cache one of them loses afterwards 
is not rebuilt. The query fails with `lost DataFrame::cache() data after its 
session was closed ... retry the query`, so a retry on the session rebuilds as 
usual, while queries started after the removal still rebuild normally. I did 
not drop the rebuilt entry instead, as a rebuild can also rebuild the caches 
inside it. Added 
`removing_a_session_fails_a_later_rebuild_for_its_running_reads` (your 
suggested test plus `lose_executor`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to