jizezhang commented on code in PR #18688:
URL: https://github.com/apache/datafusion/pull/18688#discussion_r2548318824
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -2344,14 +2348,24 @@ impl DataFrame {
/// # }
/// ```
pub async fn cache(self) -> Result<DataFrame> {
- let context =
SessionContext::new_with_state((*self.session_state).clone());
- // The schema is consistent with the output
- let plan = self.clone().create_physical_plan().await?;
- let schema = plan.schema();
- let task_ctx = Arc::new(self.task_ctx());
- let partitions = collect_partitioned(plan, task_ctx).await?;
- let mem_table = MemTable::try_new(schema, partitions)?;
- context.read_table(Arc::new(mem_table))
+ if let Some(cache_factory) = self.session_state.cache_factory() {
+ let new_plan =
+ cache_factory.create(self.plan, self.session_state.as_ref())?;
+ Ok(Self {
+ session_state: self.session_state,
+ plan: new_plan,
+ projection_requires_validation:
self.projection_requires_validation,
+ })
Review Comment:
Thanks for pointing out, just updated. For my learning, is the idea that
`projection_requires_validation` should be true in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
