alamb commented on code in PR #8510:
URL: https://github.com/apache/arrow-datafusion/pull/8510#discussion_r1425287458
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -1271,10 +1271,10 @@ impl DataFrame {
/// ```
pub async fn cache(self) -> Result<DataFrame> {
let context =
SessionContext::new_with_state(self.session_state.clone());
- let mem_table = MemTable::try_new(
- SchemaRef::from(self.schema().clone()),
- self.collect_partitioned().await?,
- )?;
+ // The schema is consistent with the output
+ let physical_plan = self.clone().create_physical_plan().await?;
+ let mem_table =
+ MemTable::try_new(physical_plan.schema(),
self.collect_partitioned().await?)?;
Review Comment:
FWIW this now runs the planner twice -- we could make it more efficient by
calling
[`collect_partitioned()`](https://docs.rs/datafusion/latest/datafusion/physical_plan/fn.collect_partitioned.html)
directly on the physical plan rather than `Self::collect_partitioned` that
will replan (and recollect)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
