milenkovicm commented on issue #17297:
URL: https://github.com/apache/datafusion/issues/17297#issuecomment-3217961595
Edited issue description with another option:
Another alternative would be to have a configuration option, like
`datafusion.execution.local_cache` which would decide should it use current
cache logic or return new logical plan element like `LogicalPlan::Cache { id:
string, lineage: LogicalPlan }`
``` rust
pub async fn cache(self) -> Result<DataFrame> {
if `datafusion.execution.local_cache` == true {
let context =
SessionContext::new_with_state((*self.session_state).clone());
// The schema is consistent with the output
let plan = self.clone().create_physical_plan().await?;
let schema = plan.schema();
let task_ctx = Arc::new(self.task_ctx());
let partitions = collect_partitioned(plan, task_ctx).await?;
let mem_table = MemTable::try_new(schema, partitions)?;
context.read_table(Arc::new(mem_table))
} else {
let lineage = self... // get current logical plan from df up to this point
Ok(
LogicalPlan::Cache(Cache { id : uuid(), lineage)
)
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
