avantgardnerio commented on code in PR #3907:
URL: https://github.com/apache/arrow-datafusion/pull/3907#discussion_r1003724353
##########
datafusion/proto/src/logical_plan.rs:
##########
@@ -93,6 +100,22 @@ pub trait AsLogicalPlan: Debug + Send + Sync + Clone {
Self: Sized;
}
+pub trait PhysicalExtensionCodec: Debug + Send + Sync {
Review Comment:
It was moved out of Ballista and into datafusion. I would think being a
libarary, DataFusion would have to represent both physical and logical plans in
memory, but not serialize them. For reasons which I am not yet familiar,
DataFusion contains the code to serialize logical plans, but Ballista is in
charge of serializing physical plans, so this PhysicalExtensionCodec was
defined there, and this PR moves it into DataFusion.
As for the reason why, libraries like delta-rs will want to extend these
traits (TableProviderFactory, LogicalExtensionCodec, and
PhysicalExtensionCodec) to provide their own tables to executables up the chain
like Ballista, and to do this, it needs to be defined below them in the
dependency tree so they can implement it... see
https://github.com/spaceandtimelabs/delta-rs/blob/379558798b644d3a269514a99b322850d5d4b5a6/rust/src/delta_datafusion.rs#L754
for an example.
Eventually this series of PRs will end in a Ballista PR with a `deltalake`
feature and if specified it will bring in delta's implementation of the above
traits and make delta tables registerable in SQL at runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
