gabotechs commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2851354781
##########
datafusion/proto/src/physical_plan/to_proto.rs:
##########
@@ -256,6 +258,47 @@ pub fn serialize_physical_expr_with_converter(
codec: &dyn PhysicalExtensionCodec,
proto_converter: &dyn PhysicalProtoConverterExtension,
) -> Result<protobuf::PhysicalExprNode> {
+ // Check for DynamicFilterPhysicalExpr before snapshotting.
+ // We need to handle it before snapshot_physical_expr because snapshot()
+ // replaces the DynamicFilterPhysicalExpr with its inner expression.
+ if let Some(df) =
value.as_any().downcast_ref::<DynamicFilterPhysicalExpr>() {
Review Comment:
I see one of the main issues is how snapshotting works with dynamic
filtering.
In the physical expression trait:
```rust
...
/// A system or function that can only deal with a hardcoded set of
`PhysicalExpr` implementations
/// or needs to serialize this state to bytes may not be able to handle
these dynamic references.
/// In such cases, we should return a simplified version of the
`PhysicalExpr` that does not
/// contain these dynamic references.
...
fn snapshot(&self) -> Result<Option<Arc<dyn PhysicalExpr>>>;
```
This doesn't return a snapshot of the `DynamicFilterPhysicalExpr`, it
returns a snapshot to the inner computed physical expression, losing all the
information of the original `DynamicFilterPhysicalExpr`.
What we want in this PR, is to serialize is the actual
`DynamicFilterPhysicalExpr`, with all it's details, so probably `
snapshot_physical_expr(Arc::clone(value))?;` should not be getting called at
all for any expression, and we should be able to treat the dynamic filter as
just another serializable `PhysicalExpr`
##########
datafusion/proto/src/physical_plan/to_proto.rs:
##########
@@ -256,6 +258,47 @@ pub fn serialize_physical_expr_with_converter(
codec: &dyn PhysicalExtensionCodec,
proto_converter: &dyn PhysicalProtoConverterExtension,
) -> Result<protobuf::PhysicalExprNode> {
+ // Check for DynamicFilterPhysicalExpr before snapshotting.
+ // We need to handle it before snapshot_physical_expr because snapshot()
+ // replaces the DynamicFilterPhysicalExpr with its inner expression.
+ if let Some(df) =
value.as_any().downcast_ref::<DynamicFilterPhysicalExpr>() {
Review Comment:
I see one of the main issues is how snapshotting works with dynamic
filtering.
In the physical expression trait:
```rust
...
/// A system or function that can only deal with a hardcoded set of
`PhysicalExpr` implementations
/// or needs to serialize this state to bytes may not be able to handle
these dynamic references.
/// In such cases, we should return a simplified version of the
`PhysicalExpr` that does not
/// contain these dynamic references.
...
fn snapshot(&self) -> Result<Option<Arc<dyn PhysicalExpr>>>;
```
This doesn't return a snapshot of the `DynamicFilterPhysicalExpr`, it
returns a snapshot to the inner computed physical expression, losing all the
information of the original `DynamicFilterPhysicalExpr`.
What we want in this PR, is to serialize the actual
`DynamicFilterPhysicalExpr`, with all it's details, so probably `
snapshot_physical_expr(Arc::clone(value))?;` should not be getting called at
all for any expression, and we should be able to treat the dynamic filter as
just another serializable `PhysicalExpr`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
