kosiew opened a new issue, #20165: URL: https://github.com/apache/datafusion/issues/20165
Add protobuf serialization and deserialization support for the unified `CastExpr` (logical) and `CastColumnExpr` (physical) to enable distributed execution, Flight SQL, CLI round‑trips, and cross-process plan exchange. ## Scope Implement complete serialization/deserialization round‑trip support in `datafusion/proto/` for both logical and physical cast expressions. ### Tasks **Logical Cast Serialization:** - [ ] Update `datafusion/proto/proto/logical_plan.proto` to support enhanced `CastExpr` - [ ] Add proto serde for enhanced `CastExpr` in `datafusion/proto/src/logical_plan.rs` - [ ] Implement round-trip tests for logical cast expressions **Physical Cast Serialization:** - [ ] Add proto message for `CastColumnExpr` in `datafusion/proto/proto/physical_plan.proto` - [ ] Implement serialization logic (`PhysicalExpr` → protobuf) - [ ] Implement deserialization logic (protobuf → `PhysicalExpr`) - [ ] Add round-trip tests for `CastColumnExpr` variants - [ ] Test with distributed/Flight SQL scenarios **Integration & Testing:** - [ ] Validate round-trip equality (expr → proto → expr) - [ ] Test with struct casting scenarios - [ ] Ensure backward compatibility (if applicable) - [ ] Add comprehensive proto round-trip test suite ### Files/Modules - `datafusion/proto/proto/logical_plan.proto` (logical cast message) - `datafusion/proto/proto/physical_plan.proto` (physical cast message) - `datafusion/proto/src/logical_plan.rs` (logical cast serde) - `datafusion/proto/src/physical_expr.rs` (physical cast serde) - Proto-related integration tests ### Key Features - **Lossless round-trip:** Serialized and deserialized expressions are equivalent - **Struct awareness:** Serialization preserves struct field casting semantics - **Nullability propagation:** Field-level nullability metadata survives round-trip - **Backward compatibility:** Graceful handling of older proto versions (if applicable) - **Clear error messages:** Helpful errors when deserialization fails ### Acceptance Criteria ✅ Protobuf messages defined for both logical and physical cast expressions ✅ Serialization/deserialization code is complete and tested ✅ Round-trip tests pass for all `CastExpr` and `CastColumnExpr` variants ✅ Struct casting scenarios work end-to-end with serialization ✅ Code compiles and all proto tests pass ✅ Documented any proto breaking changes or version notes ✅ Integration tests validate distributed/Flight SQL compatibility ## Notes - **Scope:** This PR enables distributed execution, Flight SQL, and CLI serialization for the cast expressions introduced in #20164. - **Breaking changes:** Proto updates may require versioning or migration notes if deployed to existing systems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
