NoahKusaba opened a new issue, #3002:
URL: https://github.com/apache/iceberg-rust/issues/3002
### Is your feature request related to a problem or challenge?
A distributed query engine plans on one node and executes on others, so
physical
plans must survive serialization. `project_with_partition` injects a
`PartitionExpr` into the projection for partitioned writes, and that
expression
cannot make the trip.
It wraps a `PartitionValueCalculator`, which is a live non-serializable
state. The two
things that are serializable (the `PartitionSpec` and the table schema) are
consumed at construction and not retained, so nothing on the expression says
what produced it.
The spec alone is not enough to rebuild the calculator: `PartitionSpec`
stores
only `spec_id` and `fields`, referring to columns by `source_id`. The schema
is
what resolves those ids to real columns and determines the partition type.
Reaching for the schema elsewhere at decode time isn't possible either:
`PhysicalExtensionCodec::try_encode_expr` / `try_decode_expr` receive only
the
expression, with no tab
### Describe the solution you'd like
Retain both inputs on `PartitionExpr` and expose them:
- `try_new(partition_spec, table_schema)` replacing the private
`new(calculator, spec)`, constructing the calculator internally so the
retained inputs and the calculator cannot drift apart.
- `partition_spec()` and `table_schema()` accessors.
Additive: `project_with_partition` keeps its signature, and `new` was
private,
so no existing caller changes.
### Willingness to contribute
I can contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
