NoahKusaba opened a new pull request, #2752:
URL: https://github.com/apache/datafusion-ballista/pull/2752
# Which issue does this PR close?
No issue yet. Follows the discussion on #2638 about how several table format
codecs (Iceberg, Hudi, Delta, ...) can share one Ballista deployment.
# Rationale for this change
DataFusion has `ComposedPhysicalExtensionCodec` but no logical counterpart.
Without one, each format crate has to wrap the codec it's given and invent its
own framing, and protobuf has no type identity, so a codec that "tries to
decode" bytes another codec wrote can decode them into the wrong thing.
`ComposedLogicalExtensionCodec` uses the same approach as DataFusion's
physical codec: every payload records the position of the codec that encoded
it, and only that codec decodes it.
# What changes are included in this PR?
- `ballista_core::serde::ComposedLogicalExtensionCodec`:
- payloads use the same `DataEncoderTuple { encoder_position, blob }`
layout as DataFusion's `ComposedPhysicalExtensionCodec`;
- on encode, codecs are tried in order and the first that succeeds is
recorded;
- a node that encodes to zero bytes stays with its codec;
- functions (UDF, UDAF, UDWF, higher-order) and file formats go to the
first codec that writes bytes, otherwise they are sent by name, so a later
codec can still encode them;
- when no codec handles a node, every codec's error is reported;
- a malformed payload or an out-of-range position gives an error
explaining that every node must compose the same codecs in the same order.
- Docs: a "Combining Codecs" section in `extending-components.md`, pairing
it with DataFusion's `ComposedPhysicalExtensionCodec` and noting that codec's
limits: only its first codec can encode a function, and it doesn't route custom
physical expressions, window functions or higher-order functions. Table scans,
such as Iceberg's, only need plan encode/decode, which it supports.
The module is self-contained, with a TODO to move it into `datafusion-proto`
beside `ComposedPhysicalExtensionCodec`.
# Are there any user-facing changes?
Yes, a new public `ComposedLogicalExtensionCodec` and docs. Nothing existing
changes; no `.proto` files are touched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
