MarcusKainth commented on code in PR #4204:
URL: https://github.com/apache/iggy/pull/4204#discussion_r4040014649
##########
core/connectors/sdk/src/lib.rs:
##########
@@ -157,6 +157,26 @@ pub enum Payload {
}
impl Payload {
+ /// The `Schema` describing this payload's variant.
+ ///
+ /// Not the same thing as `StreamDecoder::schema`, which names the wire
+ /// format a decoder reads rather than the variant it hands back: the Avro
+ /// and FlatBuffer decoders return `Payload::Json` whenever
`extract_as_json`
+ /// is set, and the Proto decoder returns `Payload::Json` or `Payload::Raw`
+ /// depending on the path it takes. A transform may change the variant
again
+ /// after that. Anything tagging a payload for transport has to read the
tag
+ /// off the payload it actually holds.
+ pub const fn schema(&self) -> Schema {
+ match self {
+ Payload::Json(_) => Schema::Json,
+ Payload::Raw(_) => Schema::Raw,
+ Payload::Text(_) => Schema::Text,
+ Payload::Proto(_) => Schema::Proto,
Review Comment:
Correct, and it is a regression against master, not just a round-trip gap:
on a `schema = "json"` stream with a `proto_convert` transform the sink got
`Payload::Json` on master and `Payload::Raw` on `31c0773`. ClickHouse skips
those rows; Delta, Doris and Iceberg fail the batch.
`try_into_payload` had two callers meaning different things by
`Schema::Proto` — the sink SDK meant the payload's variant,
`runtime/src/source.rs:609` means the wire format a plugin sent. Split in
d8a8ece: the sink rebuilds through `Payload::try_from_schema`, so all six
variants keep theirs. Elasticsearch, Meilisearch and the ClickHouse string
passthrough now take proto text as text; the JSON-only sinks still reject it,
noted in Compatibility.
One correction: `json_to_protobuf` only falls back to `Payload::Proto` with
no descriptor (`proto_convert.rs:656`) or non-object JSON (`:285`); with both
it returns `Payload::Raw(binary)`, which round-trips.
Regression tests in d8a8ece: the round-trip test no longer exempts Proto,
and a new Elasticsearch integration test drives a `proto_convert` pipeline
through a real runtime and container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
