DanielLeens commented on issue #11406: URL: https://github.com/apache/seatunnel/issues/11406#issuecomment-6595080503
@SEPURI-SAI-KRISHNA thanks, this is exactly the missing piece: a measured failure instead of a hypothetical one. I checked the paths on current `dev` and want to refine one point before we pick a slice. **The two worlds are layered, not parallel.** `AbstractCheckpointStorage` does use `ProtoStuffSerializer`, but what it stores for connectors is `ActionSubtaskState.state`, a `List<byte[]>`. Those bytes are opaque to protostuff; they are produced and later read by the connector's own serializer (`DefaultSerializer` by default via `SeaTunnelSource#getSplitSerializer` / `#getEnumeratorStateSerializer`; Firebase is the only source overriding both). So there are really three surfaces: 1. **Connector-owned state** inside those byte arrays (split, enumerator, sink writer/commit state). 2. **Engine-owned checkpoint fields** encoded by protostuff (the silent-default case you describe). #11962 is already proposing an explicit engine wire version for savepoints. 3. **Persisted job plan objects**, such as `ChainedMapTransform`. It is not checkpoint state; it lives in the serialized logical DAG. Your #12804 / #12938 / #12990 example is real, but it belongs to this surface. **Proposed v1 scope:** surface 1 only. Surfaces 2 and 3 stay with #11962 and the upgrade-compatibility STIP (#11897), and get cross-referenced, not duplicated. **Where the check happens:** the engine call sites that turn those bytes back into objects. Today they wrap failures with `sneaky(...)` and rethrow the raw cause, e.g. `SourceSplitEnumeratorTask#restoreState`, `RestoredSplitOperation`, `SinkFlowLifeCycle` and `SinkAggregatedCommitterTask`. So the operator currently sees a bare `InvalidClassException` with no job, action or checkpoint context. **Proposed error shape (one format for every connector):** ``` Cannot restore state of action '<action name>' (job <id>, pipeline <n>, checkpoint <id>): state produced by version <v|unknown> is not readable by this connector (readable: <range>). See the upgrade notes for <connector>. Cause: <original exception> ``` **Most useful first slice:** a PR that adds only this wrapping at those call sites, with no change to the bytes on disk. It is fully backward compatible, gives operators a consistent message right away, and is the hook where the version comparison plugs in later. Your before/after `InvalidClassException` stream is a good fixture for its test. If that sounds right, please open that focused PR. In parallel I would write the four agreed points plus this scope and error shape into a short STIP draft and start a dev@ thread, because the producer/readable version metadata itself touches the `Serializer` SPI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
