SEPURI-SAI-KRISHNA commented on issue #11406: URL: https://github.com/apache/seatunnel/issues/11406#issuecomment-6595361528
@DanielLeens thanks, agreed on the three surfaces and on keeping v1 to surface 1, with #11962 and #11897 covering the other two. I will open the focused PR you describe: wrap the restore of connector-owned state at `SourceSplitEnumeratorTask#restoreState`, `RestoredSplitOperation`, `SinkFlowLifeCycle#restoreState` and `SinkAggregatedCommitterTask#restoreState` in that one message format, keep the original exception as the cause, and change nothing in the bytes on disk. The test will restore the before and after `InvalidClassException` stream through those paths and assert the message. One point to settle first. On current `dev` the checkpoint id does not reach these call sites. `CheckpointCoordinator` knows it (it logs the id when it loads the latest completed checkpoint), but `NotifyTaskRestoreOperation` carries only the task location and the `ActionSubtaskState` list, and `RestoredSplitOperation` carries only the split bytes. Job id, pipeline id and the action are available from `TaskLocation` and `ActionStateKey`. The task runs `restoreState` asynchronously and reports a failure back as a `CheckpointErrorReportOperation` holding only the error message, and `CheckpointCoordinator#reportCheckpointErrorFromTask` then fails the pipeline with `CHECKPOINT_INSIDE_ERROR`. Two ways to include the checkpoint id: 1. Add it to `NotifyTaskRestoreOperation` and pass it down to `restoreState`. That changes the operation's wire format and the `Stateful#restoreState` signature, and still leaves `RestoredSplitOperation` without it. 2. Keep the task side message without the checkpoint id, and have the coordinator add the id of the checkpoint it restored from when it turns that report into the pipeline failure during restore. I lean to 2 because it keeps the slice free of wire format and interface changes. Until the version metadata exists, the message would say the producing version is unknown and leave out the readable range. Does that match what you had in mind? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
