nzw921rx commented on PR #10306: URL: https://github.com/apache/seatunnel/pull/10306#issuecomment-4972188155
Thank you for continuing to update the PR and address the review feedback. I would like to discuss the following idea: Based on the issues identified in the current reviews, it seems increasingly difficult for this feature to fully converge through incremental fixes alone. The current ErrorData path needs to handle checkpointing, recovery, timer flush, state/committer lifecycles, MultiTable, metrics, classloader management, and other runtime concerns separately. This makes it difficult to align with the existing execution pipeline and may significantly increase long-term maintenance costs. Therefore, I am concerned that this may no longer be only a matter of completing or refining the feature. The current abstraction boundary and implementation direction may be worth reconsidering. Based on the existing findings, I would like to suggest the following directions for discussion: 1. Build a first-class, engine-visible ErrorData shadow pipeline through `PhysicalPlan`. The Error Sink could then participate in scheduling as a normal `SinkAction` and reuse the existing `checkpoint`, `recovery`, `timer-flush`, `state-management`, `metrics`, and `classloader` lifecycles. 2. Define a unified and stable Connector capability contract. Each Connector could explicitly opt in by implementing row-level error classification, error-event emission, and terminal-outcome reporting. Connectors that do not implement this contract should retain the existing fail-fast behavior, rather than having the engine infer their error semantics through a generic wrapper. This approach could avoid reimplementing a separate Sink lifecycle inside the ErrorData path, while making the capability boundaries, delivery guarantees, and supported Connector scope clearer. @zhangshenghang @davidzollo @dybyte @hawk9821 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
