Zihan Li created GOBBLIN-1830:
---------------------------------
Summary: Improving Container Transition Tracking in Streaming Data
Ingestion
Key: GOBBLIN-1830
URL: https://issues.apache.org/jira/browse/GOBBLIN-1830
Project: Apache Gobblin
Issue Type: Improvement
Reporter: Zihan Li
Currently, we rely on the flushing event to indicate when a container
transition occurs. However, if the ingestion process fails to successfully
flush the data, we won't know which container is responsible. This becomes
problematic when the pipeline restarts, as we won't be able to identify the
root cause without manually reviewing the logs of thousands of containers. To
avoid this issue, we need to develop a more reliable way of tracking container
transitions that doesn't rely solely on the flushing event.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
