William Lo created GOBBLIN-2001:
-----------------------------------

             Summary: Add config to disable emission of failed task events
                 Key: GOBBLIN-2001
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-2001
             Project: Apache Gobblin
          Issue Type: Improvement
            Reporter: William Lo


Currently in the Gobblin framework all failed tasks will emit events when they 
fail and include some stacktrace to provide context on the failure. This is 
problematic when there are Gobblin jobs with hundreds of thousands of tasks 
(e.g. iceberg distcp) and causes Kafka partitions to lag under the larger 
events. 

We want to enable a config to disable this, and potentially extend this 
solution to disable all task level events as they can get quite noisy and logs 
generally provide better context.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to