William Lo created GOBBLIN-2001:
-----------------------------------
Summary: Add config to disable emission of failed task events
Key: GOBBLIN-2001
URL: https://issues.apache.org/jira/browse/GOBBLIN-2001
Project: Apache Gobblin
Issue Type: Improvement
Reporter: William Lo
Currently in the Gobblin framework all failed tasks will emit events when they
fail and include some stacktrace to provide context on the failure. This is
problematic when there are Gobblin jobs with hundreds of thousands of tasks
(e.g. iceberg distcp) and causes Kafka partitions to lag under the larger
events.
We want to enable a config to disable this, and potentially extend this
solution to disable all task level events as they can get quite noisy and logs
generally provide better context.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)