Will-Lo commented on PR #3667: URL: https://github.com/apache/gobblin/pull/3667#issuecomment-1500676439
Responding to the top level comment: `would we wish to pre-aggregate that within the event itself`, I think aggregation is needed because there can be thousands of tasks for large pipelines, which makes serializing all the states into an event lead to large Kafka events which we want to avoid. Also, I think most clients wouldn't care too much about the inner details of every individual task/mapper, it's mainly the concern of the Gobblin framework to deal with correctly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
