julienledem commented on PR #29940:
URL: https://github.com/apache/airflow/pull/29940#issuecomment-1483267080
> It is quite opaque to me know how this works, why it needs workers at all,
what it does to my running system to have those workers, how does it affect
task runs, what if the code fails did my task fail? etc etc.
To make the documentation effort efficient, we need a list of the points
that need documenting.
Starting from your comment, we have:
- purpose of worker in this PR
- impact of workers
- how we isolate the running job from failures in the listener
The purpose of the worker is to isolate the user task from failures or
latency in the OL event sending. (I'll let @mobuchowski expand on that)
@bolkedebruin could you elaborate on the other points that need explanation
from your perspective?
Since this code is being moved, a lot of the history is documented in the
PRs that happened on the OL project:
https://github.com/OpenLineage/OpenLineage/pulls?q=is%3Apr+-label%3Adependabot+label%3Aintegration%2Fairflow+
Incremental changes were documented and discussed over time there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
