sharingan-no-kakashi opened a new pull request #20397: URL: https://github.com/apache/airflow/pull/20397
Hello airflow community! I have seen that lately the topic "manual notes" has been mentioned by various users (#19011, #20122 etc), and they were all referred back to my original issue #16790 (and PRs #16953 and #17181). Unfortunately, the work in those PRs is far from being great and the lack of time (and feedback from the community) drove me away from that work. (Moreover the code was deleted from my fork and it was hard to reach) I am just sharing again my original idea here with this stripped down version of the previous implementations, hopefully this sparkles a discussion and/or a real implementation among the interested people (such as @euccas and @Acehaidrey) My very original idea in #19011 was to introduce a concept of "task note" for a task instance loosely based on the concept of Xcom (in retrospective it did not make much sense 😄 ). In #17181, I tried to generalize the idea by having a generic "even note" that works something like this: - I tried to select some events (user actions) which could be interesting to be noted (mostly any "confirm operation" and trigger) and there I added a field in the post request with the "note" name. - The model of an Event note is composed by `dag_id`, `task_id`, `execution_date` `owner` `timestamp` which means that an event should be related to a specific task_id of a specific dag of a specific execution_date (Which is kind of a limitation). (Actually in #17181 i also support trigger_run event note which are not related to a task id leaving the field empty) - Instead of changing of the relative APIs to handle the extra field, I created a decorated (loosely inspired by the action_logging operator) that if the request contains the `note` field it creates an entry in the even_notes table using as event name the name of the API (again inspired by the action_logging). This is also great,imho, because you can extend any API adding the support for note taking without changing it and if you want to improve/change you also need to modify it once. This being the simplest POC possible does have unanswered questions such as: - What does "event" mean? Which kind of activities/event should be notables? EG: Pausing a dag should be notable? - Should an event always be linked to a dag or/and task id (It is in this current implementation), if yes, how do you handle the "mark recursively success/failures" ? - How do users access notes? Here we are using the browser view but in #17181 there was an idea of having a dag view notes and a task instance note view as wel. - Probably many others 😄 My feeling is that there should probably be an higher level of abstraction for handling this feature properly and this is mostly a borderline wrong implementation, but without the vision of the core contributors it's complicated to implement a feature that spans different domains (frontend/backend/database/user expectations) I really hope this sparkles a discussion in the community on how to develop this feature in the best possible way 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
