Hi team, My name is Ace, I’m a data engineer at Pinterest, and I wanted to get some feedback from the community on how they’d propose designing a feature that has been asked for us multiple times, that we’d like to implement in-house.
The ask is the following: - Users want to be able to add an optional reason/description when they click on the Mark Success/Failed for both task level in graph views or dag level in tree views. This is to clearly state why the actions were taken on higher priority flows for when others are stepping in to look in it. For us the feature makes sense to have, and I was wondering if this is something the greater group would like to have upstream as well. And if so then we wanted to know what was preferred. 1. We can add this information in the UI for the confirmation view and an optional textbook to add details similar to how users can add conf variables when triggering a dag option is selected in the UI. Then after this, storing this information is the question. Do we want to store this information in the action logging extra blob and then add a view in the dag view where it has actions just pertaining to this dag and its tasks. (We plan to add this personally anyways for ourselves as going to the big list of actions for all dags and searching is not convenient always). Or do we want to add a new column in the task Instance model and dag run model to store the descriptions and retrieve the information from there. Tradeoffs here include the addition of a schema change, where the new column would generally be sparse, and the data is more resultant from an action vs necessarily being stored as part of the task run/dag run. Please let me know on any feedback, concerns, ideas, and more! Cheers, Ace
