The Log table can be used at Dag Level too if we pass task&instance=None
For example: We log all the CLI actions in the Log table where we pass task_instance as None. Code: https://github.com/apache/airflow/blob/4e32546faf227a6497ce8b282fff7450cae6f665/airflow/utils/cli.py#L133 - Kaxil On Sun, Oct 18, 2020, 13:49 Deng Xiaodong <[email protected]> wrote: > I have the same concern as Yu Qian does, and a bit more elaborations > follow below: > > - "log" table > <https://github.com/apache/airflow/blob/master/airflow/models/log.py> has > fields: *id*, *dttm*, *dag_id*, *task_id*, *event*, *execution_date*, > *owner*, *extra*. Seems to me only "*event*" or "*extra*" can provide > some room for this use case. But does it support more complicated cases > well? For example, for the same task instance, there may be back and forth > tries, and I may mark it Success/Failure for multiple times back and forth, > with multiple comments. > > - As demonstrated by the field list of the "log" table, it's at "task" > level (it has field "taks_id"). Ace has mentioned that "*Users want to be > able to add an optional reason/description when they click on the Mark > Success/Failed for both task level in graph views or dag level in tree > views*". So my question would be how the "dag level" should be supported > if we use the "log" table to implement this feature? > > > XD > > On Sun, Oct 18, 2020 at 11:13 AM Yu Qian <[email protected]> wrote: > >> Did anyone consider having a more generic comments feature? The feature >> Ace described is an example of a comment being useful. E.g. when a user >> clears a task or marks it success/failed, he probably wants to include a >> comment for his action when he clicks OK. If a task failed and the user >> decided to do nothing about it, he may also wants to leave a comment with >> the reason. >> >> It'll be nice if these comments can optionally be displayed on the UI >> somewhere. To do that, it's likely cleaner to create a dedicated comments >> table where each comment has a foreign key to associate it with the >> TaskInstance. >> >> On Sat, Oct 17, 2020 at 1:30 AM Vikram Koka <[email protected]> wrote: >> >>> +1 for feature. Strong preference for storing it in the log table, >>> rather than in the task instance. >>> >>> >>> >>> On Fri, Oct 16, 2020 at 10:22 AM Tomasz Urbaszek <[email protected]> >>> wrote: >>> >>>> +1, I know a customer who was requesting it too >>>> >>>> On Fri, Oct 16, 2020 at 6:52 PM Kaxil Naik <[email protected]> wrote: >>>> >>>>> +1 >>>>> >>>>> On Fri, Oct 16, 2020 at 5:48 PM Jarek Potiuk <[email protected]> >>>>> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> +1 for the feature. Looking at the schema of the log table, I think >>>>>>> it's perfect to store such kind of information. >>>>>>> >>>>>>> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> This could be pretty valuable for future audits. I’d personally >>>>>>>> rather avoid adding fields to the DB in general. Could we store it >>>>>>>> wherever >>>>>>>> we store normal failure messages? >>>>>>>> >>>>>>>> via Newton Mail >>>>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2> >>>>>>>> >>>>>>>> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey >>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>> Hi team, >>>>>>>> >>>>>>>> My name is Ace, I’m a data engineer at Pinterest, and I wanted to >>>>>>>> get some feedback from the community on how they’d propose designing a >>>>>>>> feature that has been asked for us multiple times, that we’d like to >>>>>>>> implement in-house. >>>>>>>> >>>>>>>> The ask is the following: >>>>>>>> - Users want to be able to add an optional reason/description when >>>>>>>> they click on the Mark Success/Failed for both task level in graph >>>>>>>> views or >>>>>>>> dag level in tree views. This is to clearly state why the actions were >>>>>>>> taken on higher priority flows for when others are stepping in to look >>>>>>>> in >>>>>>>> it. >>>>>>>> >>>>>>>> For us the feature makes sense to have, and I was wondering if this >>>>>>>> is something the greater group would like to have upstream as well. >>>>>>>> And if >>>>>>>> so then we wanted to know what was preferred. >>>>>>>> >>>>>>>> 1. We can add this information in the UI for the confirmation view >>>>>>>> and an optional textbook to add details similar to how users can add >>>>>>>> conf >>>>>>>> variables when triggering a dag option is selected in the UI. >>>>>>>> >>>>>>>> Then after this, storing this information is the question. Do we >>>>>>>> want to store this information in the action logging extra blob and >>>>>>>> then >>>>>>>> add a view in the dag view where it has actions just pertaining to >>>>>>>> this dag >>>>>>>> and its tasks. (We plan to add this personally anyways for ourselves as >>>>>>>> going to the big list of actions for all dags and searching is not >>>>>>>> convenient always). >>>>>>>> >>>>>>>> Or do we want to add a new column in the task Instance model and >>>>>>>> dag run model to store the descriptions and retrieve the information >>>>>>>> from >>>>>>>> there. >>>>>>>> >>>>>>>> Tradeoffs here include the addition of a schema change, where the >>>>>>>> new column would generally be sparse, and the data is more resultant >>>>>>>> from >>>>>>>> an action vs necessarily being stored as part of the task run/dag run. >>>>>>>> >>>>>>>> Please let me know on any feedback, concerns, ideas, and more! >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Ace >>>>>>>> >>>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Jarek Potiuk >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>> >>>>>> M: +48 660 796 129 <+48660796129> >>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>> >>>>>>
