+1, Lyft uses the extra blob in the log table? to store UI auditing information.
On Fri, Oct 16, 2020 at 9:42 AM Sumit Maheshwari <[email protected]> wrote: > +1 for the feature. Looking at the schema of the log table, I think it's > perfect to store such kind of information. > > On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman <[email protected]> > wrote: > >> This could be pretty valuable for future audits. I’d personally rather >> avoid adding fields to the DB in general. Could we store it wherever we >> store normal failure messages? >> >> via Newton Mail >> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2> >> >> On Fri, Oct 16, 2020 at 9:21 AM, Ace Haidrey >> <[email protected]> wrote: >> >> Hi team, >> >> My name is Ace, I’m a data engineer at Pinterest, and I wanted to get >> some feedback from the community on how they’d propose designing a feature >> that has been asked for us multiple times, that we’d like to implement >> in-house. >> >> The ask is the following: >> - Users want to be able to add an optional reason/description when they >> click on the Mark Success/Failed for both task level in graph views or dag >> level in tree views. This is to clearly state why the actions were taken on >> higher priority flows for when others are stepping in to look in it. >> >> For us the feature makes sense to have, and I was wondering if this is >> something the greater group would like to have upstream as well. And if so >> then we wanted to know what was preferred. >> >> 1. We can add this information in the UI for the confirmation view and an >> optional textbook to add details similar to how users can add conf >> variables when triggering a dag option is selected in the UI. >> >> Then after this, storing this information is the question. Do we want to >> store this information in the action logging extra blob and then add a view >> in the dag view where it has actions just pertaining to this dag and its >> tasks. (We plan to add this personally anyways for ourselves as going to >> the big list of actions for all dags and searching is not convenient >> always). >> >> Or do we want to add a new column in the task Instance model and dag run >> model to store the descriptions and retrieve the information from there. >> >> Tradeoffs here include the addition of a schema change, where the new >> column would generally be sparse, and the data is more resultant from an >> action vs necessarily being stored as part of the task run/dag run. >> >> Please let me know on any feedback, concerns, ideas, and more! >> >> Cheers, >> Ace >> >>
