kacpermuda commented on PR #41217: URL: https://github.com/apache/airflow/pull/41217#issuecomment-2268743259
> Also I am not sure we want to add it continuously - because this one will have basically - the same information in all events right? Yes, it would contain the same information across all events when it comes to the list of installed packages. In the future, we might want to extend this facet with more state-specific information, but for now, it's simple. > So maybe just emit it with some min frequency (i.e. max once per 10 minutes) ? I considered limiting it in some way (including it in only some of the events like START/COMPLETE, TASK/DAG), but my thinking was that this debug facet should only be used when something goes wrong. When that happens, we might only have a subset of events due to an error. If we limit the emittance in any way, we might end up not having it where we need it. However, I’m open to the idea—maybe there’s a better way of doing it that I haven’t thought of. > I think also it would be useful to get some way of temporarily enabling it (via API?). > > This could also be controllable via API / configuration I guess. I was hoping we could avoid introducing an additional OL configuration flag here. It’s easy to add them, but it can quickly lead to having so many that it becomes hard to manage. My thinking was: if I’m debugging an error, I’m already using debug logs, so I shouldn’t need to set separate variables to fully debug my issue. Rarely do I want the debug facet to appear without the debug logs enabled. Not sure what you mean by the API here, do you have any examples in mind? > So that we are not changing the behaviour of the system by observing it too much ? I’m not sure I understand this fully. Are you concerned that this debug facet might consume too many resources and thereby affect the system’s performance? If you could explain your concerns in more detail, I can address them more effectively. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
