Hi everyone, Vincent and I were chatting about the gaps in Triggerer observability, and I'd love to hear your thoughts about how we're planning to address them.
Some background: the Triggerer was introduced to handle deferrable tasks. Back then that was all it ran, so a trigger's logs and failures went into the task's logs and state, where they already had a home. Since then it has grown to run asset watchers and callbacks, which have no task to report through. Callbacks have partly caught up, but watchers haven't, which leaves three gaps: 1. Failures are silent. A watcher that crashes is restarted every ~2s forever, and one whose code can't be imported quietly stops while still counting as running. Nothing is recorded anywhere a user can see. 2. You can't read a watcher's logs. Its lines go into the triggerer's output without saying which watcher wrote them, with no UI or API to read them. 3. No view of your watchers. There's no way to see which are running or failing, when each last fired, or what it fired. The plan is to make watchers first class: record each one's status, last-fired time and last error, give each its own log, expose it all through the REST API, and put a Watchers page in the UI. One thing I'd also like to add on top is being able to pause a single watcher without pausing its Dag. Does anyone see other gaps or features I should fold in? And any preference on making this an AIP? I was thinking of following up with a design outline and going ahead without one. Best, Sean
