Hello Airflow Dev List, I'm considering proposing a refactor to import errors in order to support sending alert emails when the scheduler finds an import error (but not every time the scheduler finds the same import error). This is currently not possible because the import errors are cleared during each scheduler loop.
I'd like to poll the community for perspectives on other short commings of the import error model before proposing a refactor or other use cases folks might have for such a refactor (e.g. supporting an arbitrary callback function similar to SLA miss). My current thought is to just add an import_errors_history table to the database that is not cleared on each scheduler loop and does keep track of if an email was sent in a boolean field. The primary key could be constructed from a file hash and exception classname. Does this one use case warrant a new table? Should we just replace the import_errors table in place? If I can get a sense of high-level direction I can put together an AIP / PR. Cheers, Jake -- *Jacob Ferriero* Strategic Cloud Engineer: Data Engineering [email protected] 617-714-2509
