Hello everyone!
We had an implicit consensus about removing the email/SMTP integration from
Airflow Core and replacing it with the SMTP provider and SmtpNotifier. This
change will simplify Airflow's core, improve maintainability, and provide a
more consistent experience with other providers. Now is the best moment to
do this before Airflow 3.0.
All the Airflow configurations in the *smtp* section are supported in the
SMTP connection, so removing the whole section should have minimal impact
on users. The migration effort is expected to be very low.
This thread focuses on how best to handle the configurations currently in
the email section, mainly:
-
*default_email_on_failure*
-
*default_email_on_retry*
-
*email_backend*
-
*email_conn_id*
The other configurations could be replaced by connection extras.
Options for handling email configuration:
-
Option 1: Complete Removal
-
Remove the entire email section from the Airflow configuration.
-
Remove the *email_on_failure* and *email_on_retry* arguments from
BaseOperator.
-
Implications:
-
Users will need to explicitly define notifiers (using
on_*_callbacks) for any task that requires email notifications.
-
There will be no built-in mechanism for default email
notifications.
-
This offers the most significant simplification of Airflow's core
but may increase the effort required to configure email alerts.
-
Option 2: Replacement Configurations
-
Introduce new configuration options:
-
*default_notification_on_failure*: (boolean) Enables default
notifications on task failure.
-
*default_notification_on_retry*: (boolean) Enables default
notifications on task retry.
-
*default_notifier*: (string) Specifies a method that initializes
and returns the default notifier.
-
Replace *email_on_failure* and *email_on_retry* in BaseOperator with
*notify_on_failure* and *notify_on_retry*, which would default to the
values set in the new configuration options.
-
Replace the old email handling behavior by simply adding the notifier
to the corresponding callback.
-
Implications:
-
Preserve the concept of default notifications while moving away
from the mail-specific configuration.
-
Support other notification types and send notifications by default.
-
Requires introducing new configuration options and modifying
BaseOperator.
-
Option 3: Dedicated Notification Connection
-
Create a new connection (of any type) with a specific and reserved
name for notifications (e.g., *notifications_connection*).
-
This connection would have extras to configure the notifier and
specify whether to send notifications by default.
-
Implications:
-
Provides a centralized and extensible way to manage all
notification settings without Airflow configuration.
-
Allows users to easily switch between different notification
methods and send notifications by default.
-
May require more initial setup for users but offers greater
flexibility in the long term.
-
Require more dev on our side.
Regarding migration, a fully automated migration might not be possible in
all cases because of the need to migrate Airflow configurations, we will
provide comprehensive documentation and examples to assist users in
transitioning their email configurations.
I'm interested in hearing your thoughts and any other suggestions you may
have.
P.S: I have created an almost ready-to-merge PR for option 1 (
https://github.com/apache/airflow/pull/46041)
