dinigo commented on issue #28405: URL: https://github.com/apache/airflow/issues/28405#issuecomment-1359549462
When I opened the issue there was no such thing as "Google Cloud Composer", It has now integrated logs and alarms. There are many workaround still. The issue remains cause I see people implementing the "alarming/monitoring/SRE" as part of the "Data Engineering/Orchestration". Those are different concerns and I think there should be some "alarming by default" I suggestion was to have a common notification interface, similar to the email one (`airflow/utils/notification.py`): ```python # inspired by https://github.com/apache/airflow/blob/main/airflow/utils/email.py # see https://airflow.apache.org/docs/apache-airflow/stable/howto/email-config.html class NotificationInterface(ABC): @abstractmethod def notify( self, conn_id: str = conf.get("notification", "CONN_ID", fallback=None), message: str = conf.get("notification", "MESSAGE_TEMPLATE", fallback=None), to: List[str] = conf.get("notification", "TO", fallback=None), from_user: str = conf.get("notification", "FROM_USER", fallback=None), ): """ 1. apply jinja to message 2. retrieve the connection 3. use the connection hook/method """ Then implement this interface, for example for Slack (`airflow/providers/slack/notification.py`) ```python class SlackNotifier(NotificationInterface): def notify( self, conn_id: str = conf.get("notification", "CONN_ID", fallback=SlackHook.default_conn_name), message: str = conf.get("notification", "MESSAGE_TEMPLATE", fallback=None), to: List[str] = conf.get("notification", "TO", fallback=None), from_user: str = conf.get("notification", "FROM_USER", fallback='airflow') ): if message.endswith('.txt'): message = open(message).read() environment = jinja2.Environment() template = environment.from_string(message) rendered_message = template.render(from_user=from_user, to=to) slack_client = SlackHook(conn_id).client for channel in to: slack_client.chat_postMessage( channel=channel, text=rendered_message, username=from_user, ) ``` So that you can easily configure it from the `airflow.cfg` (or env variables): ```toml [notification] class="airflow.providers.slack.notification" conn_id="slack_api_conn" to=['#airflow-notifications'] message = "foo/bar/slack-message.txt" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
