o-nikolas commented on a change in pull request #19022:
URL: https://github.com/apache/airflow/pull/19022#discussion_r784233810
##########
File path: airflow/config_templates/airflow_local_settings.py
##########
@@ -194,13 +193,17 @@
DEFAULT_LOGGING_CONFIG['handlers'].update(S3_REMOTE_HANDLERS)
elif REMOTE_BASE_LOG_FOLDER.startswith('cloudwatch://'):
+ create_log_group = conf.getboolean('logging',
'CREATE_CLOUDWATCH_LOG_GROUP', fallback=True)
+ create_log_stream = conf.getboolean('logging',
'CREATE_CLOUDWATCH_LOG_STREAM', fallback=True)
Review comment:
@nik-davis
Hey Nik, This is a good callout and it's something we ran into while
developing MWAA (which leverages Cloudawatch for all log types and thus creates
a **LOT** of log groups and log stream requests). The correct solution here is
to actually not call create log group until you're sure it doesn't exist. Which
involves first calling describe_log_groups (which has a much higher quota) and
if the log group in question doesn't exist, then create it. This is what we do
in MWAA and this is what the Watchtower package should be doing as well. And in
fact, it is what they [do
here](https://github.com/kislyuk/watchtower/blob/a2a723787c3a0c84a1c708fd0b850d08d2a18c9d/watchtower/__init__.py#L263)
since v2.0.0. And I recently upgraded Airflow to [use v2.0.1
here](https://github.com/apache/airflow/pull/19907). So this problem should
already be solved!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
