I don't think Allison's PR fixes logging, but it's a step in the right
direction. The current approach creates an abstraction around reading logs,
whereas the final solution should define an interface for writings to logs
in addition to reading logs (which could indeed use something like
Hi Bolke,
I agree that we should make logging configurable lbut I wouldn't think
using handlers like python-elasticsearch-logger is a good idea over
flushing logs into files. Here are some reasons:
1. Such handlers do not have the built-in backpressure-sensitive
protocol that can prevent
In the light of fixing logging, I would definitely appreciate written design.
Especially, as there have been multiple attempts to fix some issues but these
have been more like stop gap fixes.
In my opinion Airflow should not stipulate in a hard coded fashion where and
how logging takes place.
Responding to some of Bolke's concerns in the github PR for this change:
> Mmm still not convinced. Especially on elastic search it is just easier
to use the start_date to shard on.
sharding on start_date isn't great because there is still some risk of
collisions and it means that we are coupling
Sent: 01 February 2017 12:55:42
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow Logging Updates
Hey Robin,
I also have an interest in logging, but to get Google Cloud logging to
work. I'm happy to work with you on this. But *first* the 1.8 needs to go
out.
But I don't think this wi
Hey Robin,
I also have an interest in logging, but to get Google Cloud logging to
work. I'm happy to work with you on this. But *first* the 1.8 needs to go
out.
But I don't think this will go on the patch branch, but maybe for the next
release. I don't know what the other people think.
On Wed,
> From: Maxime Beauchemin <maximebeauche...@gmail.com>
> Sent: 14 October 2016 17:02
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow Logging
>
> Another consideration is for configuration as code in `settings.py`
> in-place or in conjunc
gmail.com>
Sent: 14 October 2016 17:02
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow Logging
Another consideration is for configuration as code in `settings.py`
in-place or in conjunction with `airflow.cfg` to allow more dynamic
configuration, like passing your own custom log hand
Carrying on from the below. We believe it would be useful to have a number of
additional configuration options for logs as below. Do you think there are any
important options we have missed or do you have any other feedback? If so,
please let us know.
Global
CLEAR_INHERITED_LOGGING_SETTINGS
