Hi Kevin, Kindly see my reply inline:
On Mon, Dec 18, 2017 at 3:28 PM, Kevin Lam <[email protected]> wrote: > Hi, > > I'm trying to get airflow to use GCS for logging purposes and had a few > questions. > > We're currently using Airflow 1.9rc2, running in a Kubernetes Airflow > deployment (similar to https://github.com/mumoshu/kube-airflow) > > 1/ Seems like the logging code has been going through some changes in the > recent versions. What's the correct way to set up GCS for logging? Is it by > just specifying remote_base_log_folder and remote_log_conn_id in > airflow.cfg? Or by following this guide: > http://airflow.readthedocs.io/en/latest/integration.html#gcp, using the > python based logging config? Is there an Airflow version that we should use > to be most stable? > The python based logging config is the right place to make changes, in our test setup, we override the airflow_local_settings.py similarly to the link you pasted. You may also want to config: [core]task_log_reader = gcs.task > > 2/ Is there a way to encode the connection for GCS in a file so that one > doesn't have to open the webserver and create it from the admin panel? It'd > be nice if the GCS connection would be automatically created. > Unfortunately GCS connection ties to some GCP project and is impossible to pre-populate. Airflow1.9 should fix the gcp connection type issue ( https://github.com/apache/incubator-airflow/commit/2f107d8a30910fd025774004d5c4c95407ed55c5), so you can use airflow connections CLI directly. > > Thanks in advance for your help! >
