Re: new logging

Boris, I don't see where you configured the default logger . I think you'd have to at least throw in: logger = logging.getLogger('python_test_logging') and then use the logger.info(…) instead of logging.info(…) On Tue, Oct 17,

Re: programatically configure aws_default, emr_default connections?

Is there any way to programatically configure the aws_default, emr_default or other airflow connections? In a pre-1.8.x installation we used the following script: --- snip --- #!/usr/bin/python import json from airflow import models from airflow.utils import db as db_utils CONNECTIONS = [

Re: programatically configure aws_default, emr_default connections?

The connections are just db rows - is it possible to run a couple sql queries on the database connected to airflow when building the container to create them? (If you also want to avoid cli) > On Oct 16, 2017, at 5:58 PM, Frank Maritato wrote: > > Hi, > > Is there

Re: programatically configure aws_default, emr_default connections?

`airflow connections --add` might be what you're looking for. On Mon, Oct 16, 2017 at 2:58 PM, Frank Maritato wrote: > Hi, > > Is there any way to programatically configure the aws_default, emr_default > or other airflow connections? The reason I'm asking is that I'm

programatically configure aws_default, emr_default connections?

Hi, Is there any way to programatically configure the aws_default, emr_default or other airflow connections? The reason I'm asking is that I'm creating a docker container and I want it to be preconfigured without ui interaction. I looked at Puckel's docker-airflow project and I didn’t see

Re: Meetup Interest?

Glad to see there is interest! I'll work on setting this up. On Sun, Oct 15, 2017 at 10:47 AM Cade Markegard wrote: > +1 at SF meetup > > Would be interested in seeing that progress of airflow + k8s and any other > advancements the community has made. > On Sun, Oct 15,

Re: Return results optionally from spark_sql_hook

great, this is what I expected to hear but wanted to double check. thanks for all your help, Fokko On Mon, Oct 16, 2017 at 1:08 PM, Driesprong, Fokko wrote: > Hi Boris, > > When kicking off Spark jobs using Airflow, cluster mode is highly > recommended since the workload

Re: new logging

sorry i was not clear. In DAG definition file, I would normally import logging and when use with python operator like below (see print_context). In 1.8.2 i would see in Airflow log file (for that task) both 'hi from print' and 'hi from logging'. now I can only see 'hi from print'. I installed

Re: new logging

Hi Boris, Can you please elaborate? Where did the output used to end up? Did you provide any custom logging config, or are you using the default config? Maybe drop some example code. Cheers, Fokko 2017-10-16 19:02 GMT+02:00 Boris : > Hi guys, > > I used to do

Re: Return results optionally from spark_sql_hook

Hi Boris, When kicking off Spark jobs using Airflow, cluster mode is highly recommended since the workload of the driver is on the Hadoop cluster, and not on the Airflow machine itself. Personally I prefer the spark-submit operator since it will pull all the connection variables directly from

new logging

Hi guys, I used to do logging.info("somthing") from airflow python operators so i can see output in airflow logs. Worked fine in 1.8.2. Looks like this is no longer the case with master branch.I did look at updating.md but still not clear how to log custom messages from python operators. please

Re: Runbook to upgrade Airflow

would love to have a sort of step by step here. upgrading to 1.9 now as want to help as best i can and figure installing latest version one good way to help find any issues. unless its as simple as pip install. So far steps i'm taking (we just run local on single server as use BigQuery for most