Re: [DISCUSS] Tweaks to the Airflow logo

2019-08-15 Thread James Meickle
Hi all, This thread got split somehow. To bring discussion of Daniel's proposal to this thread: Daniel Gruno Wed, Aug 14, 2:33 AM (1 day ago) to dev Hi James, I don't mean to butt in much here, but as the general "guy who fixes various logos around here" (I re-did the current airflow logo as th

Re: Airflow Dynamic tasks

2019-08-15 Thread Maxime Beauchemin
What is your dynamic DAG doing? How long does it take to execute it just as a python script (`time python mydag.py`)? As an Airflow admin, people may want to lower the DAG parsing timeout configuration key to force people to not do crazy thing in DAG module scope. At some point at Airbnb we had so

Re: Airflow Dynamic tasks

2019-08-15 Thread Bacal, Eugene
Thank you for your reply, Max Dynamic DAGs query the database for tables and generates DAGs and tasks based on the output. For Python does not take much to execute: Dynamic - 500 tasks: time python PPAD_OIS_MASTER_IDI.py [2019-08-15 12:57:48,522] {settings.py:174} INFO - setting.configure_orm(

Re: Airflow Dynamic tasks

2019-08-15 Thread Daniel Standish
It's not just webserver and scheduler that will parse your dag file. During the execution of a dag run, dag file will be re-parsed at the start of every task instance. If you have 1000 tasks running in short period of time, that's 1000 queries. It's possible these queries are piling up in a queue

Re: [DISCUSS] Tweaks to the Airflow logo

2019-08-15 Thread Chris Hendrixson
Hey all, I'm the designer of the new logo that kicked off this thread. I will now and forever consider it the "shuriken" logo (I did not even know that word before this thread, so thank you). Based on the feedback I went back to the drawing board and came up with another version that is closer t

Re: [DISCUSS] Tweaks to the Airflow logo

2019-08-15 Thread Chao-Han Tsai
Thanks Chris for the new Airflow logo. Personally I like the new pinwheel design better. Its fins are more natural as they seem be made by folding four Isosceles right triangle papers, which is natural and symmetric. On the other hand, the fins of the shuriken one is too narrow and sharp, which se

Airflow meetup in India

2019-08-15 Thread ramandumcs
Hi All, Do we have airflow meetup(s) in India. We are based out of india and are using Apache Airflow as an orchestration engine to author and manage 1000(s) of multi-step workflows. Would be interested in Airflow meetup in India. Thanks, Raman Gupta