Re: programmatically creating and airflow quirks

2018-11-28 Thread soma dhavala
Great inputs James. I was premature in saying we need micro-services. Any solutioning should depend on the problem(s) being solved and promise(s) being made. thanks, -soma > On Nov 28, 2018, at 11:24 PM, James Meickle > wrote: > > I would be very interested in helping draft a rearchitecting

Re: programmatically creating and airflow quirks

2018-11-28 Thread James Meickle
I would be very interested in helping draft a rearchitecting AIP. Of course, that's a vague statement. I am interested in several specific areas of Airflow functionality that would be hard to modify without some refactoring taking place first: 1) Improving Airflow's data model so it's easier to ha

Re: programmatically creating and airflow quirks

2018-11-28 Thread Ash Berlin-Taylor
I have similar feelings around the "core" of Airflow and would _love_ to somehow find time to spend a month really getting to grips with the scheduler and the dagbag and see what comes to light with fresh eyes and the benefits of hindsight. Finding that time is going to be A Challenge thoug

Re: programmatically creating and airflow quirks

2018-11-25 Thread soma dhavala
> On Nov 26, 2018, at 7:50 AM, Maxime Beauchemin > wrote: > > The historical reason is that people would check in scripts in the repo > that had actual compute or other forms or undesired effect in module scope > (scripts with no "if __name__ == '__main__':") and Airflow would just run > this

Re: programmatically creating and airflow quirks

2018-11-25 Thread Maxime Beauchemin
The historical reason is that people would check in scripts in the repo that had actual compute or other forms or undesired effect in module scope (scripts with no "if __name__ == '__main__':") and Airflow would just run this script while seeking for DAGs. So we added this mitigation patch that wou

Re: programmatically creating and airflow quirks

2018-11-24 Thread Beau Barker
In my opinion this searching for dags is not ideal. We should be explicitly specifying the dags to load somewhere. > On 25 Nov 2018, at 10:41 am, Kevin Yang wrote: > > I believe that is mostly because we want to skip parsing/loading .py files > that doesn't contain DAG defs to save time, as sc

Re: programmatically creating and airflow quirks

2018-11-24 Thread Kevin Yang
I believe that is mostly because we want to skip parsing/loading .py files that doesn't contain DAG defs to save time, as scheduler is going to parse/load the .py files over and over again and some files can take quite long to load. Cheers, Kevin Y On Fri, Nov 23, 2018 at 12:44 AM soma dhavala w

Re: programmatically creating and airflow quirks

2018-11-23 Thread soma dhavala
happy to report that the “fix” worked. thanks Alex. btw, wondering why was it there in the first place? how does it help — saves time, early termination — what? > On Nov 23, 2018, at 8:18 AM, Alex Guziel wrote: > > Yup. > > On Thu, Nov 22, 2018 at 3:16 PM soma dhavala

Re: programmatically creating and airflow quirks

2018-11-22 Thread Alex Guziel
Yup. On Thu, Nov 22, 2018 at 3:16 PM soma dhavala wrote: > > > On Nov 23, 2018, at 3:28 AM, Alex Guziel wrote: > > It’s because of this > > “When searching for DAGs, Airflow will only consider files where the > string “airflow” and “DAG” both appear in the contents of the .py file.” > > > Have

Re: programmatically creating and airflow quirks

2018-11-22 Thread soma dhavala
> On Nov 23, 2018, at 3:28 AM, Alex Guziel wrote: > > It’s because of this > > “When searching for DAGs, Airflow will only consider files where the string > “airflow” and “DAG” both appear in the contents of the .py file.” > Have not noticed it. From airflow/models.py, in process_file — (

Re: programmatically creating and airflow quirks

2018-11-22 Thread Alex Guziel
It’s because of this “When searching for DAGs, Airflow will only consider files where the string “airflow” and “DAG” both appear in the contents of the .py file.” On Thu, Nov 22, 2018 at 2:27 AM soma dhavala wrote: > > > On Nov 22, 2018, at 3:37 PM, Alex Guziel wrote: > > I think this is what

Re: programmatically creating and airflow quirks

2018-11-22 Thread soma dhavala
> On Nov 22, 2018, at 3:37 PM, Alex Guziel wrote: > > I think this is what is going on. The dags are picked by local variables. > I.E. if you do > dag = Dag(...) > dag = Dag(…) from my_module import create_dag for file in yaml_files: dag = create_dag(file) globals()[dag.dag_i

Re: programmatically creating and airflow quirks

2018-11-22 Thread Alex Guziel
I think this is what is going on. The dags are picked by local variables. I.E. if you do dag = Dag(...) dag = Dag(...) Only the second dag will be picked up. On Thu, Nov 22, 2018 at 2:04 AM Soma S Dhavala wrote: > Hey AirFlow Devs: > In our organization, we build a Machine Learning WorkBench wi