I have similar feelings around the "core" of Airflow and would _love_ to somehow find time to spend a month really getting to grips with the scheduler and the dagbag and see what comes to light with fresh eyes and the benefits of hindsight.
Finding that time is going to be.... A Challenge though. (Oh, except no to microservices. Airflow is hard enough to operator right now without splitting things in to even more daemons) -ash > On 26 Nov 2018, at 03:06, soma dhavala <soma.dhav...@gmail.com> wrote: > > > >> On Nov 26, 2018, at 7:50 AM, Maxime Beauchemin <maximebeauche...@gmail.com> >> wrote: >> >> The historical reason is that people would check in scripts in the repo >> that had actual compute or other forms or undesired effect in module scope >> (scripts with no "if __name__ == '__main__':") and Airflow would just run >> this script while seeking for DAGs. So we added this mitigation patch that >> would confirm that there's something Airflow-related in the .py file. Not >> elegant, and confusing at times, but it also probably prevented some issues >> over the years. >> >> The solution here is to have a more explicit way of adding DAGs to the >> DagBag (instead of the folder-crawling approach). The DagFetcher proposal >> offers solutions around that, having a central "manifest" file that >> provides explicit pointers to all DAGs in the environment. > > Some rebasing needs to happen. When I looked at 1.8 code base almost an year > ago, it felt like more complex than necessary. What airflow is trying to > promise from an architectural standpoint — that was not clear to me. It is > trying to do too many things, scattered in too many places, is the feeling I > got. As a result, I stopped peeping, and just trust that it works — which it > does, btw. I tend to think that, airflow outgrew its original intents. A sort > of micro-services architecture has to be brought in. I may sound critical, > but no offense. I truly appreciate the contributions. > >> >> Max >> >> On Sat, Nov 24, 2018 at 5:04 PM Beau Barker <beauinmelbou...@gmail.com> >> wrote: >> >>> In my opinion this searching for dags is not ideal. >>> >>> We should be explicitly specifying the dags to load somewhere. >>> >>> >>>> On 25 Nov 2018, at 10:41 am, Kevin Yang <yrql...@gmail.com> wrote: >>>> >>>> I believe that is mostly because we want to skip parsing/loading .py >>> files >>>> that doesn't contain DAG defs to save time, as scheduler is going to >>>> parse/load the .py files over and over again and some files can take >>> quite >>>> long to load. >>>> >>>> Cheers, >>>> Kevin Y >>>> >>>> On Fri, Nov 23, 2018 at 12:44 AM soma dhavala <soma.dhav...@gmail.com> >>>> wrote: >>>> >>>>> happy to report that the “fix” worked. thanks Alex. >>>>> >>>>> btw, wondering why was it there in the first place? how does it help — >>>>> saves time, early termination — what? >>>>> >>>>> >>>>>> On Nov 23, 2018, at 8:18 AM, Alex Guziel <alex.guz...@airbnb.com> >>> wrote: >>>>>> >>>>>> Yup. >>>>>> >>>>>> On Thu, Nov 22, 2018 at 3:16 PM soma dhavala <soma.dhav...@gmail.com >>>>> <mailto:soma.dhav...@gmail.com>> wrote: >>>>>> >>>>>> >>>>>>> On Nov 23, 2018, at 3:28 AM, Alex Guziel <alex.guz...@airbnb.com >>>>> <mailto:alex.guz...@airbnb.com>> wrote: >>>>>>> >>>>>>> It’s because of this >>>>>>> >>>>>>> “When searching for DAGs, Airflow will only consider files where the >>>>> string “airflow” and “DAG” both appear in the contents of the .py file.” >>>>>>> >>>>>> >>>>>> Have not noticed it. From airflow/models.py, in process_file — (both >>> in >>>>> 1.9 and 1.10) >>>>>> .. >>>>>> if not all([s in content for s in (b'DAG', b'airflow')]): >>>>>> .. >>>>>> is looking for those strings and if they are not found, it is returning >>>>> without loading the DAGs. >>>>>> >>>>>> >>>>>> So having “airflow” and “DAG” dummy strings placed somewhere will make >>>>> it work? >>>>>> >>>>>> >>>>>>> On Thu, Nov 22, 2018 at 2:27 AM soma dhavala <soma.dhav...@gmail.com >>>>> <mailto:soma.dhav...@gmail.com>> wrote: >>>>>>> >>>>>>> >>>>>>>> On Nov 22, 2018, at 3:37 PM, Alex Guziel <alex.guz...@airbnb.com >>>>> <mailto:alex.guz...@airbnb.com>> wrote: >>>>>>>> >>>>>>>> I think this is what is going on. The dags are picked by local >>>>> variables. I.E. if you do >>>>>>>> dag = Dag(...) >>>>>>>> dag = Dag(…) >>>>>>> >>>>>>> from my_module import create_dag >>>>>>> >>>>>>> for file in yaml_files: >>>>>>> dag = create_dag(file) >>>>>>> globals()[dag.dag_id] = dag >>>>>>> >>>>>>> You notice that create_dag is in a different module. If it is in the >>>>> same scope (file), it will be fine. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Only the second dag will be picked up. >>>>>>>> >>>>>>>> On Thu, Nov 22, 2018 at 2:04 AM Soma S Dhavala < >>> soma.dhav...@gmail.com >>>>> <mailto:soma.dhav...@gmail.com>> wrote: >>>>>>>> Hey AirFlow Devs: >>>>>>>> In our organization, we build a Machine Learning WorkBench with >>>>> AirFlow as >>>>>>>> an orchestrator of the ML Work Flows, and have wrapped AirFlow python >>>>>>>> operators to customize the behaviour. These work flows are specified >>> in >>>>>>>> YAML. >>>>>>>> >>>>>>>> We drop a DAG loader (written python) in the default location airflow >>>>>>>> expects the DAG files. This DAG loader reads the specified YAML >>> files >>>>> and >>>>>>>> converts them into airflow DAG objects. Essentially, we are >>>>>>>> programmatically creating the DAG objects. In order to support >>> muliple >>>>>>>> parsers (yaml, json etc), we separated the DAG creation from loading. >>>>> But >>>>>>>> when a DAG is created (in a separate module) and made available to >>> the >>>>> DAG >>>>>>>> loaders, airflow does not pick it up. As an example, consider that I >>>>>>>> created a DAG picked it, and will simply unpickle the DAG and give it >>>>> to >>>>>>>> airflow. >>>>>>>> >>>>>>>> However, in current avatar of airfow, the very creation of DAG has to >>>>>>>> happen in the loader itself. As far I am concerned, airflow should >>> not >>>>> care >>>>>>>> where and how the DAG object is created, so long as it is a valid DAG >>>>>>>> object. The workaround for us is to mix parser and loader in the same >>>>> file >>>>>>>> and drop it in the airflow default dags folder. During dag_bag >>>>> creation, >>>>>>>> this file is loaded up with import_modules utility and shows up in >>> the >>>>> UI. >>>>>>>> While this is a solution, but it is not clean. >>>>>>>> >>>>>>>> What do DEVs think about a solution to this problem? Will saving the >>>>> DAG to >>>>>>>> the db and reading it from the db work? Or some core changes need to >>>>> happen >>>>>>>> in the dag_bag creation. Can dag_bag take a bunch of "created" DAGs. >>>>>>>> >>>>>>>> thanks, >>>>>>>> -soma >>>>>>> >>>>>> >>>>> >>>>> >>> >
