We use sqlite for developing Airflow DAGs locally, and doing basic checks for syntax/import errors in our CI/CD pipeline.
Alek On Thu, Dec 14, 2017 at 4:19 PM, Chris Riccomini <[email protected]> wrote: > Agree with Bolke that it's a good idea, but major work. I will push back on > this for 2.0 mostly due to time concerns. I don't want 2.0 to take months > and months to get out. > > > Btw: what about dropping sqlite support? > > I'm fine with this, but isn't this really useful for demos? Seems to me > that the out of the box experience from airflow is one of the things that > make sit so compelling. > > On Thu, Dec 14, 2017 at 2:12 PM, Bolke de Bruin <[email protected]> wrote: > > > That will take quite some work. It is a a good idea but also a major > > change. Not sure if we should target that. > > > > Btw: what about dropping sqlite support? > > > > Verstuurd vanaf mijn iPad > > > > > Op 14 dec. 2017 om 21:19 heeft Gael Magnan <[email protected]> het > > volgende geschreven: > > > > > > Hi, > > > > > > haven't been following much lately but on the import side of things, > > isn't > > > Airflow 2 the best moment to change to a pip plugin system for imports > of > > > third party stuff? > > > I.E being able to add a new type of credentials, operator etc.. without > > > touching to the airflow code itself or having them in a special folder. > > > > > > Regards > > > Gael > > > > > > > > > > > > Le jeu. 14 déc. 2017 à 14:17, Chris Riccomini <[email protected]> > a > > > écrit : > > > > > >> I'm fine with sensor refactor. Added to Wiki. > > >> > > >> On Thu, Dec 14, 2017 at 11:16 AM, Chris Riccomini < > > [email protected]> > > >> wrote: > > >> > > >>> @Bolke, > > >>> > > >>>> Should we, before 2.0, start the graduation from the incubator? > > >>> > > >>> No, I'd rather keep them separate. We can certainly start graduation, > > but > > >>> I don't want to block 2.0. Can pursue them in parallel. > > >>> > > >>> On Thu, Dec 14, 2017 at 11:15 AM, Andy Hadjigeorgiou < > > >> [email protected] > > >>>> wrote: > > >>> > > >>>> Does it make sense to include sensors.py refactor in 2.0, so we can > > >>>> retire the old import structure easily and support the new sensors > > >> package > > >>>> import structure? > > >>>> > > >>>> - Andy > > >>>> > > >>>> On Thu, Dec 14, 2017 at 2:12 PM, Driesprong, Fokko > > <[email protected] > > >>> > > >>>> wrote: > > >>>> > > >>>>> Hi all, > > >>>>> > > >>>>> Good initiative. I would be happy to refactor the sensors package. > I > > >>>>> started on it but it changes a lot, all the imports will break. > > >>>>> > > >>>>> https://github.com/apache/incubator-airflow/pull/2875 > > >>>>> > > >>>>> What do you guys think? > > >>>>> > > >>>>> Cheers, Fokko > > >>>>> > > >>>>> 2017-12-14 20:09 GMT+01:00 Chris Riccomini <[email protected] > >: > > >>>>> > > >>>>>> I have created a wiki here: > > >>>>>> > > >>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0 > > >>>>>> > > >>>>>> To track features and progress. > > >>>>>> > > >>>>>> On Thu, Dec 14, 2017 at 11:08 AM, Chris Riccomini < > > >>>>> [email protected]> > > >>>>>> wrote: > > >>>>>> > > >>>>>>>> Re: #2: Is there a current ticket out for removing the legacy > > >>>>> import > > >>>>>>> style? > > >>>>>>> > > >>>>>>> No, I don't think so, but you can create one! :) > > >>>>>>> > > >>>>>>> On Thu, Dec 14, 2017 at 11:06 AM, Andy Hadjigeorgiou < > > >>>>>> [email protected] > > >>>>>>>> wrote: > > >>>>>>> > > >>>>>>>> This sounds great, something I'd like to see updated for 2.0 > > >>>>> release (or > > >>>>>>>> before) is the Airflow documentation > > >>>>>>>> <http://airflow.readthedocs.io/en/latest/installation.html> ( > > >>>>>>>> http://airflow.readthedocs.io/en/latest/installation.html). It > > >>>>> seems > > >>>>>> that > > >>>>>>>> updating the repo does not update this site - and given that we > > >>>>> will be > > >>>>>>>> removing certain deprecated features I imagine the docs will > > >> change > > >>>>>>>> substantially. > > >>>>>>>> > > >>>>>>>> Re: #2: Is there a current ticket out for removing the legacy > > >> import > > >>>>>>>> style? > > >>>>>>>> I'm happy to help drive that forward. > > >>>>>>>> > > >>>>>>>> - Andy > > >>>>>>>> > > >>>>>>>> On Thu, Dec 14, 2017 at 1:55 PM, Chris Riccomini < > > >>>>> [email protected] > > >>>>>>> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Hey all, > > >>>>>>>>> > > >>>>>>>>> With 1.9.0 wrapping up soon (hopefully), there's been some > > >>>>> discussion > > >>>>>> on > > >>>>>>>>> the having the next release be Airflow 2.0 (rather than 1.10). > > >>>>> This > > >>>>>>>> would > > >>>>>>>>> allow us to break compatibility, and clean up some stuff. > > >> Proposed > > >>>>>>>> things > > >>>>>>>>> to include in 2.0 are: > > >>>>>>>>> > > >>>>>>>>> 1) New webserver that Joy Gao has been working on. > > >>>>>>>>> 2) Remove the legacy import style that's been deprecated since > > >> at > > >>>>>> least > > >>>>>>>> 1.8 > > >>>>>>>>> 3) New timzone feature > > >>>>>>>>> 4) Move API out of experimental > > >>>>>>>>> > > >>>>>>>>> I want to keep the list fairly tight, preferably to things that > > >>>>> have > > >>>>>>>>> already been done, so that we can ship it fairly quickly (in > the > > >>>>> next > > >>>>>>>>> couple of months). > > >>>>>>>>> > > >>>>>>>>> Does this sound like a good list? > > >>>>>>>>> > > >>>>>>>>> Cheers, > > >>>>>>>>> Chris > > >> > > >
