Hi George, It does make sense to keep SQLite on the other hand a Docker image with all the components might just be as convenient?
Dropping stuff would lessen the burden of maintenance. There is a lot of cruft that is not used. This reduces the surface for bugs and makes it easier to do the optimizations you refer too. However, I’m not married to a 2.0 release now, but it does look fresh ;-). Bolke Verstuurd vanaf mijn iPad > Op 18 dec. 2017 om 19:48 heeft George Leslie-Waksman > <geo...@cloverhealth.com.INVALID> het volgende geschreven: > > I really do not think we should drop sqlite support. We use sqlite for > local testing/development; I used it when doing my initial evaluation of > Airflow; and there are regular comments in this group about people using > sqlite locally for their workflows. It feels like a critical feature for a > lot of use cases. > > I also feel like we're kind of jumping the gun on pushing for 2.0 (is it > just because 9 is the last single digit?). There is a lot of the code base > that's pretty hairy and a number of things that are fairly non-performant / > buggy. I don't think that we NEED to break backward compatibility to clean > things up and fix things. I don't particularly like pickle or the old > import style but I don't feel like supporting them is terribly onerous (I > could be wrong). > > New webserver, releasing the API, and even timezones are all things that > can be released in a non-breaking fashion. > > --George > >> On Thu, Dec 14, 2017 at 2:20 PM Alek Storm <alek.st...@gmail.com> wrote: >> >> We use sqlite for developing Airflow DAGs locally, and doing basic checks >> for syntax/import errors in our CI/CD pipeline. >> >> Alek >> >> On Thu, Dec 14, 2017 at 4:19 PM, Chris Riccomini <criccom...@apache.org> >> wrote: >> >>> Agree with Bolke that it's a good idea, but major work. I will push back >> on >>> this for 2.0 mostly due to time concerns. I don't want 2.0 to take months >>> and months to get out. >>> >>>> Btw: what about dropping sqlite support? >>> >>> I'm fine with this, but isn't this really useful for demos? Seems to me >>> that the out of the box experience from airflow is one of the things that >>> make sit so compelling. >>> >>> On Thu, Dec 14, 2017 at 2:12 PM, Bolke de Bruin <bdbr...@gmail.com> >> wrote: >>> >>>> That will take quite some work. It is a a good idea but also a major >>>> change. Not sure if we should target that. >>>> >>>> Btw: what about dropping sqlite support? >>>> >>>> Verstuurd vanaf mijn iPad >>>> >>>>> Op 14 dec. 2017 om 21:19 heeft Gael Magnan <gaelmag...@gmail.com> >> het >>>> volgende geschreven: >>>>> >>>>> Hi, >>>>> >>>>> haven't been following much lately but on the import side of things, >>>> isn't >>>>> Airflow 2 the best moment to change to a pip plugin system for >> imports >>> of >>>>> third party stuff? >>>>> I.E being able to add a new type of credentials, operator etc.. >> without >>>>> touching to the airflow code itself or having them in a special >> folder. >>>>> >>>>> Regards >>>>> Gael >>>>> >>>>> >>>>> >>>>> Le jeu. 14 déc. 2017 à 14:17, Chris Riccomini <criccom...@apache.org >>> >>> a >>>>> écrit : >>>>> >>>>>> I'm fine with sensor refactor. Added to Wiki. >>>>>> >>>>>> On Thu, Dec 14, 2017 at 11:16 AM, Chris Riccomini < >>>> criccom...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> @Bolke, >>>>>>> >>>>>>>> Should we, before 2.0, start the graduation from the incubator? >>>>>>> >>>>>>> No, I'd rather keep them separate. We can certainly start >> graduation, >>>> but >>>>>>> I don't want to block 2.0. Can pursue them in parallel. >>>>>>> >>>>>>> On Thu, Dec 14, 2017 at 11:15 AM, Andy Hadjigeorgiou < >>>>>> andyxha...@gmail.com >>>>>>>> wrote: >>>>>>> >>>>>>>> Does it make sense to include sensors.py refactor in 2.0, so we >> can >>>>>>>> retire the old import structure easily and support the new sensors >>>>>> package >>>>>>>> import structure? >>>>>>>> >>>>>>>> - Andy >>>>>>>> >>>>>>>> On Thu, Dec 14, 2017 at 2:12 PM, Driesprong, Fokko >>>> <fo...@driesprong.frl >>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> Good initiative. I would be happy to refactor the sensors >> package. >>> I >>>>>>>>> started on it but it changes a lot, all the imports will break. >>>>>>>>> >>>>>>>>> https://github.com/apache/incubator-airflow/pull/2875 >>>>>>>>> >>>>>>>>> What do you guys think? >>>>>>>>> >>>>>>>>> Cheers, Fokko >>>>>>>>> >>>>>>>>> 2017-12-14 20:09 GMT+01:00 Chris Riccomini < >> criccom...@apache.org >>>> : >>>>>>>>> >>>>>>>>>> I have created a wiki here: >>>>>>>>>> >>>>>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0 >>>>>>>>>> >>>>>>>>>> To track features and progress. >>>>>>>>>> >>>>>>>>>> On Thu, Dec 14, 2017 at 11:08 AM, Chris Riccomini < >>>>>>>>> criccom...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>>> Re: #2: Is there a current ticket out for removing the legacy >>>>>>>>> import >>>>>>>>>>> style? >>>>>>>>>>> >>>>>>>>>>> No, I don't think so, but you can create one! :) >>>>>>>>>>> >>>>>>>>>>> On Thu, Dec 14, 2017 at 11:06 AM, Andy Hadjigeorgiou < >>>>>>>>>> andyxha...@gmail.com >>>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> This sounds great, something I'd like to see updated for 2.0 >>>>>>>>> release (or >>>>>>>>>>>> before) is the Airflow documentation >>>>>>>>>>>> <http://airflow.readthedocs.io/en/latest/installation.html> ( >>>>>>>>>>>> http://airflow.readthedocs.io/en/latest/installation.html). >> It >>>>>>>>> seems >>>>>>>>>> that >>>>>>>>>>>> updating the repo does not update this site - and given that >> we >>>>>>>>> will be >>>>>>>>>>>> removing certain deprecated features I imagine the docs will >>>>>> change >>>>>>>>>>>> substantially. >>>>>>>>>>>> >>>>>>>>>>>> Re: #2: Is there a current ticket out for removing the legacy >>>>>> import >>>>>>>>>>>> style? >>>>>>>>>>>> I'm happy to help drive that forward. >>>>>>>>>>>> >>>>>>>>>>>> - Andy >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Dec 14, 2017 at 1:55 PM, Chris Riccomini < >>>>>>>>> criccom...@apache.org >>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hey all, >>>>>>>>>>>>> >>>>>>>>>>>>> With 1.9.0 wrapping up soon (hopefully), there's been some >>>>>>>>> discussion >>>>>>>>>> on >>>>>>>>>>>>> the having the next release be Airflow 2.0 (rather than >> 1.10). >>>>>>>>> This >>>>>>>>>>>> would >>>>>>>>>>>>> allow us to break compatibility, and clean up some stuff. >>>>>> Proposed >>>>>>>>>>>> things >>>>>>>>>>>>> to include in 2.0 are: >>>>>>>>>>>>> >>>>>>>>>>>>> 1) New webserver that Joy Gao has been working on. >>>>>>>>>>>>> 2) Remove the legacy import style that's been deprecated >> since >>>>>> at >>>>>>>>>> least >>>>>>>>>>>> 1.8 >>>>>>>>>>>>> 3) New timzone feature >>>>>>>>>>>>> 4) Move API out of experimental >>>>>>>>>>>>> >>>>>>>>>>>>> I want to keep the list fairly tight, preferably to things >> that >>>>>>>>> have >>>>>>>>>>>>> already been done, so that we can ship it fairly quickly (in >>> the >>>>>>>>> next >>>>>>>>>>>>> couple of months). >>>>>>>>>>>>> >>>>>>>>>>>>> Does this sound like a good list? >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> Chris >>>>>> >>>> >>> >>
