We discussed the splitting out of hooks & operators from core internally at Astronomer again today. I should be able to allocate some time to splitting out hooks & operators for the purpose of speeding up the CI. A faster CI on core would be hugely beneficial for all contributors.
I created a Jira issue for this at https://issues.apache.org/ jira/browse/AIRFLOW-2732. If anyone has pointers for this work, I'm all ears. I've done a lot with Python unit tests in the past but this part of the Airflow codebase is new to me. Taylor *Taylor Edmiston* Blog <https://blog.tedmiston.com/> | CV <https://stackoverflow.com/cv/taylor> | LinkedIn <https://www.linkedin.com/in/tedmiston/> | AngelList <https://angel.co/taylor> | Stack Overflow <https://stackoverflow.com/users/149428/taylor-edmiston> On Sun, Jul 1, 2018 at 6:02 AM, Bolke de Bruin <[email protected]> wrote: > Separating the tests is where the effort lies. So having that as a > consequence of splitting the packages would be nice. It has come up a > couple of times but it was not picked up unfortunately. > > B. > > > On 28 Jun 2018, at 08:32, Maxime Beauchemin <[email protected]> > wrote: > > > > It would be so nice to have a fast test suite. Having to wait for Travis > > for up to an hour makes many workflows (like working on a release) super > > painful. > > > > I spoke with folks at Astronomer recently about moving all operators and > > hooks to another Python package that airflow would import. This would > allow > > for independent test suites and to have a more regular release cadence on > > hooks and operators. What do you think? > > > > Max > > > > On Wed, Jun 27, 2018 at 11:18 PM Bolke de Bruin <[email protected]> > wrote: > > > >> Arghhh. The downside of doing this late at night and wanting to go to > >> bed... :-). Will make a new one > >> > >> Sent from my iPhone > >> > >>> On 28 Jun 2018, at 00:07, Chris Fei <[email protected]> wrote: > >>> > >>> Great, thank you! I just took this for a quick spin and it looks like > >>> there's DB migration task missing. The task you committed just > recently, > >>> 9635ae0956e7_index_faskfail.py, has a down_revision of 856955da8476 > >>> which can't be found when running airflow initdb (seehttps:// > >> github.com/apache/incubator-airflow/tree/v1-10-test/airflow/ > migrations/versions > >> ). > >>> Chris > >>> > >>> > >>>> On Wed, Jun 27, 2018, at 5:09 PM, Bolke de Bruin wrote: > >>>> Hi All, > >>>> > >>>> I have created a sdist package that is available at: > >>>> > >>>> > >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu > bating.tar.gz > >>>> < > >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu > bating.tar.gz>> > >> > >>>> In order to distinguish it from an actual (apache) release it is: > >>>> > >>>> 1. Marked as beta (python package managers do not install beta > >>>> versions by default - PEP 440)> 2. It is not signed > >>>> 3. It is not at an official apache distribution location > >>>> > >>>> You can also put something like this in a requirements.txt file: > >>>> > >>>> git+ > >>>> > >> https://github.com/apache/incubator-airflow@v1-10-test#egg= > apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql, > postgres,redis,slack,s3 > >>>> < > >> https://github.com/apache/incubator-airflow@v1-10-test#egg= > apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql, > postgres,redis,slack,s3 > >>> > >>>> ]> <https://github.com/rodrigc/incubator-airflow@master#egg=apache- > >>>> airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redi > s,slack,s- > >>>> 3] > >>>> < > >> https://github.com/rodrigc/incubator-airflow@master#egg=apac > he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres, > redis,slack,s3][1] > >>> > >>>>>> > >>>> and then "pip install -r requirements.txt”. > >>>> > >>>> I hope that after this beta we can go to RC and start voting on 1.10.> > >>>> Cheers > >>>> Bolke > >>> > >>> > >>> Links: > >>> > >>> 1. > >> https://github.com/rodrigc/incubator-airflow@master#egg=apac > he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres, > redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator- > airflow@master#egg=apache-airflow[celery,crypto,emr, > hive,hdfs,ldap,mysql,postgres,redis,slack,s3] > <https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-airflow@master%23egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]> > >> <https://github.com/rodrigc/incubator-airflow@master#egg=apa > che-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres, > redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator- > airflow@master%23egg=apache-airflow[celery,crypto,emr, > hive,hdfs,ldap,mysql,postgres,redis,slack,s3]> > >> > >
