Re: [Reminder] Sep 24 Airflow Bay Area Meetup @ Google

Hi all, Please see the meetup slides/videos below: - slides: https://drive.google.com/corp/drive/folders/1maD7E3BRv1kuH2dlsFfGyOFd8pf-vAE8 - videos: Airbender - Combining Kubernetes, Airflow, and Github Functional Yet Cost-effective Data Engineering Wi

Re: Manual validation operator

It's a bit of a hack, but to save up slots you could just have an instantly-failing PythonOperator (just raise an exception in the callable) that would go in a failed state. Marking it as "success" when the conditions are met would act as a trigger. On Fri, Oct 5, 2018 at 9:07 AM Brian Greene wro

Re: Pinning dependencies for Apache Airflow

Hello Erik, I understand your concern. It's a hard one to solve in general (i.e. dependency-hell). It looks like in this case you treat Airflow as 'library', where for some other people it might be more like 'end product'. If you look at the "pinning" philosophy - the "pin everything" is good for

Re: Flask-AppBuilder has pinned versions of Click & Flask-Login in 1.10.0

Thank for this Ash. Pipenv works very well in 99% of cases and is vastly better than managing requirements files. Also, PYPA (Python Packaging Authority) officially recommends Pipenv. I gave poetry a try and it seems like it has a lot of potential. I did run into two errors, a max recursion depth

Re: Airflow Docs - RTD vs Apache Site

A few thoughts: * we absolutely have to serve a project site off of `airflow.apache.org`, that's an ASF requirement * maybe `airflow.apache.org` could be setup as a proxy to readthedocs-latest (?) [I'm on vacation and have very slow internet, so didn't research whether that's a documented use-case,

Re: Manual validation operator

My first thought was this, but my understanding is That if you had a large number of dags “waiting” the sensor would consume all the concurrency. And what if the user doesn’t approve? How about the dag you have as it’s last step writes to an api/db the status. Then 2 other dags (or one with a

Re: Pinning dependencies for Apache Airflow

For us, exact pinning of versions would be problematic. We have DAG code that shares direct and indirect dependencies with Airflow, e.g. lxml, requests, pyhive, future, thrift, tzlocal, psycopg2 and ldap3. If our DAG code for some reason needs a newer point release due to a bug that's fixed, the

Re: Pinning dependencies for Apache Airflow

I think one solution to release approach is to check as part of automated Travis build if all requirements are pinned with == (even the deep ones) and fail the build in case they are not for ALL versions (including dev). And of course we should document the approach of releases/upgrades etc. If we

Re: PR for refactoring Airflow SLAs

Hello all, Is there any update on the status of this PR? I discovered this following a request for help on StackOverflow (on creating SLAs on task duration https://stackoverflow.com/questions/52645422/sla-on-task-duration-airflow). If this is unlikely to implemented in the short term, is ther

Re: Pinning dependencies for Apache Airflow

One thing to point out here. Right now if you `pip install apache-airflow=1.10.0` in a clean environment it will fail. This is because we pin flask-login to 0.2.1 but flask-appbuilder is >= 1.11.1, so that pulls in 1.12.0 which requires flask-login >= 0.3. So I do think there is maybe somethin

Re: Pinning dependencies for Apache Airflow

Never tried poetry before, but it looks really good (it passes also my aesthetic filter for slick design of the webpage). Quick look and it passes a lot of criteria I have in my mind: - works on all platforms - easily installable with pip - uses standard PyPI repositories by default (but

Re: Flask-AppBuilder has pinned versions of Click & Flask-Login in 1.10.0

Oh I meant FAB 1.11.1. And it looks like the Jinja issue is a bug in pip-tools, where it treats a dep of "jina" as actually being "jinja>=CURRENT" https://github.com/pypa/pipenv/issues/2596 In short: pip-env isn't ready for real-world use yet? (I'm

Re: Pinning dependencies for Apache Airflow

Hi all, Have you considered looking into poetry[1]? I’ve had really good experiences with it, we specifically introduced it into our project because we were getting version conflicts, and it resolved them just fine. It properly supports semantic versioning, so package versions have upper bounds