Re: [CANCEL][VOTE] Airflow 1.9.0rc1

Hi Chris, It seems that all of them are merged into the 1.9.0-stable branch, except for one that is only committed to master. Can we get RC2 out? Bolke Sent from my iPhone > On 10 Nov 2017, at 17:27, Chris Riccomini wrote: > > Hey all, > > I'm going to cancel the

Re: [VOTE] Airflow 1.9.0rc1

Ash, There seem to be some solutions, but there are pretty hacky and poorly documented : https://stackoverflow.com/questions/18026980/python-setuptools-how-can-i-list-a-private-repository-under-install-requires https://github.com/pypa/pip/issues/2124 That said, we should be able to figure out a

[CANCEL][VOTE] Airflow 1.9.0rc1

Hey all, I'm going to cancel the RC1 vote due to the -1's. I'm tracking the following issues for RC2: https://issues.apache.org/jira/browse/AIRFLOW-1776 https://issues.apache.org/jira/browse/AIRFLOW-1794 https://issues.apache.org/jira/browse/AIRFLOW-1789

Re: [VOTE] Airflow 1.9.0rc1

The other difference is that if you depend upon airflow in a module where you want to put it in the install_requires section of a setup.py (not an application which has a requirements.txt, say) you can't use a git tag. Or at least I couldn't get it working. It doesn't make a difference a lot

Re: [VOTE] Airflow 1.9.0rc1

It’s not a major difference, but installing from a git repo via pip requires a completely different syntax, which complicates our tooling, e.g.: $ pip install 'apache-airflow[postgres,celery,rabbitmq]=={{version}}' $ pip install

Re: [VOTE] Airflow 1.9.0rc1

I agree with Bolke that it would be better to provide dev releases in PyPI, but my understanding was that, while not an official release channel, it still has the apache branding and we should be careful nonetheless. I am still confused as to why installing from a git tag or the like is not OK

Re: [VOTE] Airflow 1.9.0rc1

This is how pip handles RC/beta versions: > Pre-release Versions > Starting with v1.4, pip will only install stable versions as specified by > PEP426 by default. If a version cannot be parsed as a compliant PEP426 > version then it is assumed to be a pre-release. > If a Requirement specifier

Re: [VOTE] Airflow 1.9.0rc1

I'd be in favour of this for similar reasons to Alek. I think the "mentioned before" is in reference to my post from October 20 http://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201710.mbox/%3CD008C556-C67C-42EB-88C0-CFE440C93656%40firemirror.com%3E

Re: [VOTE] Airflow 1.9.0rc1

I think we should put this up for discussion. PyPi is not an official apache channel, so in theory we could put anything on PyPI. I also think (didn’t confirm) pip doesn’t upgrade to RC/beta etc. Any thoughts? Bolke. > On 9 Nov 2017, at 15:53, Arthur Wiedmer wrote: > > Hi

Re: [VOTE] Airflow 1.9.0rc1

Hi Alek, Technically, we cannot release a distribution on PyPI until we have voted on a release. And here usually a release artifact. It is a little convoluted in the case of Python, but we are getting the hang of it. That said, installing from a git reference is a possibility too if you want

Re: [VOTE] Airflow 1.9.0rc1

I think this has been mentioned before, but it would be much easier for us (my team) to test RCs if they were published to PyPI. Or is that against Apache guidelines? Alek On Thu, Nov 9, 2017 at 8:29 AM, Michael Crawford < michael.crawf...@modernizingmedicine.com> wrote: > Thanks. Yes I

Re: [VOTE] Airflow 1.9.0rc1

Thanks. Yes I understand it isn’t released yet. > On Nov 9, 2017, at 9:09 AM, Driesprong, Fokko wrote: > > Hi Michael, > > You have to install it from the tar.gz: > > wget >

Re: [VOTE] Airflow 1.9.0rc1

Hi Michael, You have to install it from the tar.gz: wget https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0rc1/apache-airflow-1.9.0rc1+incubating-bin.tar.gz pip install /tmp/apache-airflow.tar.gz The steps of updating, are in the UPDATING.md:

Re: [VOTE] Airflow 1.9.0rc1

Curious how you guys are installing 1.9.0rc1 in your environments. Are you upgrading 1.8 environments or setting up a new test environment? Do we have any official documentation yet as part of the 1.9 release that documents how to upgrade from 1.8 to 1.9? > On Nov 9, 2017, at 7:44 AM, Ash

Re: [VOTE] Airflow 1.9.0rc1

That final URL should have been https://issues.apache.org/jira/browse/AIRFLOW-1797 , which now has a PR for it https://github.com/apache/incubator-airflow/pull/2771 There's going to be

Re: [VOTE] Airflow 1.9.0rc1

And one more - on Python3 we can't use S3Hook.load_string due to bytes vs string issue: https://issues.apache.org/jira/browse/AIRFLOW-1796 I'll try and work on fixes for some/all of these today, including adding and expanding on the tests

Re: [VOTE] Airflow 1.9.0rc1

Thanks for picking this up. Your fix should stop the 500 error, but there's another problem (which is ultimately user misconfiguration about) https://issues.apache.org/jira/browse/AIRFLOW-1796 - the fix for that to update a doc somewhere, and probably validate this settings is correct at start

Re: [VOTE] Airflow 1.9.0rc1

Still testing this out. Put up a small fix for Ash's second exception https://github.com/apache/incubator-airflow/pull/2766 On Wed, Nov 8, 2017 at 10:48 AM, Bolke de Bruin wrote: > Hi Chris, > > Actively testing here: we found an issue in the SSHOperator introduced in >

Re: [VOTE] Airflow 1.9.0rc1

Hi Chris, Actively testing here: we found an issue in the SSHOperator introduced in 1.9.0 (fix already merged for RC2, but blocking I as it stops us from running SSH properly), some minor fixes by Airbnb should also be in RC2. There is some logging “weirdness”, that might warrant a small patch

Re: [VOTE] Airflow 1.9.0rc1

Hi all, +1 (binding) I got Airflow 1.9rc1 deployed today, had some delay. So far it works great, except one thing. I've got some weird stuff in my log, but I'm not sure if it is Airflow, or a misbehaving process on my side: [2017-11-08 16:31:05,842] {logging_mixin.py:91} WARNING - Traceback

Re: [VOTE] Airflow 1.9.0rc1

-1 (for now. Non binding. Is that how this process works?) We've built a test env for this RC and are testing, but have run into an issue reading task logs. (See below) We haven't gotten very far with this yet, we will dig more tomorrow (it's the end of the UK work day now). I suspect this

Re: [VOTE] Airflow 1.9.0rc1

+1 -- Ruslan Dautkhanov On Wed, Nov 8, 2017 at 10:46 AM, Chris Riccomini wrote: > Anyone? :/ > > On Mon, Nov 6, 2017 at 1:22 PM, Chris Riccomini > wrote: > > > Hey all, > > > > I have cut Airflow 1.9.0 RC1. This email is calling a vote on the >

Re: [VOTE] Airflow 1.9.0rc1

Anyone? :/ On Mon, Nov 6, 2017 at 1:22 PM, Chris Riccomini wrote: > Hey all, > > I have cut Airflow 1.9.0 RC1. This email is calling a vote on the release, > which will last fo 72 hours. Consider this my (binding) +1. > > Airflow 1.9.0 RC1 is available at: > >

[VOTE] Airflow 1.9.0rc1

Hey all, I have cut Airflow 1.9.0 RC1. This email is calling a vote on the release, which will last fo 72 hours. Consider this my (binding) +1. Airflow 1.9.0 RC1 is available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0rc1/ apache-airflow-1.9.0rc1+incubating-source.tar.gz