Hi Dan, (Thread renamed to make sure it does not clash, dev@ now added)
It surprises me that you found regression from 1.8.0 to 1.8.1 as 1.8.1 is very much focused on bug fixes. Were the regressions shared yet? The whole 1.8.X release will be bug fix focused (per release management) and minor feature updates. The 1.9.0 release will be the first release with major feature updates. So what you want, more robustness and focus on stability, is now underway. I agree with beefing up tests and including the major operators in this. Executors should also be on this list btw. Turning on coverage reporting might be a first step in helping this (it isn’t the solution obviously). Cheers Bolke > On 3 May 2017, at 20:28, Dan Davydov <dan.davy...@airbnb.com> wrote: > > We saw several regressions moving from 1.8.0 to 1.8.1 the first time we > tried, and while I think we merged all our fixes to master (not sure if they > all made it into 1.8.1 however), we have put releasing on hold due to > stability issues from the last couple of releases. It's either the case that: > A) Airbnb requires more robustness from new releases. > or > B) Most companies using Airflow require more robustness and we should halt on > feature work until we are more confident in our testing > > I think the biggest problem currently is the lack of unit testing coverage, > e.g. when the backfill framework was refactored (which was the right > long-term fix), it caused a lot of breakages that weren't caught by tests. I > think we need to audit the major operators/classes and beef up the unit > testing coverage. The coverage metric does not necessarily cover these cases > (e.g. cyclomatic complexity). Writing regression tests is good but we > shouldn't have so many new blocker issues in our releases. > > We are fighting some fires internally at the moment (not Airflow related), > but Alex and I have been working on some stuff that we will push to the > community once we are done. Alex is working on a good solution for python > package isolation, and I'm working on integration with Kubernetes at the > executor level. > > Feel free to forward any of my messages to the dev mailing list. > > On Wed, May 3, 2017 at 11:18 AM, Bolke de Bruin <bdbr...@gmail.com > <mailto:bdbr...@gmail.com>> wrote: > Grrr, I seriously dislike to send button on the touch bar…here goes again. > > Hi Dan, > > (Please note I would like to forward the next message to dev@, but let me > know if you don’t find it comfortable) > > I understand your point. The gap between 1.7.1 was large in terms of > functionality changes etc. It was going to be a (bit?) rough and as you guys > are using many of the edge cases you probably found more issues than any of > us. Still, between 1.8.0 and 1.8.1 we have added many tests (coverage > increased from 67% to close to 69%, which is a lot as you know). It would be > nice if you can share where your areas of concern are so we can address those > and a suggestion on how to proceed with integration tests is also welcome. > > You guys (=Airbnb) have been a bit quiet over the past couple of days, so I > am getting a bit worried in terms of engagement. Is that warranted? > > Cheers > Bolke > > >> On 3 May 2017, at 20:13, Bolke de Bruin <bdbr...@gmail.com >> <mailto:bdbr...@gmail.com>> wrote: >> >> Hi Dan, >> >> (Please note I would like to forward the next message to dev@, but let me >> know if you don’t find it comfortable) >> >> I understand your point. The gap between 1.7.1 was large in terms of >> functionality changes etc. It was going to be a (bit?) rough and as you guys >> are using many of the edge cases you probably found more issues than any of >> us. Still, between 1.8.0 and 1.8.1 we have added many tests (coverage >> increased from 67 >>> On 3 May 2017, at 19:41, Arthur Wiedmer <arthur.wied...@airbnb.com >>> <mailto:arthur.wied...@airbnb.com>> wrote: >>> >>> As a counterpoint, >>> >>> I am comfortable voting +1 on this release in the sense that it fixes some >>> of the issues with 1.8.0. It is unfortunate that we cannot test it on the >>> Airbnb production for now and we should definitely invest in increasing >>> testing coverage, but some of the fixes are needed for ease of use/adoption >>> (See for instance AIRFLOW-832), and this release is a step in the right >>> direction. >>> >>> Best, >>> Arthur >>> >>> On Wed, May 3, 2017 at 10:30 AM, Dan Davydov <dan.davy...@airbnb.com >>> <mailto:dan.davy...@airbnb.com>> wrote: >>> I'm not comfortable voting without doing comprehensive staging and we >>> aren't comfortable doing an internal lease for now until we fix the state >>> of unit test coverage and integration tests. >>> >>> On May 3, 2017 8:42 AM, "Bolke de Bruin" <bdbr...@gmail.com >>> <mailto:bdbr...@gmail.com>> wrote: >>> Hey Guys, >>> >>> Chris has been preparing the 1.8.1 release and the vote is running for it. >>> Only one day left though! Would you mind casting your vote? Only Chris and >>> I have voted binding until so far. >>> >>> (Please reply to the message on the list, not this message). >>> >>> Cheers! >>> Bolke >>> >>>> Begin forwarded message: >>>> >>>> From: Chris Riccomini <criccom...@apache.org >>>> <mailto:criccom...@apache.org>> >>>> Subject: [VOTE] Release Airflow 1.8.1 RC2 >>>> Date: 1 May 2017 at 19:58:41 GMT+2 >>>> To: dev@airflow.incubator.apache.org >>>> <mailto:dev@airflow.incubator.apache.org> >>>> Reply-To: dev@airflow.incubator.apache.org >>>> <mailto:dev@airflow.incubator.apache.org> >>>> >>>> Dear All, >>>> >>>> _WARN: The package version for this RC is 1.8.1 (does not include RC2 in >>>> version number). As such, any future 1.8.1 installatinos will have to be >>>> force installed. PIP will not be able to distinguish between RCs and final >>>> versions. Again, you'll have to force install the package. This can be done >>>> by adding `--force-reinstall` to your `pip install` commands._ >>>> >>>> I've made Airflow 1.8.1 RC2 available at: >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow >>>> <https://dist.apache.org/repos/dist/dev/incubator/airflow>, public keys are >>>> available at https://dist.apache.org/repos/dist/release/incubator/airflow >>>> <https://dist.apache.org/repos/dist/release/incubator/airflow>. >>>> >>>> New issues fixed in 1.8.1 RC2: >>>> >>>> [AIRFLOW-1142] SubDAG Tasks Not Executed Even Though All Dependen >>>> [AIRFLOW-1004] `airflow webserver -D` runs in foreground >>>> [AIRFLOW-492] Insert into dag_stats table results into failed ta >>>> >>>> Issues fixed in 1.8.1 RC0/RC1, and included in RC2: >>>> >>>> [AIRFLOW-1138] Add licenses to files in scripts directory >>>> [AIRFLOW-1127] Move license notices to LICENSE instead of NOTICE >>>> [AIRFLOW-1124] Do not set all task instances to scheduled on back >>>> [AIRFLOW-1120] Update version view to include Apache prefix >>>> [AIRFLOW-1062] DagRun#find returns wrong result if external_trigg >>>> [AIRFLOW-1054] Fix broken import on test_dag >>>> [AIRFLOW-1050] Retries ignored - regression >>>> [AIRFLOW-1033] TypeError: can't compare datetime.datetime to None >>>> [AIRFLOW-1017] get_task_instance should return None instead of th >>>> [AIRFLOW-1011] Fix bug in BackfillJob._execute() for SubDAGs >>>> [AIRFLOW-1001] Landing Time shows "unsupported operand type(s) fo >>>> [AIRFLOW-1000] Rebrand to Apache Airflow instead of Airflow >>>> [AIRFLOW-989] Clear Task Regression >>>> [AIRFLOW-974] airflow.util.file mkdir has a race condition >>>> [AIRFLOW-906] Update Code icon from lightning bolt to file >>>> [AIRFLOW-858] Configurable database name for DB operators >>>> [AIRFLOW-853] ssh_execute_operator.py stdout decode default to A >>>> [AIRFLOW-832] Fix debug server >>>> [AIRFLOW-817] Trigger dag fails when using CLI + API >>>> [AIRFLOW-816] Make sure to pull nvd3 from local resources >>>> [AIRFLOW-815] Add previous/next execution dates to available def >>>> [AIRFLOW-813] Fix unterminated unit tests in tests.job (tests/jo >>>> [AIRFLOW-812] Scheduler job terminates when there is no dag file >>>> [AIRFLOW-806] UI should properly ignore DAG doc when it is None >>>> [AIRFLOW-794] Consistent access to DAGS_FOLDER and SQL_ALCHEMY_C >>>> [AIRFLOW-785] ImportError if cgroupspy is not installed >>>> [AIRFLOW-784] Cannot install with funcsigs > 1.0.0 >>>> [AIRFLOW-780] The UI no longer shows broken DAGs >>>> [AIRFLOW-777] dag_is_running is initlialized to True instead of >>>> [AIRFLOW-719] Skipped operations make DAG finish prematurely >>>> [AIRFLOW-694] Empty env vars do not overwrite non-empty config v >>>> [AIRFLOW-139] Executing VACUUM with PostgresOperator >>>> [AIRFLOW-111] DAG concurrency is not honored >>>> [AIRFLOW-88] Improve clarity Travis CI reports >>>> >>>> I would like to raise a VOTE for releasing 1.8.1 based on release candidate >>>> 2. >>>> >>>> Please respond to this email by: >>>> >>>> +1,0,-1 with *binding* if you are a PMC member or *non-binding* if you are >>>> not. >>>> >>>> Vote will run for 72 hours (ends this Thursday). >>>> >>>> Thanks! >>>> Chris >>>> >>>> My VOTE: +1 (binding) >>> >>> >>> >>> >>> -- >>> Arthur Wiedmer >>> (Pronouns: He, Him) >>> Data Engineering, Airbnb >>> www.airbnb.com <http://www.airbnb.com/> > >
