I know of one that Alex wanted to get in, but wasn’t targeted for 1.8.1 in Jira and thus didn’t make the cut at RC time. There is is another one out that seems to have stalled a bit (https://github.com/apache/incubator-airflow/pull/2205 <https://github.com/apache/incubator-airflow/pull/2205>).
Reading the changelog of 1.8.1 I see bug fixes, apache requirements and one “new” feature (UI lightning bolt). Regressions could have happened but we have been quite vigilant on the fact that these bug fixes needed proper tests, so I am very interested in 1.8.0 -> 1.8.1 regressions. If it is a pre-backfill-change 1.8.0 to 1.8.1 regression then I would also like to know, cause I made that change and feel responsible for it. Cheers Bolke > On 3 May 2017, at 22:13, Dan Davydov <[email protected]> wrote: > > cc Alex and Rui who were working on fixes, I'm not sure if their commits got > in before 1.8.1. > > On Wed, May 3, 2017 at 1:09 PM, Bolke de Bruin <[email protected] > <mailto:[email protected]>> wrote: > Hi Dan, > > (Thread renamed to make sure it does not clash, dev@ now added) > > It surprises me that you found regression from 1.8.0 to 1.8.1 as 1.8.1 is > very much focused on bug fixes. Were the regressions shared yet? > > The whole 1.8.X release will be bug fix focused (per release management) and > minor feature updates. The 1.9.0 release will be the first release with major > feature updates. So what you want, more robustness and focus on stability, is > now underway. I agree with beefing up tests and including the major operators > in this. Executors should also be on this list btw. Turning on coverage > reporting might be a first step in helping this (it isn’t the solution > obviously). > > Cheers > Bolke > > >> On 3 May 2017, at 20:28, Dan Davydov <[email protected] >> <mailto:[email protected]>> wrote: >> >> We saw several regressions moving from 1.8.0 to 1.8.1 the first time we >> tried, and while I think we merged all our fixes to master (not sure if they >> all made it into 1.8.1 however), we have put releasing on hold due to >> stability issues from the last couple of releases. It's either the case that: >> A) Airbnb requires more robustness from new releases. >> or >> B) Most companies using Airflow require more robustness and we should halt >> on feature work until we are more confident in our testing >> >> I think the biggest problem currently is the lack of unit testing coverage, >> e.g. when the backfill framework was refactored (which was the right >> long-term fix), it caused a lot of breakages that weren't caught by tests. I >> think we need to audit the major operators/classes and beef up the unit >> testing coverage. The coverage metric does not necessarily cover these cases >> (e.g. cyclomatic complexity). Writing regression tests is good but we >> shouldn't have so many new blocker issues in our releases. >> >> We are fighting some fires internally at the moment (not Airflow related), >> but Alex and I have been working on some stuff that we will push to the >> community once we are done. Alex is working on a good solution for python >> package isolation, and I'm working on integration with Kubernetes at the >> executor level. >> >> Feel free to forward any of my messages to the dev mailing list. >> >> On Wed, May 3, 2017 at 11:18 AM, Bolke de Bruin <[email protected] >> <mailto:[email protected]>> wrote: >> Grrr, I seriously dislike to send button on the touch bar…here goes again. >> >> Hi Dan, >> >> (Please note I would like to forward the next message to dev@, but let me >> know if you don’t find it comfortable) >> >> I understand your point. The gap between 1.7.1 was large in terms of >> functionality changes etc. It was going to be a (bit?) rough and as you guys >> are using many of the edge cases you probably found more issues than any of >> us. Still, between 1.8.0 and 1.8.1 we have added many tests (coverage >> increased from 67% to close to 69%, which is a lot as you know). It would be >> nice if you can share where your areas of concern are so we can address >> those and a suggestion on how to proceed with integration tests is also >> welcome. >> >> You guys (=Airbnb) have been a bit quiet over the past couple of days, so I >> am getting a bit worried in terms of engagement. Is that warranted? >> >> Cheers >> Bolke >> >> >>> On 3 May 2017, at 20:13, Bolke de Bruin <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hi Dan, >>> >>> (Please note I would like to forward the next message to dev@, but let me >>> know if you don’t find it comfortable) >>> >>> I understand your point. The gap between 1.7.1 was large in terms of >>> functionality changes etc. It was going to be a (bit?) rough and as you >>> guys are using many of the edge cases you probably found more issues than >>> any of us. Still, between 1.8.0 and 1.8.1 we have added many tests >>> (coverage increased from 67 >>>> On 3 May 2017, at 19:41, Arthur Wiedmer <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> As a counterpoint, >>>> >>>> I am comfortable voting +1 on this release in the sense that it fixes some >>>> of the issues with 1.8.0. It is unfortunate that we cannot test it on the >>>> Airbnb production for now and we should definitely invest in increasing >>>> testing coverage, but some of the fixes are needed for ease of >>>> use/adoption (See for instance AIRFLOW-832), and this release is a step in >>>> the right direction. >>>> >>>> Best, >>>> Arthur >>>> >>>> On Wed, May 3, 2017 at 10:30 AM, Dan Davydov <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> I'm not comfortable voting without doing comprehensive staging and we >>>> aren't comfortable doing an internal lease for now until we fix the state >>>> of unit test coverage and integration tests. >>>> >>>> On May 3, 2017 8:42 AM, "Bolke de Bruin" <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> Hey Guys, >>>> >>>> Chris has been preparing the 1.8.1 release and the vote is running for it. >>>> Only one day left though! Would you mind casting your vote? Only Chris and >>>> I have voted binding until so far. >>>> >>>> (Please reply to the message on the list, not this message). >>>> >>>> Cheers! >>>> Bolke >>>> >>>>> Begin forwarded message: >>>>> >>>>> From: Chris Riccomini <[email protected] >>>>> <mailto:[email protected]>> >>>>> Subject: [VOTE] Release Airflow 1.8.1 RC2 >>>>> Date: 1 May 2017 at 19:58:41 GMT+2 >>>>> To: [email protected] >>>>> <mailto:[email protected]> >>>>> Reply-To: [email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> Dear All, >>>>> >>>>> _WARN: The package version for this RC is 1.8.1 (does not include RC2 in >>>>> version number). As such, any future 1.8.1 installatinos will have to be >>>>> force installed. PIP will not be able to distinguish between RCs and final >>>>> versions. Again, you'll have to force install the package. This can be >>>>> done >>>>> by adding `--force-reinstall` to your `pip install` commands._ >>>>> >>>>> I've made Airflow 1.8.1 RC2 available at: >>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow >>>>> <https://dist.apache.org/repos/dist/dev/incubator/airflow>, public keys >>>>> are >>>>> available at https://dist.apache.org/repos/dist/release/incubator/airflow >>>>> <https://dist.apache.org/repos/dist/release/incubator/airflow>. >>>>> >>>>> New issues fixed in 1.8.1 RC2: >>>>> >>>>> [AIRFLOW-1142] SubDAG Tasks Not Executed Even Though All Dependen >>>>> [AIRFLOW-1004] `airflow webserver -D` runs in foreground >>>>> [AIRFLOW-492] Insert into dag_stats table results into failed ta >>>>> >>>>> Issues fixed in 1.8.1 RC0/RC1, and included in RC2: >>>>> >>>>> [AIRFLOW-1138] Add licenses to files in scripts directory >>>>> [AIRFLOW-1127] Move license notices to LICENSE instead of NOTICE >>>>> [AIRFLOW-1124] Do not set all task instances to scheduled on back >>>>> [AIRFLOW-1120] Update version view to include Apache prefix >>>>> [AIRFLOW-1062] DagRun#find returns wrong result if external_trigg >>>>> [AIRFLOW-1054] Fix broken import on test_dag >>>>> [AIRFLOW-1050] Retries ignored - regression >>>>> [AIRFLOW-1033] TypeError: can't compare datetime.datetime to None >>>>> [AIRFLOW-1017] get_task_instance should return None instead of th >>>>> [AIRFLOW-1011] Fix bug in BackfillJob._execute() for SubDAGs >>>>> [AIRFLOW-1001] Landing Time shows "unsupported operand type(s) fo >>>>> [AIRFLOW-1000] Rebrand to Apache Airflow instead of Airflow >>>>> [AIRFLOW-989] Clear Task Regression >>>>> [AIRFLOW-974] airflow.util.file mkdir has a race condition >>>>> [AIRFLOW-906] Update Code icon from lightning bolt to file >>>>> [AIRFLOW-858] Configurable database name for DB operators >>>>> [AIRFLOW-853] ssh_execute_operator.py stdout decode default to A >>>>> [AIRFLOW-832] Fix debug server >>>>> [AIRFLOW-817] Trigger dag fails when using CLI + API >>>>> [AIRFLOW-816] Make sure to pull nvd3 from local resources >>>>> [AIRFLOW-815] Add previous/next execution dates to available def >>>>> [AIRFLOW-813] Fix unterminated unit tests in tests.job (tests/jo >>>>> [AIRFLOW-812] Scheduler job terminates when there is no dag file >>>>> [AIRFLOW-806] UI should properly ignore DAG doc when it is None >>>>> [AIRFLOW-794] Consistent access to DAGS_FOLDER and SQL_ALCHEMY_C >>>>> [AIRFLOW-785] ImportError if cgroupspy is not installed >>>>> [AIRFLOW-784] Cannot install with funcsigs > 1.0.0 >>>>> [AIRFLOW-780] The UI no longer shows broken DAGs >>>>> [AIRFLOW-777] dag_is_running is initlialized to True instead of >>>>> [AIRFLOW-719] Skipped operations make DAG finish prematurely >>>>> [AIRFLOW-694] Empty env vars do not overwrite non-empty config v >>>>> [AIRFLOW-139] Executing VACUUM with PostgresOperator >>>>> [AIRFLOW-111] DAG concurrency is not honored >>>>> [AIRFLOW-88] Improve clarity Travis CI reports >>>>> >>>>> I would like to raise a VOTE for releasing 1.8.1 based on release >>>>> candidate >>>>> 2. >>>>> >>>>> Please respond to this email by: >>>>> >>>>> +1,0,-1 with *binding* if you are a PMC member or *non-binding* if you are >>>>> not. >>>>> >>>>> Vote will run for 72 hours (ends this Thursday). >>>>> >>>>> Thanks! >>>>> Chris >>>>> >>>>> My VOTE: +1 (binding) >>>> >>>> >>>> >>>> >>>> -- >>>> Arthur Wiedmer >>>> (Pronouns: He, Him) >>>> Data Engineering, Airbnb >>>> www.airbnb.com <http://www.airbnb.com/> >> >> > >
