I don't think any of the fixes I did were regressions. On Thu, May 4, 2017 at 8:11 AM, Bolke de Bruin <[email protected]> wrote:
> I know of one that Alex wanted to get in, but wasn’t targeted for 1.8.1 in > Jira and thus didn’t make the cut at RC time. There is is another one out > that seems to have stalled a bit (https://github.com/apache/ > incubator-airflow/pull/2205). > > Reading the changelog of 1.8.1 I see bug fixes, apache requirements and > one “new” feature (UI lightning bolt). Regressions could have happened but > we have been quite vigilant on the fact that these bug fixes needed proper > tests, so I am very interested in 1.8.0 -> 1.8.1 regressions. If it is a > pre-backfill-change 1.8.0 to 1.8.1 regression then I would also like to > know, cause I made that change and feel responsible for it. > > Cheers > Bolke > > > On 3 May 2017, at 22:13, Dan Davydov <[email protected]> wrote: > > cc Alex and Rui who were working on fixes, I'm not sure if their commits > got in before 1.8.1. > > On Wed, May 3, 2017 at 1:09 PM, Bolke de Bruin <[email protected]> wrote: > >> Hi Dan, >> >> (Thread renamed to make sure it does not clash, dev@ now added) >> >> It surprises me that you found regression from 1.8.0 to 1.8.1 as 1.8.1 is >> very much focused on bug fixes. Were the regressions shared yet? >> >> The whole 1.8.X release will be bug fix focused (per release management) >> and minor feature updates. The 1.9.0 release will be the first release with >> major feature updates. So what you want, more robustness and focus on >> stability, is now underway. I agree with beefing up tests and including the >> major operators in this. Executors should also be on this list btw. Turning >> on coverage reporting might be a first step in helping this (it isn’t the >> solution obviously). >> >> Cheers >> Bolke >> >> >> On 3 May 2017, at 20:28, Dan Davydov <[email protected]> wrote: >> >> We saw several regressions moving from 1.8.0 to 1.8.1 the first time we >> tried, and while I think we merged all our fixes to master (not sure if >> they all made it into 1.8.1 however), we have put releasing on hold due to >> stability issues from the last couple of releases. It's either the case >> that: >> A) Airbnb requires more robustness from new releases. >> or >> B) Most companies using Airflow require more robustness and we should >> halt on feature work until we are more confident in our testing >> >> I think the biggest problem currently is the lack of unit testing >> coverage, e.g. when the backfill framework was refactored (which was the >> right long-term fix), it caused a lot of breakages that weren't caught by >> tests. I think we need to audit the major operators/classes and beef up the >> unit testing coverage. The coverage metric does not necessarily cover these >> cases (e.g. cyclomatic complexity). Writing regression tests is good but we >> shouldn't have so many new blocker issues in our releases. >> >> We are fighting some fires internally at the moment (not Airflow >> related), but Alex and I have been working on some stuff that we will push >> to the community once we are done. Alex is working on a good solution for >> python package isolation, and I'm working on integration with Kubernetes at >> the executor level. >> >> Feel free to forward any of my messages to the dev mailing list. >> >> On Wed, May 3, 2017 at 11:18 AM, Bolke de Bruin <[email protected]> >> wrote: >> >>> Grrr, I seriously dislike to send button on the touch bar…here goes >>> again. >>> >>> Hi Dan, >>> >>> (Please note I would like to forward the next message to dev@, but let >>> me know if you don’t find it comfortable) >>> >>> I understand your point. The gap between 1.7.1 was large in terms of >>> functionality changes etc. It was going to be a (bit?) rough and as you >>> guys are using many of the edge cases you probably found more issues than >>> any of us. Still, between 1.8.0 and 1.8.1 we have added many tests >>> (coverage increased from 67% to close to 69%, which is a lot as you know). >>> It would be nice if you can share where your areas of concern are so we can >>> address those and a suggestion on how to proceed with integration tests is >>> also welcome. >>> >>> You guys (=Airbnb) have been a bit quiet over the past couple of days, >>> so I am getting a bit worried in terms of engagement. Is that warranted? >>> >>> Cheers >>> Bolke >>> >>> >>> On 3 May 2017, at 20:13, Bolke de Bruin <[email protected]> wrote: >>> >>> Hi Dan, >>> >>> (Please note I would like to forward the next message to dev@, but let >>> me know if you don’t find it comfortable) >>> >>> I understand your point. The gap between 1.7.1 was large in terms of >>> functionality changes etc. It was going to be a (bit?) rough and as you >>> guys are using many of the edge cases you probably found more issues than >>> any of us. Still, between 1.8.0 and 1.8.1 we have added many tests >>> (coverage increased from 67 >>> >>> On 3 May 2017, at 19:41, Arthur Wiedmer <[email protected]> >>> wrote: >>> >>> As a counterpoint, >>> >>> I am comfortable voting +1 on this release in the sense that it fixes >>> some of the issues with 1.8.0. It is unfortunate that we cannot test it on >>> the Airbnb production for now and we should definitely invest in increasing >>> testing coverage, but some of the fixes are needed for ease of use/adoption >>> (See for instance AIRFLOW-832), and this release is a step in the right >>> direction. >>> >>> Best, >>> Arthur >>> >>> On Wed, May 3, 2017 at 10:30 AM, Dan Davydov <[email protected]> >>> wrote: >>> >>>> I'm not comfortable voting without doing comprehensive staging and we >>>> aren't comfortable doing an internal lease for now until we fix the state >>>> of unit test coverage and integration tests. >>>> >>>> On May 3, 2017 8:42 AM, "Bolke de Bruin" <[email protected]> wrote: >>>> >>>>> Hey Guys, >>>>> >>>>> Chris has been preparing the 1.8.1 release and the vote is running for >>>>> it. Only one day left though! Would you mind casting your vote? Only Chris >>>>> and I have voted binding until so far. >>>>> >>>>> (Please reply to the message on the list, not this message). >>>>> >>>>> Cheers! >>>>> Bolke >>>>> >>>>> Begin forwarded message: >>>>> >>>>> *From: *Chris Riccomini <[email protected]> >>>>> *Subject: **[VOTE] Release Airflow 1.8.1 RC2* >>>>> *Date: *1 May 2017 at 19:58:41 GMT+2 >>>>> *To: *[email protected] >>>>> *Reply-To: *[email protected] >>>>> >>>>> Dear All, >>>>> >>>>> _WARN: The package version for this RC is 1.8.1 (does not include RC2 >>>>> in >>>>> version number). As such, any future 1.8.1 installatinos will have to >>>>> be >>>>> force installed. PIP will not be able to distinguish between RCs and >>>>> final >>>>> versions. Again, you'll have to force install the package. This can be >>>>> done >>>>> by adding `--force-reinstall` to your `pip install` commands._ >>>>> >>>>> I've made Airflow 1.8.1 RC2 available at: >>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow, public keys >>>>> are >>>>> available at https://dist.apache.org/repos/ >>>>> dist/release/incubator/airflow. >>>>> >>>>> New issues fixed in 1.8.1 RC2: >>>>> >>>>> [AIRFLOW-1142] SubDAG Tasks Not Executed Even Though All Dependen >>>>> [AIRFLOW-1004] `airflow webserver -D` runs in foreground >>>>> [AIRFLOW-492] Insert into dag_stats table results into failed ta >>>>> >>>>> Issues fixed in 1.8.1 RC0/RC1, and included in RC2: >>>>> >>>>> [AIRFLOW-1138] Add licenses to files in scripts directory >>>>> [AIRFLOW-1127] Move license notices to LICENSE instead of NOTICE >>>>> [AIRFLOW-1124] Do not set all task instances to scheduled on back >>>>> [AIRFLOW-1120] Update version view to include Apache prefix >>>>> [AIRFLOW-1062] DagRun#find returns wrong result if external_trigg >>>>> [AIRFLOW-1054] Fix broken import on test_dag >>>>> [AIRFLOW-1050] Retries ignored - regression >>>>> [AIRFLOW-1033] TypeError: can't compare datetime.datetime to None >>>>> [AIRFLOW-1017] get_task_instance should return None instead of th >>>>> [AIRFLOW-1011] Fix bug in BackfillJob._execute() for SubDAGs >>>>> [AIRFLOW-1001] Landing Time shows "unsupported operand type(s) fo >>>>> [AIRFLOW-1000] Rebrand to Apache Airflow instead of Airflow >>>>> [AIRFLOW-989] Clear Task Regression >>>>> [AIRFLOW-974] airflow.util.file mkdir has a race condition >>>>> [AIRFLOW-906] Update Code icon from lightning bolt to file >>>>> [AIRFLOW-858] Configurable database name for DB operators >>>>> [AIRFLOW-853] ssh_execute_operator.py stdout decode default to A >>>>> [AIRFLOW-832] Fix debug server >>>>> [AIRFLOW-817] Trigger dag fails when using CLI + API >>>>> [AIRFLOW-816] Make sure to pull nvd3 from local resources >>>>> [AIRFLOW-815] Add previous/next execution dates to available def >>>>> [AIRFLOW-813] Fix unterminated unit tests in tests.job (tests/jo >>>>> [AIRFLOW-812] Scheduler job terminates when there is no dag file >>>>> [AIRFLOW-806] UI should properly ignore DAG doc when it is None >>>>> [AIRFLOW-794] Consistent access to DAGS_FOLDER and SQL_ALCHEMY_C >>>>> [AIRFLOW-785] ImportError if cgroupspy is not installed >>>>> [AIRFLOW-784] Cannot install with funcsigs > 1.0.0 >>>>> [AIRFLOW-780] The UI no longer shows broken DAGs >>>>> [AIRFLOW-777] dag_is_running is initlialized to True instead of >>>>> [AIRFLOW-719] Skipped operations make DAG finish prematurely >>>>> [AIRFLOW-694] Empty env vars do not overwrite non-empty config v >>>>> [AIRFLOW-139] Executing VACUUM with PostgresOperator >>>>> [AIRFLOW-111] DAG concurrency is not honored >>>>> [AIRFLOW-88] Improve clarity Travis CI reports >>>>> >>>>> I would like to raise a VOTE for releasing 1.8.1 based on release >>>>> candidate >>>>> 2. >>>>> >>>>> Please respond to this email by: >>>>> >>>>> +1,0,-1 with *binding* if you are a PMC member or *non-binding* if you >>>>> are >>>>> not. >>>>> >>>>> Vote will run for 72 hours (ends this Thursday). >>>>> >>>>> Thanks! >>>>> Chris >>>>> >>>>> My VOTE: +1 (binding) >>>>> >>>>> >>>>> >>> >>> >>> -- >>> Arthur Wiedmer >>> (Pronouns: He, Him) >>> Data Engineering, Airbnb >>> www.airbnb.com >>> >>> >>> >>> >> >> > >
