Can I help you somehow with failing master? On Mon, Feb 3, 2020 at 7:44 AM Michał Słowikowski < michal.slowikow...@polidea.com> wrote:
> Thanks Jarek, awesome work! > > On Sun, Feb 2, 2020 at 11:09 AM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > >> Still seems that the "timeout" in the last kerberos job is back now >> (intermittent) - seems to appear when we run more of those builds in >> parallel. >> So still one more diagnosis/fix is needed I am afraid. >> >> On Sun, Feb 2, 2020 at 11:06 AM Ash Berlin-Taylor <a...@apache.org> wrote: >> >> > Great work Jarek! >> > >> > On 2 February 2020 09:18:52 GMT, Jarek Potiuk <jarek.pot...@polidea.com >> > >> > wrote: >> >> >> >> Ok. The master is fixed now (finally!). The master is now working so >> please >> >> rebase all of your open PRs to master. >> >> >> >> At the end we had a number of different problems, some coincidences >> at the >> >> same time that’s why it was so hectic and difficult to diagnose: >> >> >> >> - Travis queue was stalled (at some point in time we had some 20 >> builds >> >> waiting in a queue) so we did not rebase some merges to save time >> and >> >> merged them from old masters >> >> - Some of the master merges were cancelled - so we could not see >> which >> >> commit broke the build - that make us come up with different >> hypothesis for >> >> the problem >> >> - Our optimisations for CI builds optimisations (skip Kubernetes >> builds >> >> when no kubernetes-related changes) cause the contrib/example_dags >> move to >> >> slip under the radar of PR CI checks >> >> - Even if we did not have the optimisations - Kubernetes Git Sync >> uses >> >> master of Airflow, so we would not have detected that by PR failure >> (only >> >> after merge) >> >> - We had a number of “false positives” and lack of detailed logs for >> >> Kubernetes. >> >> - We had a mysterious hang on kerberos tests - but it was caused >> likely >> >> by Travis environment change (it’s gone now) >> >> - We had Redis test failures caused by 3.4 release of redis-py >> libraries >> >> which contained a change (Redis class became un-hashable by adding >> __eq__ >> >> hook) - luckily they reverted it two hours ago ( >> >> https://github.com/andymccurdy/redis-py/blob/master/CHANGES) >> >> - We downloaded Apache RAT tool from a maven repository. And this >> maven >> >> repo is very unstable recently. >> >> - There are a number of follow-up PRs (already merged or building on >> >> Travis now) that will resolve those problems and prevent it in the >> future. >> >> >> >> J. >> >> >> >> >> >> On Thu, Jan 30, 2020 at 11:16 AM Ash Berlin-Taylor <a...@apache.org> >> wrote: >> >> >> >> Spent a little bit of time looking at this and it seems it was (super) >> >>> flaky tests -- I've managed to get 1 commit back on master passing >> by just >> >>> retrying the one failed job. >> >>> >> >>> Looking at the latest commit now. >> >>> >> >>> On Jan 30 2020, at 7:54 am, Jarek Potiuk <jarek.pot...@polidea.com> >> wrote: >> >>> >> >>>> It looks like we have a failing master - seems that yesterday's >> Travis' >> >>>> super-slow queue and a number of PRs that were merged without >> rebasing >> >>>> >> >>> and >> >>> >> >>>> caused master to be broken. >> >>>> >> >>>> I will not be at my PC for couple of hours at least so maybe some >> other >> >>>> committers can take a look in the meantime. >> >>>> >> >>>> J. >> >>>> >> >>>> -- >> >>>> Jarek Potiuk >> >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >>>> >> >>>> M: +48 660 796 129 <+48660796129> >> >>>> [image: Polidea] <https://www.polidea.com/> >> >>>> >> >>>> >> >>> >> >>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> > > > -- > > Michał Słowikowski > Polidea <https://www.polidea.com/> | Junior Software Engineer > > E: michal.slowikow...@polidea.com > > Unique Tech > Check out our projects! <https://www.polidea.com/our-work> > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Junior Software Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
