Re: New PMCs: Daniel Imberman, Tomek Turbaszek & Kamil Breguła
Congrats guys! On Mon, Jul 20, 2020 at 8:53 PM QP Hou wrote: > Congratulations to Daniel, Tomek and Kamil, well deserved! > > Thanks, > QP Hou > > > On Mon, Jul 20, 2020 at 7:21 AM Kaxil Naik wrote: > > > Hello Airflow Community, > > > > Most of you might already be aware of this news but let me officially > > announce it. > > > > The Project Management Committee (PMC) for Apache Airflow > > has invited *Daniel Imberman*, *Tomek Turbaszek* & *Kamil Breguła* to > > become PMC Members and we are pleased > > to announce that they have accepted. > > > > Being a PMC member enables assistance with the management > > and to guide the direction of the project. > > > > All 3 of them have been a part of the community and Airflow committers > for > > a long time. > > > > Congratulations Daniel, Tomek & Kamil > > > > Regards, > > Kaxil > > > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Junior Software Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: New committer: Jiajie Zhong
Congratulations! Best regards M. wt., 31 mar 2020, 21:00 użytkownik Xinbin Huang napisał: > Congratulations! > > On Tue, Mar 31, 2020, 11:13 AM Felix Uellendall > wrote: > > > Welcome Jiajie! Congrats! Well deserved! :tada: > > > > Sent from ProtonMail Mobile > > > > On Tue, Mar 31, 2020 at 19:50, Sid Anand wrote: > > > > > Hi Jarek, > > > When you have time, please add this news to > > > > > > 1. Airflow twitter account > > > 2. Announcements wiki : > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements > > > > > > -s > > > > > > On Tue, Mar 31, 2020 at 10:47 AM Sid Anand wrote: > > > > > >> Welcome aboard Jiajie!! Glad to have you here. > > >> > > >> -s > > >> > > >> On Tue, Mar 31, 2020 at 10:41 AM Kevin Yang > wrote: > > >> > > >>> Congrats Jiajie! > > >>> > > >>> On Tue, Mar 31, 2020 at 10:39 AM Kaxil Naik > > wrote: > > >>> > > >>> > Congratulations Jiajie, welcome aboard. > > >>> > > > >>> > On Tue, Mar 31, 2020 at 6:31 PM Tomasz Urbaszek < > > turbas...@apache.org> > > >>> > wrote: > > >>> > > > >>> > > Congratulations Jiajie! Well deserved ;) > > >>> > > > > >>> > > T. > > >>> > > > > >>> > > On Tue, Mar 31, 2020 at 7:29 PM Jarek Potiuk < > > >>> jarek.pot...@polidea.com> > > >>> > > wrote: > > >>> > > > > >>> > > > The Project Management Committee (PMC) for Apache Airflow > > >>> > > > has invited Jiajie Zhong to become a committer and we are > pleased > > >>> > > > to announce that he has accepted. > > >>> > > > > > >>> > > > For quite some time Jiajie makes contributions that touch many > > >>> > different > > >>> > > > aspects of Apache Airflow - from the development environment to > > dag > > >>> > > > dependencies and the UI. Jiajie contributed a steady stream of > > >>> > > > improvements over more than a year. > > >>> > > > > > >>> > > > Also he answers a lot of Airflow questions at Stack Overflow > and > > >>> > > > runs a blog in > > >>> > > > Chinese where you can see Airflow a lot: > http://zhongjiajie.com/ > > >>> > > > > > >>> > > > Being a committer enables easier contribution to the > > >>> > > > project since there is no need to go via the patch > > >>> > > > submission process. This should enable better productivity. > > >>> > > > Being a PMC member enables assistance with the management > > >>> > > > and to guide the direction of the project. > > >>> > > > > > >>> > > > Please join me in congratulating Jiajie as a committer! > > >>> > > > > > >>> > > > J. > > >>> > > > > > >>> > > > > > >>> > > > -- > > >>> > > > > > >>> > > > Jarek Potiuk > > >>> > > > Polidea | Principal Software Engineer > > >>> > > > > > >>> > > > M: +48 660 796 129 > > >>> > > > > > >>> > > > > >>> > > > >>> > > >> >
Re: Requesting to give details on FacebookAdsHook project
Hi Chetan, we also have a slack channel. https://apache-airflow-slack.herokuapp.com/ Come on and join. Best regards Michał On Wed, Mar 25, 2020 at 7:16 PM Daniel Imberman wrote: > Hi Chetan, > > Please see our documentation here https://airflow.apache.org/. Please > feel free to reach out if you have further questions. > On Mar 25, 2020, 10:29 AM -0700, Chetan Agarwal , > wrote: > > Dear Sir, > > > > I am Chetan Agarwal, third-year Computer Science and Engineering > > Student at National Institute of Technology Karnataka, Surathkal. I would > > like to know more about this project. Can you please share some resources > > or details of this project. I have some prior experience with google Ads > on > > mobile. > > > > Yours sincerely, > > Chetan Agarwal > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Junior Software Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: [VOTE] Switch from using Jira to Github Issues
+1 (non-binding) On Tue, Mar 17, 2020 at 10:13 PM Sid Anand wrote: > (resending from my apache account .. sorry) > > +1 Binding > > Airflow's returning full-circle to Github issues after 4 years :-) > -s > > On Tue, Mar 17, 2020 at 2:12 PM Sid Anand wrote: > > > +1 Binding > > > > Airflow's returning full-circle to Github issues after 4 years :-) > > > > -s > > > > On Tue, Mar 17, 2020 at 6:48 AM Daniel Imberman < > daniel.imber...@gmail.com> > > wrote: > > > >> +1 Binding > >> > >> No point in having a whole extra site if we’re not using any of the JIRA > >> specific features > >> > >> Daniel > >> On Mar 16, 2020, 9:58 PM -0700, Sumit Maheshwari < > sumeet.ma...@gmail.com>, > >> wrote: > >> > +1 binding > >> > > >> > On Mon, Mar 16, 2020 at 11:44 PM Kevin Yang > wrote: > >> > > >> > > +1 binding > >> > > > >> > > On Mon, Mar 16, 2020 at 9:34 AM Tao Feng > wrote: > >> > > > >> > > > +1 > >> > > > > >> > > > On Mon, Mar 16, 2020 at 7:36 AM Ash Berlin-Taylor > > >> > > wrote: > >> > > > > >> > > > > The original discuss thread is here. > >> > > > > > >> > > > > >> > > > >> > https://lists.apache.org/thread.html/r19014fac5aa2aa15a86890aa438940da263e430cb535ca451ef451a4%40%3Cdev.airflow.apache.org%3E > >> > > > > > >> > > > > Proposal is: > >> > > > > > >> > > > > - We create a few appropriate issue templates in ./github -- > >> > > > specifically > >> > > > > also one for "Ask for help" that basically says "Don't ask here, > >> this > >> > > > will > >> > > > > be closed". > >> > > > > - We ask Infra to enable Github Issues on our apache/airflow > repo > >> > > > > - We remove the requirement on code changes to need an > >> accompanying > >> > > Jira > >> > > > > ticket (and we don't replace it with needing a Github Issue > >> either) > >> > > > > - We use Github Milestones to target which PRs should be > >> backported to > >> > > > > release branches. > >> > > > > - We update/port the dev/airflow_jira script from looking at > >> FixVersion > >> > > > in > >> > > > > Jira to look instead at the Milestone. > >> > > > > - We (committers/interested parties) start reviewing the open > Jira > >> > > > issues, > >> > > > > manually migrate the ones we think are still relevant to Github > >> Issues, > >> > > > > leave a link (Add Link, not just a comment) on the Jira Issue, > and > >> > > close > >> > > > it > >> > > > > with status "Auto Closed"(?). We could also add a > >> "migrated-to-github" > >> > > > > label to migrated issues. > >> > > > > - After the review is done we bulk-comment on any remaining open > >> issue > >> > > > > saying "We are migrating to Github, and believe this issue is no > >> longer > >> > > > > relevant. If you think otherwise please create an issue at > ". > >> > > > > - Ask Infra to disable creating new Jira issues for the AIRFLOW > >> > > project. > >> > > > > > >> > > > > There is a bit of a "contention/locking" issue about the > >> migration -- > >> > > how > >> > > > > to avoid creating duplicate issues. Not the end of the world if > >> we do I > >> > > > > guess. > >> > > > > > >> > > > > This vote will last for 72 hours, ending 2020-03-19T14:35:00Z, > and > >> > > until > >> > > > 3 > >> > > > > +1 votes are cast. > >> > > > > > >> > > > > This is my binding +1 vote. > >> > > > > > >> > > > > Thanks, > >> > > > > ash > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Junior Software Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: [DISCUSS] Stop using Jira (since we aren't using it properly)
ckets, > > rather > > > than searching to see if there already exists a > > ticket for > > > > > that feature. > > > For example: > ht > > tps:// > > > issues.apache.org/jira/browse/AIRFLOW-6987 and > > > > > > > https://issues.apache.org/jira/browse/AIRFLOW-2824 (I'm not trying > > > to > > > > > > > > > pick on anyone involved here, I just happened to notice this) > > > > > > > > > > > > > > > > Additionally most of the committers follow a similar path of "work > > > > on > > > > > > > > > feature, open Jira ticket just before creating PR". > > > I am > > > proposing we > > > migrate over to Github issues and drop the > > > requirement > > > to have a jira > > > ticket for PRs. > > > The one downside is we > > > might > > get > > > people opening > > > > > > issues for as an > "help, how do I do this" -- I think we can > > > address that > > > by having an issue > template saying something > > > like > > "DO > > > NOT OPEN AN ISSUE > > > ASKING FOR HELP - ask > on user > s@ or join > > > slack". > > > The only > > other thing Jira currently gives us is > > > > the ability mark tasks > for > > "backporting" -- I think we can > > > replace > > that > > > > with Github Milestones. > > > Kaxil or I will happily update the > > > scripts > > we > > > use > to build/check the > > status > of releases. > > > Thoughts? > > > > > > The only > outstandi > > ng question is then what do we do about > > migrating > > > > the issue (do > we > > copy issues across to Github?). Perhaps it > > > > might > > > be a good > opportunity > > > for a clean slate. > > > -ash > > > > > > > > > > > > > > > > > > > > -- Jarek Potiuk > > Polidea | Principal Software Engineer M: > > > > > > > +48 > > > 660 796 129 <+48660796129> > > [image: Polidea] > > > > > > > > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > > > === > > Please access the attached hyperlink for an important electronic > communications disclaimer: > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > === > > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Junior Software Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: Rebase to latest master
Thanks Jarek! On Sat, Mar 7, 2020 at 3:56 AM Jarek Potiuk wrote: > Please rebase to latest master - it has a fix for a strange Travis failure > - where some "system" tests were run where they should not be. > > Rebasing to the latest master should fix it. > > J. > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Junior Software Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: [DISCUSS] AIP-4 System tests
we already have separation to > >>> > > unit/integration tests in place - we can add support to system > tests > >>> using > >>> > > the same mechanisms. > >>> > > - With AIP-21 we grouped the tests into "providers" package and > that > >>> > > makes it easy to define boundaries of "systems" - every provider > is a > >>> > > "system" to test. > >>> > > - We have plenty of system tests implemented for GCP which we are > >>> going > >>> > > to use to run tests for backported packages from AIP-21 - we > followed > >>> > > system test automation for more than a year in GCP operators and we > >>> have it > >>> > > fully automated already. > >>> > > - In the latest PR - https://github.com/apache/airflow/pull/7389 > we > >>> even > >>> > > extracted all the GCP-specific way we run system tests in the way > to > >>> a) > >>> > > make it easy for everyone to write automated system tests b) make > it > >>> > > possible to be automated. > >>> > > - We have credits provided by Google to run our tests and we can > use > >>> > > them for regular runs of the system tests > >>> > > - We are close to switch-over to GitHub Actions, which will make it > >>> easy > >>> > > to write manually or regularly scheduled actions that will have > >>> securely > >>> > > stored credentials to run the system tests - in a way that it will > be > >>> > > controlled by committers and not abusable by contributors who > >>> prepare PRs. > >>> > > - I would like to start and lead a community-driven effort where we > >>> will > >>> > > split amongst community members writing missing tests - so that our > >>> new > >>> > > backport packages can be tested against latest-released version of > >>> 1.10.*. > >>> > > We will provide GCP tests as examples, we will also setup the > >>> automation > >>> > > needed to run the tests regularly - the only thing we will ask the > >>> members > >>> > > of the community is to write missing tests. This way I hope we can > >>> get very > >>> > > high coverage of backported packages. > >>> > > > >>> > > There are of course still a number of open questions - like how to > >>> store > >>> > > credentials, how often to run the tests etc. but I think those are > >>> > > implementation details that we can work out while we are > >>> implementing it. > >>> > > > >>> > > What do you think about it? If I have a lot of "yes's" quickly, I > >>> would > >>> > > love to start voting on AIP-4. > >>> > > > >>> > > J. > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > -- > >>> > > Jarek Potiuk > >>> > > Polidea <https://www.polidea.com/> | Principal Software Engineer > >>> > > > >>> > > M: +48 660 796 129 <+48660796129> > >>> > > [image: Polidea] <https://www.polidea.com/> > >>> > > >>> > > >>> > >>> > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Junior Software Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: Another day - another Travis CI problem
Jarek, retweeted & liked. Thanks for the information. On Thu, Feb 6, 2020 at 10:49 AM Jarek Potiuk wrote: > This time it seems that we continue having intermittent network issues from > Travis CI: > > I created a community issue here: > > https://travis-ci.community/t/repeated-network-errors-timeouts-duing-builds/7125 > > > And twited to @travisci -> > https://twitter.com/higrys/status/1225352453778034688 > > I'd love some community pressure on it and Travis's interest, so if you > could retweet/comment/like/upvote, I'd highly appreciate it. > > J. > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Junior Software Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: Failing Master
Can I help you somehow with failing master? On Mon, Feb 3, 2020 at 7:44 AM Michał Słowikowski < michal.slowikow...@polidea.com> wrote: > Thanks Jarek, awesome work! > > On Sun, Feb 2, 2020 at 11:09 AM Jarek Potiuk > wrote: > >> Still seems that the "timeout" in the last kerberos job is back now >> (intermittent) - seems to appear when we run more of those builds in >> parallel. >> So still one more diagnosis/fix is needed I am afraid. >> >> On Sun, Feb 2, 2020 at 11:06 AM Ash Berlin-Taylor wrote: >> >> > Great work Jarek! >> > >> > On 2 February 2020 09:18:52 GMT, Jarek Potiuk > > >> > wrote: >> >> >> >> Ok. The master is fixed now (finally!). The master is now working so >> please >> >> rebase all of your open PRs to master. >> >> >> >> At the end we had a number of different problems, some coincidences >> at the >> >> same time that’s why it was so hectic and difficult to diagnose: >> >> >> >>- Travis queue was stalled (at some point in time we had some 20 >> builds >> >>waiting in a queue) so we did not rebase some merges to save time >> and >> >>merged them from old masters >> >>- Some of the master merges were cancelled - so we could not see >> which >> >>commit broke the build - that make us come up with different >> hypothesis for >> >>the problem >> >>- Our optimisations for CI builds optimisations (skip Kubernetes >> builds >> >>when no kubernetes-related changes) cause the contrib/example_dags >> move to >> >>slip under the radar of PR CI checks >> >>- Even if we did not have the optimisations - Kubernetes Git Sync >> uses >> >>master of Airflow, so we would not have detected that by PR failure >> (only >> >>after merge) >> >>- We had a number of “false positives” and lack of detailed logs for >> >>Kubernetes. >> >>- We had a mysterious hang on kerberos tests - but it was caused >> likely >> >>by Travis environment change (it’s gone now) >> >>- We had Redis test failures caused by 3.4 release of redis-py >> libraries >> >>which contained a change (Redis class became un-hashable by adding >> __eq__ >> >>hook) - luckily they reverted it two hours ago ( >> >>https://github.com/andymccurdy/redis-py/blob/master/CHANGES) >> >>- We downloaded Apache RAT tool from a maven repository. And this >> maven >> >>repo is very unstable recently. >> >>- There are a number of follow-up PRs (already merged or building on >> >>Travis now) that will resolve those problems and prevent it in the >> future. >> >> >> >> J. >> >> >> >> >> >> On Thu, Jan 30, 2020 at 11:16 AM Ash Berlin-Taylor >> wrote: >> >> >> >> Spent a little bit of time looking at this and it seems it was (super) >> >>> flaky tests -- I've managed to get 1 commit back on master passing >> by just >> >>> retrying the one failed job. >> >>> >> >>> Looking at the latest commit now. >> >>> >> >>> On Jan 30 2020, at 7:54 am, Jarek Potiuk >> wrote: >> >>> >> >>>> It looks like we have a failing master - seems that yesterday's >> Travis' >> >>>> super-slow queue and a number of PRs that were merged without >> rebasing >> >>>> >> >>> and >> >>> >> >>>> caused master to be broken. >> >>>> >> >>>> I will not be at my PC for couple of hours at least so maybe some >> other >> >>>> committers can take a look in the meantime. >> >>>> >> >>>> J. >> >>>> >> >>>> -- >> >>>> Jarek Potiuk >> >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >>>> >> >>>> M: +48 660 796 129 <+48660796129> >> >>>> [image: Polidea] <https://www.polidea.com/> >> >>>> >> >>>> >> >>> >> >>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> > > > -- > > Michał Słowikowski > Polidea <https://www.polidea.com/> | Junior Software Engineer > > E: michal.slowikow...@polidea.com > > Unique Tech > Check out our projects! <https://www.polidea.com/our-work> > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Junior Software Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: Failing Master
Thanks Jarek, awesome work! On Sun, Feb 2, 2020 at 11:09 AM Jarek Potiuk wrote: > Still seems that the "timeout" in the last kerberos job is back now > (intermittent) - seems to appear when we run more of those builds in > parallel. > So still one more diagnosis/fix is needed I am afraid. > > On Sun, Feb 2, 2020 at 11:06 AM Ash Berlin-Taylor wrote: > > > Great work Jarek! > > > > On 2 February 2020 09:18:52 GMT, Jarek Potiuk > > wrote: > >> > >> Ok. The master is fixed now (finally!). The master is now working so > please > >> rebase all of your open PRs to master. > >> > >> At the end we had a number of different problems, some coincidences at > the > >> same time that’s why it was so hectic and difficult to diagnose: > >> > >>- Travis queue was stalled (at some point in time we had some 20 > builds > >>waiting in a queue) so we did not rebase some merges to save time and > >>merged them from old masters > >>- Some of the master merges were cancelled - so we could not see > which > >>commit broke the build - that make us come up with different > hypothesis for > >>the problem > >>- Our optimisations for CI builds optimisations (skip Kubernetes > builds > >>when no kubernetes-related changes) cause the contrib/example_dags > move to > >>slip under the radar of PR CI checks > >>- Even if we did not have the optimisations - Kubernetes Git Sync > uses > >>master of Airflow, so we would not have detected that by PR failure > (only > >>after merge) > >>- We had a number of “false positives” and lack of detailed logs for > >>Kubernetes. > >>- We had a mysterious hang on kerberos tests - but it was caused > likely > >>by Travis environment change (it’s gone now) > >>- We had Redis test failures caused by 3.4 release of redis-py > libraries > >>which contained a change (Redis class became un-hashable by adding > __eq__ > >>hook) - luckily they reverted it two hours ago ( > >>https://github.com/andymccurdy/redis-py/blob/master/CHANGES) > >>- We downloaded Apache RAT tool from a maven repository. And this > maven > >>repo is very unstable recently. > >>- There are a number of follow-up PRs (already merged or building on > >>Travis now) that will resolve those problems and prevent it in the > future. > >> > >> J. > >> > >> > >> On Thu, Jan 30, 2020 at 11:16 AM Ash Berlin-Taylor > wrote: > >> > >> Spent a little bit of time looking at this and it seems it was (super) > >>> flaky tests -- I've managed to get 1 commit back on master passing by > just > >>> retrying the one failed job. > >>> > >>> Looking at the latest commit now. > >>> > >>> On Jan 30 2020, at 7:54 am, Jarek Potiuk > wrote: > >>> > >>>> It looks like we have a failing master - seems that yesterday's > Travis' > >>>> super-slow queue and a number of PRs that were merged without rebasing > >>>> > >>> and > >>> > >>>> caused master to be broken. > >>>> > >>>> I will not be at my PC for couple of hours at least so maybe some > other > >>>> committers can take a look in the meantime. > >>>> > >>>> J. > >>>> > >>>> -- > >>>> Jarek Potiuk > >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>> > >>>> M: +48 660 796 129 <+48660796129> > >>>> [image: Polidea] <https://www.polidea.com/> > >>>> > >>>> > >>> > >>> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Junior Software Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: Improved CI stability and much more enjoyable Breeze development environment
Jarek, wow. Seems to be a great improvement for Breeze and pytests. Great
news.
Best regards
On Mon, Jan 13, 2020 at 10:57 AM Jarek Potiuk
wrote:
> TL;DR; I have a Green and ready for review PR that should vastly improve
> the stability of CI and make our development environment much more
> enjoyable: https://github.com/apache/airflow/pull/7091
>
> From the first experiences it is much more stable on Travis and we also
> have a working version with Github Actions so we should be able to move to
> GA shortly.
>
> It brings huge improvements to CI environment as well to local development
> one. It will be much easier and more convenient to reproduce failing CI
> tests.
>
> This is all documented in TESTING.rst and BREEZE.rst but here some summary
> of what I came up with.
>
> *Using pytest markers to separate Integration/Kubernetes tests:*
>
> 1) Appropriate tests are marked now with
> *pytest.mark.integration("")* marker (*cassandra kerberos
> mongo openldap rabbitmq redis*) - in case you do not have an integration
> enabled they will be skipped and the message will clearly say what
> integration is needed. You can also run only the tests for those
> integrations only with *pytest --integration cassandra* for example of
> *pytest
> --integration all*
>
> 2) There are some tests that require/work with certain backends only
> (*sqlite
> mysql postgres*). They are marked with *pytest.markers.backend("mysql")* or
> even *pytest.mark.backend("mysql", "postgres")* - if they are supposed to
> work with more than one backend. There are ~30 such tests. I corrected all
> custom skips/skipifs/skipunless to those new markers. Same as with
> integrations you can run all backend tests with *pytest --backend mysql*
>
> 3) There are Kubernetes tests that are marked with
> *pytest.mark.runtime("kubernetes")
> *. I chose a different marker (runtime) for those because they are
> completely independent and require more fundamental environment change -
> starting and deploying Kubernetes cluster. There are ~30 such tests. Again
> you can run only those tests with *pytest --runtime kubernetes.*
>
> 4) Breeze is now by default not running any additional
> containers/integrations - only base airflow. There are *~ 4400 *tests that
> should run in that environment not requiring any integrations (i.e. other
> containers) or runtime (i.e. kubernetes). So vast majority of the tests
> will run with this default setup. This makes the whole "Breeze" experience
> a lot more enjoyable as the default setting requires far less resources on
> your local machine.
>
> 5) You can also start Breeze with *--integration* flag (for example
> *./breeze
> --integration cassandra*) so that you have the integration to run
> appropriate integration tests (for example cassandra tests).
>
> 6) You can also start Breeze with "*--start-kind-cluster*" . This will
> start "Kubernetes in Docker" cluster. And with two scripts you can deploy
> Airflow to this cluster and run Kubernetes tests there.
>
> 7) Last but not least - we have more jobs in Travis (soon moving to Github
> Actions). These jobs should be far more stable, because either they do not
> require integrations to be started (so much less memory needed) or they run
> only integrations/kubernetes tests (they run 30 tests each so the memory
> requirement is also much smaller).
>
> We see much better now which tests are skipped and why and it will allow us
> to further cleanup our test base now.
>
> You can see a successful build here:
> https://travis-ci.org/apache/airflow/builds/636205535
>
> Looking forward to prompt review and merging it quickly.
>
> J.
>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
--
Michał Słowikowski
Polidea <https://www.polidea.com/> | Test Engineer
E: michal.slowikow...@polidea.com
Unique Tech
Check out our projects! <https://www.polidea.com/our-work>
Re: New Committer: Tomasz Urbaszek
Great job Tomek! Congratulations! On Fri, Dec 20, 2019 at 4:32 PM Kaxil Naik wrote: > Congrats Tomasz, welcome aboard and well deserved. > > On Fri, Dec 20, 2019, 15:28 Jarek Potiuk wrote: > > > Congrats Tomek! Great job! Well deserved indeed! > > > > J. > > > > On Fri, Dec 20, 2019 at 4:14 PM Felix Uellendall > > > wrote: > > > > > Congrats Tomek! Well deserved :) ! > > > > > > Sent from ProtonMail Mobile > > > > > > On Fri, Dec 20, 2019 at 16:10, Michał Słowikowski < > > > michal.slowikow...@polidea.com> wrote: > > > > > > > Great job Tomek! Congratulations! > > > > > > > > On Fri, Dec 20, 2019 at 3:58 PM cla.klaus > > > > > wrote: > > > > > > > >> Congratulations Tomasz!Claudio > > > >> Messaggio originale Da: "Driesprong, Fokko" > > > >> Data: 20/12/19 15:47 (GMT+01:00) A: > > > >> dev@airflow.apache.org, us...@airflow.apache.org Oggetto: New > > > Committer: > > > >> Tomasz Urbaszek Folks,The Project Management Committee (PMC) for > > Apache > > > >> Airflowhas invited Tomasz Urbaszek to become a committer and we > > > arepleased > > > >> to announce that he has accepted.He's super active in the community > > and > > > >> active on both Slackand the devlist helping others. He has done an > > > awesome > > > >> jobon moving from nosetests to pytest and has contributed > andreviewed > > > many > > > >> PR's.Being a committer enables easier contribution to theproject > since > > > >> there is no need to go via the patchsubmission process. This should > > > enable > > > >> better productivity.Being a PMC member enables assistance with the > > > >> managementand to guide the direction of the project.Please join me > in > > > >> congratulating Tomasz as a committer!Cheers, Fokko > > > > > > > > -- > > > > > > > > Michał Słowikowski > > > > Polidea <https://www.polidea.com/> | Test Engineer > > > > > > > > E: michal.slowikow...@polidea.com > > > > > > > > Unique Tech > > > > Check out our projects! <https://www.polidea.com/our-work> > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Test Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: New Committer: Tomasz Urbaszek
Great job Tomek! Congratulations! On Fri, Dec 20, 2019 at 3:58 PM cla.klaus wrote: > Congratulations Tomasz!Claudio > Messaggio originale Da: "Driesprong, Fokko" > Data: 20/12/19 15:47 (GMT+01:00) A: > dev@airflow.apache.org, us...@airflow.apache.org Oggetto: New Committer: > Tomasz Urbaszek Folks,The Project Management Committee (PMC) for Apache > Airflowhas invited Tomasz Urbaszek to become a committer and we arepleased > to announce that he has accepted.He's super active in the community and > active on both Slackand the devlist helping others. He has done an awesome > jobon moving from nosetests to pytest and has contributed andreviewed many > PR's.Being a committer enables easier contribution to theproject since > there is no need to go via the patchsubmission process. This should enable > better productivity.Being a PMC member enables assistance with the > managementand to guide the direction of the project.Please join me in > congratulating Tomasz as a committer!Cheers, Fokko -- Michał Słowikowski Polidea <https://www.polidea.com/> | Test Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: [DISCUSS] Revenge of the asserts
I like this idea, and such assertion looks very promising. assert "maximum recursion" in str(excinfo.value) Have a nice day! On Mon, Dec 9, 2019 at 6:16 PM Tomasz Urbaszek wrote: > It's definitely worth a try! I even mentioned this in AIP-27. But I think > this script does not change SetUp / TearDown to an appropriate fixture. > > T. > > On Mon, Dec 9, 2019 at 6:03 PM Jarek Potiuk > wrote: > > > Well. I think (as discussed with Kamil) maybe even convert the tests > > automatically. > > > > How about we try: https://github.com/pytest-dev/unittest2pytest > > > > J. > > > > On Mon, Dec 9, 2019 at 5:48 PM Tomasz Urbaszek < > > tomasz.urbas...@polidea.com> wrote: > > > >> I agree. Should we encourage people to use asserts when adding new > tests? > >> > >> T. > >> > >> On Mon, Dec 9, 2019 at 4:35 PM Kaxil Naik wrote: > >> > >>> Yup, we should. > >>> > >>> On Mon, Dec 9, 2019 at 3:20 PM Ash Berlin-Taylor > wrote: > >>> > >>> > Absolutely, and I don't think any of the concers about it in run-time > >>> code > >>> > apply! Plus it is the way pytest recommends, and I think we get nicer > >>> > failure messages using assert-style too? > >>> > > >>> > -a > >>> > > >>> > On 9 December 2019 15:06:07 GMT, Jarek Potiuk < > >>> jarek.pot...@polidea.com> > >>> > wrote: > >>> > >Hello everyone. > >>> > > > >>> > >So asserts are now banned from our main code. However with the > recent > >>> > >introduction of pytest we now have a chance to switch to using the > >>> > >standard > >>> > >asserts instead of deriving from TestCase class and using > >>> > >assertSomething() > >>> > >methods. > >>> > > > >>> > >I find it much more readable and nice and pytest is great in > reporting > >>> > >the > >>> > >errors in a clear and readable way. And all the cases where asserts > >>> are > >>> > >optimized away are not valid in this case. > >>> > > > >>> > >I think we should gradually switch to using asserts in our tests. > >>> > > > >>> > >WDYT? > >>> > > > >>> > >More info: > >>> > > > >>> > >Doc about asserts in pytest: > >>> > >http://doc.pytest.org/en/latest/assert.html > >>> > > > >>> > >Demo of common assertion errors produced with pytest: > >>> > > > >>> > http://doc.pytest.org/en/latest/example/reportingdemo.html#tbreportdemo > >>> > > > >>> > >J. > >>> > > > >>> > >-- > >>> > > > >>> > >Jarek Potiuk > >>> > >Polidea <https://www.polidea.com/> | Principal Software Engineer > >>> > > > >>> > >M: +48 660 796 129 <+48660796129> > >>> > >[image: Polidea] <https://www.polidea.com/> > >>> > > >>> > >> > >> > >> -- > >> > >> Tomasz Urbaszek > >> Polidea <https://www.polidea.com/> | Junior Software Engineer > >> > >> M: +48 505 628 493 <+48505628493> > >> E: tomasz.urbas...@polidea.com > >> > >> Unique Tech > >> Check out our projects! <https://www.polidea.com/our-work> > >> > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- > > Tomasz Urbaszek > Polidea <https://www.polidea.com/> | Junior Software Engineer > > M: +48 505 628 493 <+48505628493> > E: tomasz.urbas...@polidea.com > > Unique Tech > Check out our projects! <https://www.polidea.com/our-work> > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Test Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
Re: [VOTE] Accept new Airflow website contribution
; > > > > > > > > >> > > >> On Sat, Nov 16, 2019 at 12:10 AM Kaxil Naik < > > > > > > > > > kaxiln...@gmail.com > > > > > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >>> Sent on Slack > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > > >>> On Fri, Nov 15, 2019 at 10:45 PM Kamil > Breguła < > > > > > > > > > > >> > > >> kamil.breg...@polidea.com> > > > > > > > > > > >> > > >>> wrote: > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > > >>>> Hello, > > > > > > > > > > >> > > >>>> > > > > > > > > > > >> > > >>>> The mailing list does not support > attachments. > > > Can > > > > > you > > > > > > > > > forward > > > > > > > > > > a > > > > > > > > > > >> photo > > > > > > > > > > >> > > >>> to > > > > > > > > > > >> > > >>>> me? > > > > > > > > > > >> > > >>>> > > > > > > > > > > >> > > >>>> Best regards, > > > > > > > > > > >> > > >>>> > > > > > > > > > > >> > > >>>> On Fri, Nov 15, 2019 at 11:41 PM Kaxil Naik < > > > > > > > > > > kaxiln...@gmail.com > > > > > > > > > > >> > > > > > > > > > > > >> > > >> wrote: > > > > > > > > > > >> > > >>>>> > > > > > > > > > > >> > > >>>>> Awesome job guys. > > > > > > > > > > >> > > >>>>> > > > > > > > > > > >> > > >>>>> Few small comments: > > > > > > > > > > >> > > >>>>> > > > > > > > > > > >> > > >>>>> Can we please use Airflow logo with > > transparent > > > > > > > background > > > > > > > > > in > > > > > > > > > > >> > > >> favicon? > > > > > > > > > > >> > > >>>>> Does the box have more padding than needed > in > > > the > > > > > > > header? > > > > > > > > > > >> Screenshot > > > > > > > > > > >> > > >>>> below: > > > > > > > > > > >> > > >>>>> > > > > > > > > > > >> > > >>>>> Should we Remove the text in RoadMap and > show > > > TBD > > > > or > > > > > > > > > something > > > > > > > > > > >> when > > > > > > > > > > >> > > >> the > > > > > > > > > > >> > > >>>> website is live? > > > > > > > > > > >> > > >>>>> Can we remove the placeholder videos on > > > > > > > > > > >> > > >>>> > > > > > > > > > > >> > > > > > > > > > > > > https://airflow-website-s5z26d2t7a-ew.a.run.app/meetups/#Vkprq5F6jkM > > > > > > > > > > >> > > >>>>> We would need to remove placeholder for > > Project > > > > > > > Management > > > > > > > > > > >> Committee > > > > > > > > > > >> > > >>>> names > > > > > > > > > > > https://airflow-website-s5z26d2t7a-ew.a.run.app/community/ > > > > > > > > > > >> > > >> before > > > > > > > > > > >> > > >>>> we are live ! > > > > > > > > > > >> > > >>>>> > > > > > > > > > > >> > > >>>>> Regards, > > > > > > > > > > >> > > >>>>> Kaxil > > > > > > > > > > >> > > >>>>> > > > > > > > > > > >> > > >>>>> On Fri, Nov 15, 2019 at 10:30 PM Aizhamal > > > Nurmamat > > > > > > kyzy > > > > > > > < > > > > > > > > > > >> > > >>>> aizha...@apache.org> wrote: > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> Hi everyone, > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> We have completed the project of > developing a > > > new > > > > > > > > website, > > > > > > > > > > >> yay! This > > > > > > > > > > >> > > >>>> vote > > > > > > > > > > >> > > >>>>>> is to get your approval to merge the latest > > > > > prototype > > > > > > > [1] > > > > > > > > > > into > > > > > > > > > > >> > > >>>>>> airflow.apache.org. The vote will last > till > > > > > > Wednesday > > > > > > > > 19th > > > > > > > > > > >> 5pm PST, > > > > > > > > > > >> > > >>>> more > > > > > > > > > > >> > > >>>>>> than 72 hours because of the weekend. > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> If you have specific technical questions > > about > > > > how > > > > > to > > > > > > > > > > >> contribute to > > > > > > > > > > >> > > >> / > > > > > > > > > > >> > > >>>> build > > > > > > > > > > >> > > >>>>>> / test / release the website, please check > > out > > > > the > > > > > > > > > README[3] > > > > > > > > > > >> and > > > > > > > > > > >> > > >>>>>> CONTRIBUTE[4] files that are part of the > Pull > > > > > > Request. > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> There are still a couple issues that will > be > > > > solved > > > > > > > once > > > > > > > > > the > > > > > > > > > > >> website > > > > > > > > > > >> > > >>> is > > > > > > > > > > >> > > >>>>>> merged and they are being tracked in this > > > > document > > > > > > [2]. > > > > > > > > > These > > > > > > > > > > >> are > > > > > > > > > > >> > > >>>> related > > > > > > > > > > >> > > >>>>>> to setting up CI/CD for it. > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> We also have a wishlist in the same > document > > > that > > > > > is > > > > > > > not > > > > > > > > > part > > > > > > > > > > >> of the > > > > > > > > > > >> > > >>>>>> current scope of work, and it will be great > > to > > > > get > > > > > > more > > > > > > > > > > >> > > >> contributions > > > > > > > > > > >> > > >>>> from > > > > > > > > > > >> > > >>>>>> the community to address them. I will > create > > > JIRA > > > > > > > issues > > > > > > > > > for > > > > > > > > > > >> them > > > > > > > > > > >> > > >>> after > > > > > > > > > > >> > > >>>> we > > > > > > > > > > >> > > >>>>>> conclude merging the new website. > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> For those who would like changes to the > > > graphics > > > > / > > > > > > > > > appearance > > > > > > > > > > >> of the > > > > > > > > > > >> > > >>>>>> website, I would encourage you to allow > this > > PR > > > > to > > > > > > move > > > > > > > > > > >> forward so > > > > > > > > > > >> > > >> we > > > > > > > > > > >> > > >>>> can > > > > > > > > > > >> > > >>>>>> finalize the project; and work with us to > > > realize > > > > > any > > > > > > > > > gradual > > > > > > > > > > >> > > >>>> improvements > > > > > > > > > > >> > > >>>>>> after the current version is merged. > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> To see what the website looks like, please > > > check > > > > > the > > > > > > > > staged > > > > > > > > > > >> > > >>>> prototype[5]. > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> Thanks, > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> Aizhamal > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> [1] > > > > > https://github.com/apache/airflow-site/pull/174 > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> [2] > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1YIgz0vZiw7zJsLzCiPun6n1VqPH1OuZGaSA_7EBixT8/edit?usp=sharing > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> [3] > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow-site/blob/e601af1e7d121038e461d16cb7376086196e9059/README.md > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>>>> [4] > > > > > > > > > > >> > > >>>>>> > > > > > > > > > > >> > > >>>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow-site/blob/e601af1e7d121038e461d16cb7376086196e9059/CONTRIBUTE.md > > > > > > > > > > >> > > >>>>>> [5] > > > > > https://airflow-website-s5z26d2t7a-ew.a.run.app/ > > > > > > > > > > >> > > >>>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> -- > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> Tomasz Urbaszek > > > > > > > > > > >> > > >> Polidea <https://www.polidea.com/> | Junior > > > Software > > > > > > > > Engineer > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> M: +48 505 628 493 <+48%20505%20628%20493> > > > > > > > <+48%20505%20628%20493> <+48505628493 <+48%20505%20628%20493> > > > > > > > > > <+48%20505%20628%20493>> > > > > > > > > > > >> > > >> E: tomasz.urbas...@polidea.com < > > > > > > > tomasz.urbasz...@polidea.com > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> Unique Tech > > > > > > > > > > >> > > >> Check out our projects! < > > > > > > https://www.polidea.com/our-work> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > -- > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > Jarek Potiuk > > > > > > > > > > >> > > > Polidea <https://www.polidea.com/> | Principal > > > > Software > > > > > > > > > Engineer > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > M: +48 660 796 129 <+48%20660%20796%20129> > > > > > > > <+48%20660%20796%20129> <+48660796129 <+48%20660%20796%20129> > > > > > > > > > <+48%20660%20796%20129>> > > > > > > > > > > >> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Michał Słowikowski Polidea <https://www.polidea.com/> | Test Engineer E: michal.slowikow...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
