I am in favour of decommissioning Travis. Moreover, I wanted to use this thread to raise another issue with Travis that I have discovered recently; many of the builds running in my private Travis account are timing out in the compilation stage (i.e., compilation takes more than 50 minutes). This means that I am not able to reliably run a full build on a CI server without creating a pull request. If other developers also experience this issue, it would speak for putting more effort into making Azure Pipelines the project-wide default.
Best, Gary On Thu, Mar 26, 2020 at 12:26 PM Yu Li <car...@gmail.com> wrote: > Thanks for the clarification Robert. > > Since the first step plan is to replace the travis PR runs, I checked all > PR builds from 2020-01-01 (PR#10735-11526) [1], and below is the result: > > * Travis FAILURE: 298 > * Travis SUCCESS: 649 (68.5%) > * Azure FAILURE: 420 > * Azure SUCCESS: 571 (57.6%) > > Since the patch for each run is equivalent for Travis and Azure, there > seems to be slightly higher failure rate (~10%) when running in Azure. > > However, with the just-merged fix for uploading logs (FLINK-16480), I > believe the success rate of Azure could compete with Travis now (uploading > files contribute to 20% of the failures according to the report [2]). > > So I'm +1 to disable travis runs according to the numbers. > > Best Regards, > Yu > > [1] > https://github.com/apache/flink/pulls?q=is%3Apr+created%3A%3E%3D2020-01-01 > [2] > > https://dev.azure.com/rmetzger/Flink/_pipeline/analytics/stageawareoutcome?definitionId=4 > > On Thu, 26 Mar 2020 at 03:28, Robert Metzger <rmetz...@apache.org> wrote: > > > Thank you for your responses. > > > > @Yu Li: In the current master, the log upload always fails, if the e2e > job > > failed. I just merged a PR that fixes this issue [1]. The problem was not > > really the network stability, rather a problem with the interaction of > the > > jobs in the pipeline (the e2e job did not set the right variables for the > > log upload) > > Secondly, you are looking at the report of the "flink-ci.flink" pipeline, > > where pull requests are build. Naturally, pull request builds fail all > the > > time, because the PRs are not yet perfect. > > > > "flink-ci.flink-master" is the right pipeline to look at: > > > > > https://dev.azure.com/rmetzger/Flink/_pipeline/analytics/stageawareoutcome?definitionId=8&contextType=build > > We have a fairly high number of failures there, because we currently have > > some issues downloading the maven artifacts [2]. I'm working already with > > Chesnay on merging a fix for that. > > > > > > [1] > > > > > https://github.com/apache/flink/commit/1c86b8b9dd05615a3b2600984db738a9bf388259 > > [2]https://issues.apache.org/jira/browse/FLINK-16720 > > > > > > > > On Wed, Mar 25, 2020 at 4:48 PM Chesnay Schepler <ches...@apache.org> > > wrote: > > > > > The easiest way to disable travis for pushes is to remove all builds > > > from the .travis.yml with a push/pr condition. > > > > > > On 25/03/2020 15:03, Robert Metzger wrote: > > > > Thank you for the feedback so far. > > > > > > > > Responses to the items Chesnay raised: > > > > > > > > - by virtue of maintaining the past 2 releases we will have to > maintain > > > any > > > >> Travis infrastructure as long as 1.10 is supported, i.e., until 1.12 > > > >> > > > > Okay. I wasn't sure about the exact policy there. > > > > > > > > > > > >> - the azure setup doesn't appear to be equivalent yet since the java > > e2e > > > >> profile isn't setting the hadoop switch (-Pe2e-hadoop), as a result > of > > > >> which SQLClientKafkaITCase isn't run > > > >> > > > > I filed a ticket to address this: > > > > https://issues.apache.org/jira/browse/FLINK-16778 > > > > > > > > > > > >> - the nightly scripts still seems to be using a maven version other > > than > > > >> 3.2.5; from today on master: > > > >> 2020-03-25T05:31:52.7412964Z [INFO] --------< > > > >> org.apache.flink:flink-end-to-end-tests-common-kafka >-------- > > > >> 2020-03-25T05:31:52.7413854Z [INFO] Building > > > >> flink-end-to-end-tests-common-kafka 1.11-SNAPSHOT [39/46] > > > >> 2020-03-25T05:31:52.7414689Z [INFO] > --------------------------------[ > > > jar > > > >> ]--------------------------------- > > > >> 2020-03-25T05:31:52.7518360Z [INFO] > > > >> 2020-03-25T05:31:52.7519770Z [INFO] --- > > > maven-checkstyle-plugin:2.17:check > > > >> (validate) @ flink-end-to-end-tests-common-kafka --- > > > >> > > > > I'm planning to address this as part of > > > > https://issues.apache.org/jira/browse/FLINK-16411, where I work on > > > > centralizing all mvn invocations. > > > > > > > > > > > >> - there is no real benefit in retiring the travis support in CiBot; > > the > > > >> important part is whether Travis is run or not for pull requests. > > > >> From what I can tell though azure seems to be working fine for pull > > > >> requests, so +1 to at least disable the travis PR runs. > > > > > > > > So we disable Travis for https://github.com/flink-ci/flink ? I will > do > > > it > > > > once there are no new concerns and above tickets are resolved. > > > > > > > > What about disabling travis for master pushes? (e.g. removing the > > > > .travis.yml file from master)? > > > > > > > > > > > > @Dian: > > > > Thanks a lot for your feedback. > > > > > > > > - The report of Azure is still not viewable[1] (I noticed that Hequn > > has > > > >> also reported this issue in another thread). This is very useful > > > >> information. > > > > > > > > You are referring to the emails send to builds@f.a.o right? > > > > I have reported this both as a bug [1] and a feature request [2] to > > > Azure. > > > > But I don't believe they will resolve this issue anytime soon. > > > > Azure has an notifications API that we could use to build a service > > that > > > > sends emails to that list, but I feel that this is really a waste of > > > time. > > > > The URL in the link even contains the ID of the build. We would just > > need > > > > to extract this ID and generate the appropriate URL. I will try to > > > directly > > > > reach the product management of AZP, maybe I can get some attention > > from > > > > there. > > > > > > > > > > > > > > > > [1] > > > > > > > > > > https://developercommunity.visualstudio.com/content/problem/957778/third-parties-are-unable-to-access-notification-li.html?childToView=960403#comment-960403 > > > > [2] > > > > > > > > > > https://developercommunity.visualstudio.com/content/idea/960472/third-parties-are-unable-to-access-notification-li-1.html > > > > > > > > > > > > > > > > On Wed, Mar 25, 2020 at 10:34 AM Chesnay Schepler < > ches...@apache.org> > > > > wrote: > > > > > > > >> It was left out since it adds significant additional complexity and > > the > > > >> value is dubious at best for PRs that aren't merged shortly after > the > > > >> build has finished. > > > >> > > > >> On 25/03/2020 10:28, Dian Fu wrote: > > > >>> Thanks for the information. I'm sorry that I'm not aware of this > > before > > > >> and I have checked the build log of travis and confirmed that this > is > > > true. > > > >>> @Chesnay Are there any specific reasons for this and is it possible > > to > > > >> add this back for Azure Pipelines? > > > >>> Thanks, > > > >>> Dian > > > >>> > > > >>>> 在 2020年3月25日，下午4:43，Chesnay Schepler <ches...@apache.org> 写道： > > > >>>> > > > >>>> @Dian we haven't been rebasing PR's against master for months, > ever > > > >> since we switched to CiBot. > > > >>>> On 25/03/2020 09:29, Dian Fu wrote: > > > >>>>> Hi Robert, > > > >>>>> > > > >>>>> Thanks a lot for your great work! > > > >>>>> > > > >>>>> Overall I'm +1 to switch to Azure as the primary CI tool if it's > > > >> stable enough as I think there is no need to run both the travis and > > > Azure > > > >> for one single PR. > > > >>>>> However, there are still some improvements need to do and it > would > > be > > > >> great if these issues could be addressed before fully switch to > Azure: > > > >>>>> - The report of Azure is still not viewable[1] (I noticed that > > Hequn > > > >> has also reported this issue in another thread). This is very useful > > > >> information. > > > >>>>> - For PR test of Azure pipeline, it seems that it will not rebase > > the > > > >> master code before running the tests. > > > >>>>> Thanks, > > > >>>>> Dian > > > >>>>> > > > >>>>> [1] > > > >> > > > > > > https://dev.azure.com/rmetzger/web/build.aspx?pcguid=03e2a4fd-f647-46c5-a324-527d2c2984ce&builduri=vstfs%3a%2f%2f%2fBuild%2fBuild%2f6593&tracking_data=eyJTb3VyY2UiOiJFbWFpbCIsIlR5cGUiOiJOb3RpZmljYXRpb24iLCJTSUQiOiIzMzk0MzciLCJTVHlwZSI6IkdSUCIsIlJlY2lwIjoxLCJfeGNpIjp7Ik5JRCI6NDAyODQ3NzksIk1SZWNpcCI6Im0wPTEgIiwiQWN0IjoiMTNjNDc3YWMtZTBjYS00MjJkLTkxOTItZWI0NzFkZmUzMWY0In0sIkVsZW1lbnQiOiJoZXJvL2N0YSJ9 > > > >> < > > > >> > > > > > > https://dev.azure.com/rmetzger/web/build.aspx?pcguid=03e2a4fd-f647-46c5-a324-527d2c2984ce&builduri=vstfs%3a%2f%2f%2fBuild%2fBuild%2f6593&tracking_data=eyJTb3VyY2UiOiJFbWFpbCIsIlR5cGUiOiJOb3RpZmljYXRpb24iLCJTSUQiOiIzMzk0MzciLCJTVHlwZSI6IkdSUCIsIlJlY2lwIjoxLCJfeGNpIjp7Ik5JRCI6NDAyODQ3NzksIk1SZWNpcCI6Im0wPTEgIiwiQWN0IjoiMTNjNDc3YWMtZTBjYS00MjJkLTkxOTItZWI0NzFkZmUzMWY0In0sIkVsZW1lbnQiOiJoZXJvL2N0YSJ9 > > > > > > > >> < > > > >> > > > > > > https://dev.azure.com/rmetzger/web/build.aspx?pcguid=03e2a4fd-f647-46c5-a324-527d2c2984ce&builduri=vstfs:///Build/Build/6593&tracking_data=eyJTb3VyY2UiOiJFbWFpbCIsIlR5cGUiOiJOb3RpZmljYXRpb24iLCJTSUQiOiIzMzk0MzciLCJTVHlwZSI6IkdSUCIsIlJlY2lwIjoxLCJfeGNpIjp7Ik5JRCI6NDAyODQ3NzksIk1SZWNpcCI6Im0wPTEgIiwiQWN0IjoiMTNjNDc3YWMtZTBjYS00MjJkLTkxOTItZWI0NzFkZmUzMWY0In0sIkVsZW1lbnQiOiJoZXJvL2N0YSJ9 > > > >> < > > > >> > > > > > > https://dev.azure.com/rmetzger/web/build.aspx?pcguid=03e2a4fd-f647-46c5-a324-527d2c2984ce&builduri=vstfs:///Build/Build/6593&tracking_data=eyJTb3VyY2UiOiJFbWFpbCIsIlR5cGUiOiJOb3RpZmljYXRpb24iLCJTSUQiOiIzMzk0MzciLCJTVHlwZSI6IkdSUCIsIlJlY2lwIjoxLCJfeGNpIjp7Ik5JRCI6NDAyODQ3NzksIk1SZWNpcCI6Im0wPTEgIiwiQWN0IjoiMTNjNDc3YWMtZTBjYS00MjJkLTkxOTItZWI0NzFkZmUzMWY0In0sIkVsZW1lbnQiOiJoZXJvL2N0YSJ9 > > > >>>>>> 在 2020年3月25日，下午3:33，Chesnay Schepler <ches...@apache.org> 写道： > > > >>>>>> > > > >>>>>> Some thoughts: > > > >>>>>> - by virtue of maintaining the past 2 releases we will have to > > > >> maintain any Travis infrastructure as long as 1.10 is supported, > i.e., > > > >> until 1.12 > > > >>>>>> - the azure setup doesn't appear to be equivalent yet since the > > java > > > >> e2e profile isn't setting the hadoop switch (-Pe2e-hadoop), as a > > result > > > of > > > >> which SQLClientKafkaITCase isn't run > > > >>>>>> - the nightly scripts still seems to be using a maven version > > other > > > >> than 3.2.5; from today on master: > > > >>>>>> 2020-03-25T05:31:52.7412964Z [INFO] --------< > > > >> org.apache.flink:flink-end-to-end-tests-common-kafka >-------- > > > >>>>>> 2020-03-25T05:31:52.7413854Z [INFO] Building > > > >> flink-end-to-end-tests-common-kafka 1.11-SNAPSHOT [39/46] > > > >>>>>> 2020-03-25T05:31:52.7414689Z [INFO] > > > --------------------------------[ > > > >> jar ]--------------------------------- > > > >>>>>> 2020-03-25T05:31:52.7518360Z [INFO] > > > >>>>>> 2020-03-25T05:31:52.7519770Z [INFO] --- > > > >> maven-checkstyle-plugin:2.17:check (validate) @ > > > >> flink-end-to-end-tests-common-kafka --- > > > >>>>>> - there is no real benefit in retiring the travis support in > > CiBot; > > > >> the important part is whether Travis is run or not for pull > requests. > > > >>>>>> From what I can tell though azure seems to be working fine for > > > pull > > > >> requests, so +1 to at least disable the travis PR runs. > > > >>>>>> On 23/03/2020 14:48, Robert Metzger wrote: > > > >>>>>>> Hey devs, > > > >>>>>>> > > > >>>>>>> I would like to discuss whether it makes sense to fully switch > to > > > >> Azure > > > >>>>>>> Pipelines and phase out our Travis integration. > > > >>>>>>> More information on our Azure integration can be found here: > > > >>>>>>> > > > >> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/2020/03/22/Migrating+Flink%27s+CI+Infrastructure+from+Travis+CI+to+Azure+Pipelines > > > >>>>>>> Travis will stay for the release-1.10 and older branches, as I > > have > > > >> set up > > > >>>>>>> Azure only for the master branch. > > > >>>>>>> > > > >>>>>>> Proposal: > > > >>>>>>> - We keep the flinkbot infrastructure supporting both Travis > and > > > >> Azure > > > >>>>>>> around, while we are still receive pull requests and pushes for > > the > > > >>>>>>> "master" and "release-1.10" branches. > > > >>>>>>> - We remove the travis-specific files from "master", so that > > builds > > > >> are not > > > >>>>>>> triggered anymore > > > >>>>>>> - once we receive no more builds at Travis (because 1.11 has > been > > > >>>>>>> released), we remove the remaining travis-related > infrastructure > > > >>>>>>> > > > >>>>>>> What do you think? > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> Best, > > > >>>>>>> Robert > > > >> > > > >> > > > > > > > > >
