Re: [DISCUSS] Reducing build times

Hi, we have not decided how to proceed on the issue yet. I'm currently in the process of setting up some metrics for understanding the impact on the end-to-end build times of Travis vs Azure Pipelines. In my current PoC [1], we are also using self-hosted pipeline agents. Feel free to reach out to

Re: [DISCUSS] Reducing build times

Hi, How is the CI migration going? I notice that Azure pipelines support adding self-hosted pipeline agent.[1] How do you think that OpenLab donate some ARM resources to Flink community as a agent pool? In this way, X86 and ARM can be handled by the same CI config easily. Then we don't need do

Re: [DISCUSS] Reducing build times

I do have a working Azure setup, yes. E2E tests are not included in the 3.5hrs. Yesterday, I became aware of a major blocker with Azure pipelines: Apache Infra does not allow it to be integrated with Apache GitHub repositories, because it requires write access (for a simple usability feature)

Re: [DISCUSS] Reducing build times

I assume you already have a working (and verified) azure setup? Once we're running things on azure on the apache repo people will invariably use that as a source of truth because fancy check marks will yet again appear on commits. Hence I'm wary of running experiments here; I would prefer if

Re: [DISCUSS] Reducing build times

e2e tests on Travis add another 4-5 hours, but we never optimized these to make use of the cached Flink artifact. On 04/09/2019 13:26, Till Rohrmann wrote: How long do we need to run all e2e tests? They are not included in the 3,5 hours I assume. Cheers, Till On Wed, Sep 4, 2019 at 11:59 AM

Re: [DISCUSS] Reducing build times

How long do we need to run all e2e tests? They are not included in the 3,5 hours I assume. Cheers, Till On Wed, Sep 4, 2019 at 11:59 AM Robert Metzger wrote: > Yes, we can ensure the same (or better) experience for contributors. > > On the powerful machines, builds finish in 1.5 hours (without

Re: [DISCUSS] Reducing build times

Yes, we can ensure the same (or better) experience for contributors. On the powerful machines, builds finish in 1.5 hours (without any caching enabled). Azure Pipelines offers 10 concurrent builds with a timeout of 6 hours for a build for open source projects. Flink needs 3.5 hours on that infra

Re: [DISCUSS] Reducing build times

Will using more powerful for the project make it more difficult to ensure that contributor builds are still running in a reasonable time? As an example of this happening on Travis, contributors currently cannot run all e2e tests since they timeout, but on apache we have a larger timeout. On

Re: [DISCUSS] Reducing build times

+1 for Azure Pipelines, had very good experiences in the past with it and the open source and payment models are much better. The upcoming Github CI/CD seems also like a promising alternative, but from the first looks, it seems like the small brother of Azure Pipeline. So, any effort going into

Re: [DISCUSS] Reducing build times

Hi all, I wanted to give a short update on this: - Arvid, Aljoscha and I have started working on a Gradle PoC, currently working on making all modules compile and test with Gradle. We've also identified some problematic areas (shading being the most obvious one) which we will analyse as part of

Re: [DISCUSS] Reducing build times

Hi all, I have summarized all arguments mentioned so far + some additional research into a Wiki page here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=125309279 I'm happy to hear further comments on my summary! I'm pretty sure we can find more pro's and con's for the

Re: [DISCUSS] Reducing build times

I did a quick test: a normal "mvn clean install -DskipTests -Drat.skip=true -Dmaven.javadoc.skip=true -Punsafe-mapr-repo” on my machine takes about 14 minutes. After removing all mentions of maven-shade-plugin the build time goes down to roughly 11.5 minutes. (Obviously the resulting Flink

Re: [DISCUSS] Reducing build times

Hi all, I wanted to understand the impact of the hardware we are using for running our tests. Each travis worker has 2 virtual cores, and 7.5 gb memory [1]. They are using Google Cloud Compute Engine *n1-standard-2* instances. Running a full "mvn clean verify" takes *03:32 h* on such a machine

Re: [DISCUSS] Reducing build times

@Aljoscha Shading takes a few minutes for a full build; you can see this quite easily by looking at the compile step in the misc profile ; all modules that longer than a fraction of a section are usually caused by shading lots of classes.

Re: [DISCUSS] Reducing build times

Speaking of flink-shaded, do we have any idea what the impact of shading is on the build time? We could get rid of shading completely in the Flink main repository by moving everything that we shade to flink-shaded. Aljoscha > On 16. Aug 2019, at 14:58, Bowen Li wrote: > > +1 to Till's points

Re: [DISCUSS] Reducing build times

+1 to Till's points on #2 and #5, especially the potential non-disruptive, gradual migration approach if we decide to go that route. To add on, I want to point it out that we can actually start with flink-shaded project [1] which is a perfect candidate for PoC. It's of much smaller size, totally

Re: [DISCUSS] Reducing build times

For the sake of keeping the discussion focused and not cluttering the discussion thread I would suggest to split the detailed reporting for reusing JVMs to a separate thread and cross linking it from here. Cheers, Till On Fri, Aug 16, 2019 at 1:36 PM Chesnay Schepler wrote: > Update: > >

Re: [DISCUSS] Reducing build times

Update: TL;DR: table-planner is a good candidate for enabling fork reuse right away, while flink-tests has the potential for huge savings, but we have to figure out some issues first. Build link: https://travis-ci.org/zentol/flink/builds/572659220 4/8 profiles failed. No speedup in

Re: [DISCUSS] Reducing build times

There appears to be a general agreement that 1) should be looked into; I've setup a branch with fork reuse being enabled for all tests; will report back the results. On 15/08/2019 09:38, Chesnay Schepler wrote: Hello everyone, improving our build times is a hot topic at the moment so let's

Re: [DISCUSS] Reducing build times

6. CI service I'm not very familar with tarvis, but according to its offical doc[1][2]. Is it possible to run jobs in parallel? AFAIK, many CI system supports this kind of feature. [1]: https://docs.travis-ci.com/user/speeding-up-the-build/#parallelizing-your-builds-across-virtual-machines

Re: [DISCUSS] Reducing build times

Thank you for starting the discussion as well! +1 to 1. it seems to be a quite low-hanging fruit that we should try to employ as much as possible. -0 to 2. the build setup is already very complicated. Adding new functionality that I would expect to come out of the box of a modern build tool

Re: [DISCUSS] Reducing build times

Thanks Chesnay for starting this discussion. +1 for #1, it might be the easiest way to get a significant speedup. If the only reason is for isolation. I think we can fix the static fields or global state used in Flink if possible. +1 for #2, and thanks Aleksey for the prototype. I think it's a

Re: [DISCUSS] Reducing build times

Hi all! Thanks for starting this discussion. I'd like to also add my 2 cents: +1 for #2, differential build scripts. I've worked on the approach. And with it, I think it's possible to reduce total build time with relatively low effort, without enforcing any new build tool and low maintenance

Re: [DISCUSS] Reducing build times

Thanks for starting this discussion Chesnay. I think it has become obvious to the Flink community that with the existing build setup we cannot really deliver fast build times which are essential for fast iteration cycles and high developer productivity. The reasons for this situation are manifold