On Mon, Dec 4, 2017 at 1:45 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote:
> > > Le 4 déc. 2017 21:45, "Eugene Kirpichov" <kirpic...@google.com> a écrit : > > Romain - as far as I understand, Maven *does* have a retry strategy, but > it is a poor retry strategy and there is no way to tweak it. In particular, > if it uses https://maven.apache.org/guides/mini/guide-http-settings.html , > that means it uses Apache Http Client 4.1.2 whose default retry settings > are extremely conservative > > https://github.com/apache/httpcomponents-client/blob/4.1.x/httpclient/src/main/java/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.java#L93 > > > Right > > > > The errors we're hitting with Maven are some of those: "connection > refused" and alike. > > > > Would a retry guarantee you it works. Should be quite easy to switch wagon > with a configurable retry handler instance through mvnwrapper or custom mvn > setup and test it but the logic looks sane to me. > Retries indeed don't give any guarantees, but they're very good at reducing error rate, and our builds could definitely use some of that. I tried to figure out how to make Maven use a configurable http wagon with a custom retry handler, but I couldn't - if you know more, could you give some pointers? I think we should definitely do that. > > If it is on asf repo we must work with infra to fix it rather than working > around the clients. If on a repo we dont control we can try to get rid of > it maybe? > It happens often enough with the maven central repo repo.maven.apache.org. At Beam's (moderate) scale of hundreds of builds per day resolving hundreds of dependencies each, even 99.99% availability with a poor retry strategy would translate to builds failing regularly. There's also multiple parts that can fail - Maven Central itself, network issues between Jenkins and Maven Central, network issues on the Jenkins machine itself and so on - I doubt that ASF can fix all of this for us. I think this sort of issue *has* to be resolved on the client. > > > > On Mon, Dec 4, 2017 at 12:15 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> 2017-12-04 18:58 GMT+01:00 Kenneth Knowles <k...@google.com>: >> > On Sat, Dec 2, 2017 at 3:06 AM, Romain Manni-Bucau < >> rmannibu...@gmail.com> >> > wrote: >> >> >> >> Need to check but if plugin dependencies were not tuned it should be >> the >> >> default with a retry of 3 IIRC. >> >> >> >> But arent you sure it was a repo/server issue any client cant solve? >> > >> > >> > Not sure what you mean here. What we need is for mvn and gradle to >> succeed >> > even though there are frequent transient failures of the repo. Anything >> that >> > makes this happen is success. >> >> What I meant is you will likely never get it with whatever build >> system until you go 100% local which wouldn't make the build >> reproducible. >> >> Concretely you can do >> >> 1. >> >> for (int i = 0; i < 3; i++) { >> if (download()) break; >> } >> >> >> 2. >> >> for (int i = 0; i < 3; i++) { >> if (download()) break; >> exponentialBackoff(); >> } >> >> >> Or other wait strategy with a timeout. >> but you can't have: >> >> while (!download()) { >> waitUntilItWorks(); >> } >> >> >> So concretely if your down time > reasonnable time you will see the >> failure anyway. >> >> Maven default has a retry (you see it sometimes in the logs) so should >> tolerante a restart or small downtime. Ivy (gradle) uses httpclient as >> well so same kind of config is available but with the same limitation >> by design. >> >> Having a local cache is the best way to avoid these issues but it >> requires at least one green build for dependencies. >> >> What I often saw was to do a pre-build step just resolving >> dependencies+plugins and use a .m2/repository caching. >> >> > >> > Kenn >> > >> >> >> >> >> >> Le 1 déc. 2017 23:27, "Kenneth Knowles" <k...@google.com> a écrit : >> >>> >> >>> How do you instruct maven or gradle to use that all the time? >> >>> >> >>> On Fri, Dec 1, 2017 at 1:58 PM, Romain Manni-Bucau >> >>> <rmannibu...@gmail.com> wrote: >> >>>> >> >>>> If you use wagon http - and not lightweigh - it should have retries >> by >> >>>> default. >> >>>> >> >>>> Le 1 déc. 2017 22:31, "Valentyn Tymofieiev" <valen...@google.com> a >> >>>> écrit : >> >>>>> >> >>>>> Has this ever been brought up in Maven dev community? Perhaps they >> have >> >>>>> some suggestions. It sounds like a reasonable feature request. >> >>>>> >> >>>>> On Fri, Dec 1, 2017 at 1:18 PM, Kenneth Knowles <k...@google.com> >> wrote: >> >>>>>> >> >>>>>> I've repeatedly searched around for a way to just add proper retry >> to >> >>>>>> maven or gradle, and haven't found anything :-( >> >>>>>> >> >>>>>> I had thought that we altered our builds in such a way that the .m2 >> >>>>>> directory was permitted to survive across builds. True that it >> isn't >> >>>>>> hermetic, precisely, but it is pretty safe to treat as a cache of >> immutable >> >>>>>> data, which is no more dangerous than having a caching incremental >> build >> >>>>>> system. >> >>>>>> >> >>>>>> Kenn >> >>>>>> >> >>>>>> On Wed, Nov 29, 2017 at 3:39 PM, Eugene Kirpichov >> >>>>>> <kirpic...@google.com> wrote: >> >>>>>>> >> >>>>>>> Our builds often hit transient Maven network issues, e.g. this one >> >>>>>>> >> >>>>>>> >> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/5331/consoleFull >> >>>>>>> >> >>>>>>> 2017-11-29T02:18:02.936 [ERROR] Failed to execute goal on project >> >>>>>>> beam-sdks-java-io-hadoop-jdk1.8-tests: Could not resolve >> dependencies for >> >>>>>>> project >> >>>>>>> >> org.apache.beam:beam-sdks-java-io-hadoop-jdk1.8-tests:jar:2.3.0-SNAPSHOT: >> >>>>>>> Could not transfer artifact org.apache.derby:derby:jar:10.10.2.0 >> from/to >> >>>>>>> central (https://repo.maven.apache.org/maven2): GET request of: >> >>>>>>> org/apache/derby/derby/10.10.2.0/derby-10.10.2.0.jar from >> central failed: >> >>>>>>> Connection reset -> [Help 1] >> >>>>>>> >> >>>>>>> It'd be good to increase reliability of our builds. >> >>>>>>> repo.maven.apache.org seems quite unreliable. >> >>>>>>> >> >>>>>>> I tried finding a way to configure Maven to retry such network >> errors >> >>>>>>> and it appears to be impossible [will be happy if someone proves >> me wrong]. >> >>>>>>> >> >>>>>>> Would this issue be resolved if we used multiple mirrors? >> >>>>>>> https://maven.apache.org/guides/mini/guide-mirror-settings.html >> Any other >> >>>>>>> suggestions? >> >>>>>> >> >>>>>> >> >>>>> >> >>> >> > >> > >
