Le 5 déc. 2017 06:20, "Eugene Kirpichov" <kirpic...@google.com> a écrit :
On Mon, Dec 4, 2017 at 1:45 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > Le 4 déc. 2017 21:45, "Eugene Kirpichov" <kirpic...@google.com> a écrit : > > Romain - as far as I understand, Maven *does* have a retry strategy, but > it is a poor retry strategy and there is no way to tweak it. In particular, > if it uses https://maven.apache.org/guides/mini/guide-http-settings.html , > that means it uses Apache Http Client 4.1.2 whose default retry settings > are extremely conservative > https://github.com/apache/httpcomponents-client/blob/4. > 1.x/httpclient/src/main/java/org/apache/http/impl/client/ > DefaultHttpRequestRetryHandler.java#L93 > > > Right > > > > The errors we're hitting with Maven are some of those: "connection > refused" and alike. > > > > Would a retry guarantee you it works. Should be quite easy to switch wagon > with a configurable retry handler instance through mvnwrapper or custom mvn > setup and test it but the logic looks sane to me. > Retries indeed don't give any guarantees, but they're very good at reducing error rate, and our builds could definitely use some of that. I tried to figure out how to make Maven use a configurable http wagon with a custom retry handler, but I couldn't - if you know more, could you give some pointers? I think we should definitely do that. It is not configurable but to ensure it is worth a mvn patch we can add a mvn wrapper script where we patch wagon and use that on the CI. > > If it is on asf repo we must work with infra to fix it rather than working > around the clients. If on a repo we dont control we can try to get rid of > it maybe? > It happens often enough with the maven central repo repo.maven.apache.org. At Beam's (moderate) scale of hundreds of builds per day resolving hundreds of dependencies each, even 99.99% availability with a poor retry strategy would translate to builds failing regularly. Hmm, this is an asf server and behing should be repo1 and repo2 so either there is a huge client issue or the proxy can be better configure. If you have some failure logs, do you mind pinging infra with them? There's also multiple parts that can fail - Maven Central itself, network issues between Jenkins and Maven Central, network issues on the Jenkins machine itself and so on - I doubt that ASF can fix all of this for us. I think this sort of issue *has* to be resolved on the client. Not all kind of failures, connection breakdown during the transfert yes but other kind means the server is down so retry shouldnt help. Let me know if you nees help with wagon patching/testing. Can have a few cycles end of the week to give some help. > > > > On Mon, Dec 4, 2017 at 12:15 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> 2017-12-04 18:58 GMT+01:00 Kenneth Knowles <k...@google.com>: >> > On Sat, Dec 2, 2017 at 3:06 AM, Romain Manni-Bucau < >> rmannibu...@gmail.com> >> > wrote: >> >> >> >> Need to check but if plugin dependencies were not tuned it should be >> the >> >> default with a retry of 3 IIRC. >> >> >> >> But arent you sure it was a repo/server issue any client cant solve? >> > >> > >> > Not sure what you mean here. What we need is for mvn and gradle to >> succeed >> > even though there are frequent transient failures of the repo. Anything >> that >> > makes this happen is success. >> >> What I meant is you will likely never get it with whatever build >> system until you go 100% local which wouldn't make the build >> reproducible. >> >> Concretely you can do >> >> 1. >> >> for (int i = 0; i < 3; i++) { >> if (download()) break; >> } >> >> >> 2. >> >> for (int i = 0; i < 3; i++) { >> if (download()) break; >> exponentialBackoff(); >> } >> >> >> Or other wait strategy with a timeout. >> but you can't have: >> >> while (!download()) { >> waitUntilItWorks(); >> } >> >> >> So concretely if your down time > reasonnable time you will see the >> failure anyway. >> >> Maven default has a retry (you see it sometimes in the logs) so should >> tolerante a restart or small downtime. Ivy (gradle) uses httpclient as >> well so same kind of config is available but with the same limitation >> by design. >> >> Having a local cache is the best way to avoid these issues but it >> requires at least one green build for dependencies. >> >> What I often saw was to do a pre-build step just resolving >> dependencies+plugins and use a .m2/repository caching. >> >> > >> > Kenn >> > >> >> >> >> >> >> Le 1 déc. 2017 23:27, "Kenneth Knowles" <k...@google.com> a écrit : >> >>> >> >>> How do you instruct maven or gradle to use that all the time? >> >>> >> >>> On Fri, Dec 1, 2017 at 1:58 PM, Romain Manni-Bucau >> >>> <rmannibu...@gmail.com> wrote: >> >>>> >> >>>> If you use wagon http - and not lightweigh - it should have retries >> by >> >>>> default. >> >>>> >> >>>> Le 1 déc. 2017 22:31, "Valentyn Tymofieiev" <valen...@google.com> a >> >>>> écrit : >> >>>>> >> >>>>> Has this ever been brought up in Maven dev community? Perhaps they >> have >> >>>>> some suggestions. It sounds like a reasonable feature request. >> >>>>> >> >>>>> On Fri, Dec 1, 2017 at 1:18 PM, Kenneth Knowles <k...@google.com> >> wrote: >> >>>>>> >> >>>>>> I've repeatedly searched around for a way to just add proper retry >> to >> >>>>>> maven or gradle, and haven't found anything :-( >> >>>>>> >> >>>>>> I had thought that we altered our builds in such a way that the .m2 >> >>>>>> directory was permitted to survive across builds. True that it >> isn't >> >>>>>> hermetic, precisely, but it is pretty safe to treat as a cache of >> immutable >> >>>>>> data, which is no more dangerous than having a caching incremental >> build >> >>>>>> system. >> >>>>>> >> >>>>>> Kenn >> >>>>>> >> >>>>>> On Wed, Nov 29, 2017 at 3:39 PM, Eugene Kirpichov >> >>>>>> <kirpic...@google.com> wrote: >> >>>>>>> >> >>>>>>> Our builds often hit transient Maven network issues, e.g. this one >> >>>>>>> >> >>>>>>> https://builds.apache.org/job/beam_PostCommit_Java_ >> MavenInstall/5331/consoleFull >> >>>>>>> >> >>>>>>> 2017-11-29T02:18:02.936 [ERROR] Failed to execute goal on project >> >>>>>>> beam-sdks-java-io-hadoop-jdk1.8-tests: Could not resolve >> dependencies for >> >>>>>>> project >> >>>>>>> org.apache.beam:beam-sdks-java-io-hadoop-jdk1.8-tests: >> jar:2.3.0-SNAPSHOT: >> >>>>>>> Could not transfer artifact org.apache.derby:derby:jar:10.10.2.0 >> from/to >> >>>>>>> central (https://repo.maven.apache.org/maven2): GET request of: >> >>>>>>> org/apache/derby/derby/10.10.2.0/derby-10.10.2.0.jar from >> central failed: >> >>>>>>> Connection reset -> [Help 1] >> >>>>>>> >> >>>>>>> It'd be good to increase reliability of our builds. >> >>>>>>> repo.maven.apache.org seems quite unreliable. >> >>>>>>> >> >>>>>>> I tried finding a way to configure Maven to retry such network >> errors >> >>>>>>> and it appears to be impossible [will be happy if someone proves >> me wrong]. >> >>>>>>> >> >>>>>>> Would this issue be resolved if we used multiple mirrors? >> >>>>>>> https://maven.apache.org/guides/mini/guide-mirror-settings.html >> Any other >> >>>>>>> suggestions? >> >>>>>> >> >>>>>> >> >>>>> >> >>> >> > >> > >
