As I mentioned in an earlier email, we rsync periodically, and override our internal DNS to redirect 'repo1.maven.org'. All internal developers automatically use the local copy.
We don't crawl the repository. You may know people using Nexus for over a year that can't live without it. We've been using Apache web server mod_dav for three+ years with LDAP-based authentication and authorization - it has worked perfectly. As for all of your points about not having control of developers - a repository manager won't give me that either. There is no amount technology that can be applied to completely prevent any of those issues or countless others. I prefer to allow open access, facilitate developers, educate them and then review multiple times. The review points are where we begin to lock down and restrict access and the builds go through quality assurance, which can be automated. Locking down, filtering or controlling access up front prematurely limits innovation. I prefer to allow people to access what they want, as they want and as they move to later points in the process, then we tighten control as their builds move closer to manufacturing and work through dependency approval, license compliance, etc. -Nathan -----Original Message----- From: Jason van Zyl [mailto:[EMAIL PROTECTED] Sent: Monday, September 29, 2008 9:59 AM To: Maven Users List Subject: Re: Are we blocked by central Maven repo? On 29-Sep-08, at 10:21 AM, Beyer,Nathan wrote: > What would you suggest then? Anything that requires customized maven > installs or modifying 'settings.xml' post install is not feasible in > our environment - development is too distributed. So how do you used your rsync'd repository? How do you get all your developers to use your in-house repository which is a copy of central? Using repository managers makes distributed development an order of magnitude easier. > > > In the long-run I believe the rsync approach does reduce bandwith, > but more importantly, the concurrent access to the central repo via > HTTP is close to nil. Believe, as the one who looks at the logs and watch people crawl the repository this is not the case. In the long run you will see that you use less then 3% of what's in central so there is no point in pulling the bulk of the content. > > > Additionally, as I mentioned, the repository managers are NOT stable > and require too much configuration and setup. Hardly. We have people who have been using Nexus for over a year and they couldn't live without it now > These are not acceptable options. The repository managers aren't > providing any other value beyond central repo caching for us. > You have no idea what you're talking about. Sorry but you're are sadly mistaken. - Does your organization like your developers' builds crawling around every repository listed in a POM? You have no control over that without a repository manager. Tell your management that you're not controlling access to external repositories and see how much they like that. - You can create and manage access by all your developers from one location, if you have multiple repositories which most organizations have, this is a nightmare without a repository manager - IDE integration? Using a Nexus index you get complete autocompletion in the POM editor, ability to search for all plugins available, all archetypes available. - Routing around bad metadata protecting your developers from mis- formed POMs which can happen - Optimized searching for dependencies i.e. don't think around the world for your company's artifacts or only take Apache artifacts from the Apache repository - Repository federation, by proxing other repositories in Nexus you can search them all - Deployment with a simple PUT, no requirement for the WebDAV provider - Fine grained access to repositories i.e. far more powerful then access via Apache > If you're going to cut off anonymous rsync access, you might as well > just kill anonymous central repo access too, as that's the only way > you'll be able to force people into use repository managers. There is no anonymous rsync access to central, there never has been because the bandwidth charges would have made the situation unmanageable. We aren't forcing anyone to use repository managers, it boils down to a matter of cost in bandwidth. And people are using repository managers because it's just the smarter way to work with Maven. > > > I would suggest more granular rsync access, so that requests can be > more targeted. That pretty much amounts to the way a repository manager works. You're not going to get more targeted access then that. You get what you need and that's it. Run your CI system working against a repository manager will always keep your repository primed for use by your developers. > > > -Nathan > > -----Original Message----- > From: Jason van Zyl [mailto:[EMAIL PROTECTED] > Sent: Friday, September 26, 2008 3:51 PM > To: Maven Users List > Subject: Re: Are we blocked by central Maven repo? > > > On 26-Sep-08, at 9:31 PM, Beyer,Nathan wrote: > >> I disagree. 10gb or even 20gb isn't that much data, and rsync isn't >> pulling that same amount down every time it runs. We're doing it and >> it's working quite well. It's much more stable and reliable than any >> other current mirroring practices. The internal DNS modification >> makes user setup easy, since there isn't any. The use of mirror >> settings per device is a non-starter for large, disparate >> organizations. All of the various caching servers just aren't stable >> enough yet, in my opinion. >> >> It is possible to get blocked by the central repo - we were >> contacted about our significant usage and told we were on the verge >> of being blacklisted, which is what lead us to rsync the mirror. >> > > There is no way you could use less bandwidth rsyncing then using a > repository manager. If everyone rsynced and we allowed that against > central we would get destroyed. We only allow mirrors to rsync, not > users and mirrors will probably also stop providing rsync access > because the first hit is just too high now if everyone did it. > >> -Nathan >> >> -----Original Message----- >> From: Wayne Fay [mailto:[EMAIL PROTECTED] >> Sent: Friday, September 26, 2008 11:11 AM >> To: Maven Users List >> Subject: Re: Are we blocked by central Maven repo? >> >> IIRC Central is well over 10gb at this point (possibly 20gb) and a >> given organization will really only use at the most 1gb of it, so >> rsync'ing it is just a bad idea unless you are setting up an actual >> external mirror that will be available to the community. >> >> They are already using Artifactory, and I certainly hope/assume they >> are caching the results. This would limit their use of Central to one >> access per artifact (GAV) plus some hits by people not using their >> Artifactory instance. >> >> I would generally doubt they are actually blocked by Central, but >> rather this is an intermittent failure that will eventually resolve >> itself. >> >> Wayne >> >> 2008/9/26 Beyer,Nathan <[EMAIL PROTECTED]>: >>> It's possible that from the central repo's perspective, all traffic >>> from your company may seem like it's coming from one IP address >>> because of NAT. >>> >>> Using an internal mirror can help alleviate things. The most non- >>> invasive mirror would be to rsync the central repo periodically and >>> then modify internal DNS to point 'repo1.maven.org' to an internal >>> IP address. You can save a lot of bandwidth and time this way. >>> >>> -Nathan >>> >>> -----Original Message----- >>> From: 陈思淼 [mailto:[EMAIL PROTECTED] >>> Sent: Friday, September 26, 2008 10:47 AM >>> To: Maven Users List >>> Subject: Re: Are we blocked by central Maven repo? >>> >>> we didn't do that kind of thing. we have a company-level artifactory >>> repository.someone didn't follow the rule but most of us are good >>> citizen, >>> and follow the maven RULE, >>> Is maven block strategy to block IP too strict? >>> Can I do anything to Fix it Up? >>> >>> >>> >>> 2008/9/26 Wayne Fay <[EMAIL PROTECTED]> >>> >>>> It is possible to get blocked if you are acting as a bad citizen >>>> (downloading the entire Central repo using wget, for example). Have >>>> you (or someone else at your company) attempted to do this from >>>> your >>>> IP address? >>>> >>>> If not, the repo is probably just busy, or you had some random >>>> Internet connection failure. Try again. "Normal" Maven usage of the >>>> repo will not get you blocked. >>>> >>>> Wayne >>>> >>>> On Fri, Sep 26, 2008 at 7:37 AM, 陈思淼 <[EMAIL PROTECTED]> >>>> wrote: >>>>> This's log from artifactory. >>>>> >>>>> 2008-09-26 22:27:28,025 [WARN ] (RemoteRepoBase.java: >>>>> 259{10}) - >>>> repo1: >>>>> Error in getting information for 'org/apache/maven >>>>> /maven-model/2.0.4/maven-model-2.0.4.pom.sha1' >>>>> (org.apache.commons.httpclient.ConnectionPoolTimeoutException: >>>>> Timeout >>>>> waiting >>>>> for connection). >>>>> >>>>> we company only have one outlet IP address ,someone may download >>>>> Maven >>>> from >>>>> apache and didn't set the Mirror of central in the conf/ >>>>> setting.xml. so >>>> they >>>>> download the pom directly from central? Is that the reason why the >>>> central >>>>> repo block our IP address? >>>>> >>>> >>> >>> ---------------------------------------------------------------------- >>> CONFIDENTIALITY NOTICE This message and any included attachments >>> are from Cerner Corporation and are intended only for the >>> addressee. The information contained in this message is >>> confidential and may constitute inside or non-public information >>> under international, federal, or state securities laws. >>> Unauthorized forwarding, printing, copying, distribution, or use of >>> such information is strictly prohibited and may be unlawful. If you >>> are not the addressee, please promptly delete this message and >>> notify the sender of the delivery error by e-mail or you may call >>> Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) >>> (816)221-1024. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>> For additional commands, e-mail: [EMAIL PROTECTED] >>> >>> > > Thanks, > > Jason > > ---------------------------------------------------------- > Jason van Zyl > Founder, Apache Maven > jason at sonatype dot com > ---------------------------------------------------------- > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > ---------------------------------------------------------------------- > CONFIDENTIALITY NOTICE This message and any included attachments are > from Cerner Corporation and are intended only for the addressee. The > information contained in this message is confidential and may > constitute inside or non-public information under international, > federal, or state securities laws. Unauthorized forwarding, > printing, copying, distribution, or use of such information is > strictly prohibited and may be unlawful. If you are not the > addressee, please promptly delete this message and notify the sender > of the delivery error by e-mail or you may call Cerner's corporate > offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024. Thanks, Jason ---------------------------------------------------------- Jason van Zyl Founder, Apache Maven jason at sonatype dot com ---------------------------------------------------------- the course of true love never did run smooth ... -- Shakespeare --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
