Hi, Although it is obvious Brian that you did an excellent debugging-job with identifying the root of the DOS, the final 'Note' at the end of your initial mail looked more like a careless call for a flame-war:
On Fri 28 Nov 2008 14:50:17 Brian E. Fox wrote: > ... > Note that other tools like the Nexus Maven Repository Manager > (http://nexus.sonatype.org), M2e and Q4e use the Nexus Indexer API and > are immune to this problem and are not blocked from downloading the > index. A new version of Nexus and the Nexus Indexer API will be > published soon along with M2e that will leverage incremental indexes to > significantly reduce the download requirements and allow near real time > index updates. > > Brian Fox > Apache Maven PMC * Why did you choose to compare Artifactory with other projects? * Why did you have to supply the links to Nexus at this very moment? * Why did you inform about the new version of Nexus coming-soon? I hope that you do not hold any bad feeling for the people around Artifactory just because they provided a decent maven-proxy when there was none available? I applaud both efforts, Nexus and Artifactory, (we are actually using both!) but personally i don't like anyone using its "maven-generated" status to enforce any one project! Such mind-sets are not fitted for the free-software programmers. With Respect Kostis Anagnostopoulos On Fri, Nov 28, 2008 at 7:28 PM, Brian E. Fox <[EMAIL PROTECTED]> wrote: > Yoav, > I sent the notice to everyone who might be affected by this as soon as we > understood this issue and confirmed it by selectively blocking the suspected > incidents (which was before 8am local time when I woke up and saw that > Contegix had confirmed it). Since it was clear that users could directly > affect this themselves by adjusting the cron time, I thought it prudent to > make that request ASAP. Waiting longer serves no one in this case. Since the > bandwidth costs real money, and was causing users real problems, not blocking > wasn't an option once we understood how to fix it. Blocking and not notifying > everyone involved immediately wouldn't be in the user's best interests either. > > If my motivation was other than to protect Central, we could have more easily > just blocked the high offending IPs and gone back to enjoying the holiday > quietly. Instead, we spent a lot of time really understanding the problem to > make sure we didn't needlessly take out all Artifactory users in a way that > actually broke builds. > > I have no doubt that the fix is easy, but it will take time to get people > running fixed versions. Adjusting cron can be done now and will at least > minimize the impact in the meantime so we can unblock the index. > > --Brian > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > Sent: Friday, November 28, 2008 12:01 PM > To: Maven Users List > Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; Maven Project Management Committee > List > Subject: Re: [Artifactory-users] Artifactory 1.3 has been DOS'ing the Central > repository > > Yoav Landman wrote: >> Brian, >> >> I think it is only a reasonable expectation from you, to contact our team >> immediately as soon as you found out about this bug, so that we can let our >> users know how to work around it and if they are affected at all. However, >> since you are the Nexus commercial project manager, we are concerned that >> the way you chose to handle this is not coming purely of sincere concern for >> the Maven community. >> > Well, since the blocking started today, then obviously the culprit was > found just lately. The performance of central repo was way bad recently, > so I agree with Brian that the blocking was necessary. He questioned me > about the Netbeans integration and how it download the nexus index this > week as well, so I don't think there are some hideous motivations behind > the move. I'm sure that once you fix the issue and the central hit > flooding stops (means people using your betas upgrade to a version that > is fixed), artifactory clients will be reenabled. > > Regards > > Milos > >> This issue is only with the 1.3.0 *betas* (up to beta-6) and has 2 simple >> workarounds: Use an invalid CRON value or exclude repo1 from periodic index >> downloads. >> In any case, Artifactory will download the index on demand, which will use a >> HEAD request first to test for unmodified index (sadly repo1 does not like >> conditional GETs). >> >> We will easily fix this issue in the upcoming release. >> >> To disable the indexer cron set in the UI or in artifactory.config.xml file: >> <indexer> >> <cronExp>invalid</cronExp> >> </indexer> >> And you should see in your log: >> [ERROR] (o.a.r.index. >> IndexerManagerImpl:72) Bad indexer cron expression 'invalid' will be ignored >> (Illegal characters for this position: 'INV'). >> >> To remove repo1 from periodic index downloads: >> <indexer> >> <cronExp>0 0 /1 * * *</cronExp> >> <excludedRepositories> >> <repositoryRef>repo1</repositoryRef> >> </excludedRepositories> >> </indexer> >> >> Happy Thanksgiving, >> >> Yoav Landman >> The Artifactory Team >> >> >> On Fri, Nov 28, 2008 at 2:50 PM, Brian E. Fox <[EMAIL PROTECTED]>wrote: >> >> >>> Since approximately Mid August, the load on Central has been growing at >>> an exponential rate. You may have noticed slowdowns or dropped connections >>> recently as a side effect. We first had issues with Apache HTTPD load >>> increasing above the capacity of the machine. We switched over to Nginx ( >>> http://blogs.sonatype.com/people/brian/2008/10/29/nginx-is-centrals-new-friend/) >>> and this resolved the load, but then the 100mbps connection was regularly >>> becoming saturated. Every hour, on the hour for about 20 minutes around the >>> clock, the connection would max out and then return to about 50% >>> utilization. We spent many days working with Contegix to diagnose the >>> problem but no single source stood out immediately. >>> >>> >>> >>> Yesterday we finally discovered that nearly all the traffic, both the >>> hourly spikes and the 50% background traffic is being caused by downloads of >>> the nexus-index.zip. After investigating the various tools that use this >>> data, we have concluded that Artifactory has a critical bug ( apparently >>> since June: http://issues.jfrog.org/jira/browse/RTFACT-390) that is >>> causing every 1.3 instance to repeatedly download the 27mb zip file. We >>> found many cases of a single ip downloading the index more than 1000 times a >>> day! In the config it is set as follows: >>> >>> >>> >>> <!-- The cron definition to control the activation of the m2eclipse >>> indexer. --> >>> >>> <indexer> >>> >>> <!-- By Default index every 5 hours --> >>> >>> <cronExp>0 0 /5 * * ?</cronExp> >>> >>> </indexer> >>> >>> >>> >>> (this is a quartz syntax which is "s m h…") >>> >>> >>> >>> This by itself wouldn't be a huge issue except for the fact that >>> Artifactory ignores the index.properties file which contains the last update >>> timestamp AND doesn't first issue a HEAD to check the timestamp. This means >>> that every Artifactory 1.3 instance is grabbing this 27mb file at least >>> every 5 hours (we can't explain why certain ips are doing it 1000+ times a >>> day..perhaps the config was modified there or some other scheduling issue is >>> present). >>> >>> >>> >>> For reference, the index on central is only updated once a week on Sundays. >>> >>> >>> >>> >>> To protect the Maven Community from ongoing troubles, we have had to take >>> the extra-ordinary step of blocking all downloads of the index file by >>> Artifactory until this is resolved. Upon doing this, the traffic has fallen >>> to 10% of what it has been in the recent past. If you are using Artifactory, >>> please adjust this cron definition to run only weekly and save yourself and >>> us tons of wasted bandwidth and money. >>> >>> >>> >>> Note that other tools like the Nexus Maven Repository Manager ( >>> http://nexus.sonatype.org), M2e and Q4e use the Nexus Indexer API and are >>> immune to this problem and are not blocked from downloading the index. A new >>> version of Nexus and the Nexus Indexer API will be published soon along with >>> M2e that will leverage incremental indexes to significantly reduce the >>> download requirements and allow near real time index updates. >>> >>> >>> >>> Brian Fox >>> >>> Apache Maven PMC >>> >>> http://blogs.sonatype.org/people/brian >>> >>> ------------------------------------------------------------------------- >>> This SF.Net email is sponsored by the Moblin Your Move Developer's >>> challenge >>> Build the coolest Linux based applications with Moblin SDK & win great >>> prizes >>> Grand prize is a trip for two to an Open Source event anywhere in the world >>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>> _______________________________________________ >>> Artifactory-users mailing list >>> [EMAIL PROTECTED] >>> https://lists.sourceforge.net/lists/listinfo/artifactory-users >>> >>> >>> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Artifactory-users mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/artifactory-users > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
