Yoav,
If we wanted to tarpit you or be more malicious that would be very
easy for us to do. We could completely screw you over and cripple your
access forever and there's not really much you could do about it if
that was our motivation. Brian is completely open about what happened
and actually being pretty forgiving. What you've done is essentially
amounts to gross negligence.
You cost us very real amounts of money in wasted bandwidth.
You cost every Maven user real losses with respect to accessibility to
the central repository.
To even insinuate that Brian is doing anything other then trying to
protect the accessibility of central is pretty low.
I would be happy to accept your apology along with US20K that you've
cost us in wasted bandwidth since your shoddily implemented strategy
to grab the Nexus index. You've wasted everyone's time and money. Not
appreciated. You'll be sorely mistaken if you think I'll stand by and
watch you try to implicate Brian partaking in some plan to screw
Artifactory over when that's exactly what Artifactory did to the Maven
community. Take your lumps and fix it.
It's not Apache that pays for the resources, it's the Maven PMC as
individuals with their contributions of time and money that keep it
all working. Nothing is free, certainly not the machines, personnel
and bandwidth required to keep central going. Brian acts on behalf of
the Maven PMC not as the Nexus project manager and there isn't a
single person on the Maven PMC who doesn't agree with him.
On 28-Nov-08, at 9:28 AM, Brian E. Fox wrote:
Yoav,
I sent the notice to everyone who might be affected by this as soon
as we understood this issue and confirmed it by selectively blocking
the suspected incidents (which was before 8am local time when I woke
up and saw that Contegix had confirmed it). Since it was clear that
users could directly affect this themselves by adjusting the cron
time, I thought it prudent to make that request ASAP. Waiting longer
serves no one in this case. Since the bandwidth costs real money,
and was causing users real problems, not blocking wasn't an option
once we understood how to fix it. Blocking and not notifying
everyone involved immediately wouldn't be in the user's best
interests either.
If my motivation was other than to protect Central, we could have
more easily just blocked the high offending IPs and gone back to
enjoying the holiday quietly. Instead, we spent a lot of time really
understanding the problem to make sure we didn't needlessly take out
all Artifactory users in a way that actually broke builds.
I have no doubt that the fix is easy, but it will take time to get
people running fixed versions. Adjusting cron can be done now and
will at least minimize the impact in the meantime so we can unblock
the index.
--Brian
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Sent: Friday, November 28, 2008 12:01 PM
To: Maven Users List
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
; Maven Project Management Committee List
Subject: Re: [Artifactory-users] Artifactory 1.3 has been DOS'ing
the Central repository
Yoav Landman wrote:
Brian,
I think it is only a reasonable expectation from you, to contact
our team
immediately as soon as you found out about this bug, so that we can
let our
users know how to work around it and if they are affected at all.
However,
since you are the Nexus commercial project manager, we are
concerned that
the way you chose to handle this is not coming purely of sincere
concern for
the Maven community.
Well, since the blocking started today, then obviously the culprit was
found just lately. The performance of central repo was way bad
recently,
so I agree with Brian that the blocking was necessary. He questioned
me
about the Netbeans integration and how it download the nexus index
this
week as well, so I don't think there are some hideous motivations
behind
the move. I'm sure that once you fix the issue and the central hit
flooding stops (means people using your betas upgrade to a version
that
is fixed), artifactory clients will be reenabled.
Regards
Milos
This issue is only with the 1.3.0 *betas* (up to beta-6) and has 2
simple
workarounds: Use an invalid CRON value or exclude repo1 from
periodic index
downloads.
In any case, Artifactory will download the index on demand, which
will use a
HEAD request first to test for unmodified index (sadly repo1 does
not like
conditional GETs).
We will easily fix this issue in the upcoming release.
To disable the indexer cron set in the UI or in
artifactory.config.xml file:
<indexer>
<cronExp>invalid</cronExp>
</indexer>
And you should see in your log:
[ERROR] (o.a.r.index.
IndexerManagerImpl:72) Bad indexer cron expression 'invalid' will
be ignored
(Illegal characters for this position: 'INV').
To remove repo1 from periodic index downloads:
<indexer>
<cronExp>0 0 /1 * * *</cronExp>
<excludedRepositories>
<repositoryRef>repo1</repositoryRef>
</excludedRepositories>
</indexer>
Happy Thanksgiving,
Yoav Landman
The Artifactory Team
On Fri, Nov 28, 2008 at 2:50 PM, Brian E. Fox <[EMAIL PROTECTED]
>wrote:
Since approximately Mid August, the load on Central has been
growing at
an exponential rate. You may have noticed slowdowns or dropped
connections
recently as a side effect. We first had issues with Apache HTTPD
load
increasing above the capacity of the machine. We switched over to
Nginx (
http://blogs.sonatype.com/people/brian/2008/10/29/nginx-is-centrals-new-friend/)
and this resolved the load, but then the 100mbps connection was
regularly
becoming saturated. Every hour, on the hour for about 20 minutes
around the
clock, the connection would max out and then return to about 50%
utilization. We spent many days working with Contegix to diagnose
the
problem but no single source stood out immediately.
Yesterday we finally discovered that nearly all the traffic, both
the
hourly spikes and the 50% background traffic is being caused by
downloads of
the nexus-index.zip. After investigating the various tools that
use this
data, we have concluded that Artifactory has a critical bug
( apparently
since June: http://issues.jfrog.org/jira/browse/RTFACT-390) that is
causing every 1.3 instance to repeatedly download the 27mb zip
file. We
found many cases of a single ip downloading the index more than
1000 times a
day! In the config it is set as follows:
<!-- The cron definition to control the activation of the
m2eclipse
indexer. -->
<indexer>
<!-- By Default index every 5 hours -->
<cronExp>0 0 /5 * * ?</cronExp>
</indexer>
(this is a quartz syntax which is "s m h…")
This by itself wouldn't be a huge issue except for the fact that
Artifactory ignores the index.properties file which contains the
last update
timestamp AND doesn't first issue a HEAD to check the timestamp.
This means
that every Artifactory 1.3 instance is grabbing this 27mb file at
least
every 5 hours (we can't explain why certain ips are doing it 1000+
times a
day..perhaps the config was modified there or some other
scheduling issue is
present).
For reference, the index on central is only updated once a week on
Sundays.
To protect the Maven Community from ongoing troubles, we have had
to take
the extra-ordinary step of blocking all downloads of the index
file by
Artifactory until this is resolved. Upon doing this, the traffic
has fallen
to 10% of what it has been in the recent past. If you are using
Artifactory,
please adjust this cron definition to run only weekly and save
yourself and
us tons of wasted bandwidth and money.
Note that other tools like the Nexus Maven Repository Manager (
http://nexus.sonatype.org), M2e and Q4e use the Nexus Indexer API
and are
immune to this problem and are not blocked from downloading the
index. A new
version of Nexus and the Nexus Indexer API will be published soon
along with
M2e that will leverage incremental indexes to significantly reduce
the
download requirements and allow near real time index updates.
Brian Fox
Apache Maven PMC
http://blogs.sonatype.org/people/brian
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great
prizes
Grand prize is a trip for two to an Open Source event anywhere in
the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Artifactory-users mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
Our achievements speak for themselves. What we have to keep track
of are our failures, discouragements and doubts. We tend to forget
the past difficulties, the many false starts, and the painful
groping. We see our past achievements as the end result of a
clean forward thrust, and our present difficulties as
signs of decline and decay.
-- Eric Hoffer, Reflections on the Human Condition
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
