Here's the latest update. Throughout the day Thursday, the indexing queue gradually climed from 19,000 to 23,000. I figured that maybe after hours when there's less activity, it would make headway. But Friday morning, the queue was at 27,000.
I had to bounce it because some users were reporting 405 errors when trying to upload to Archiva. After the bounce, the queue was very small...it started at 3 and slowly climbed, it was at 898 when the hourly indexes kicked in. The bounce did seem to fix the 405 errors though. But then this afternoon, we got reports of 405 errors again. I bounced, and the queue was at 0! During this time, CPU usage was normal...the archiva java process was using trivial CPU. But then the queue went up to 3, and now we have 100% CPU usage again. So it appears that ANY indexing queue at all causes us to hit 100% CPU. The queue is currently at 7...I'm not sure how often it actually is empty. I see that 1.4-M1 was released a few days ago...does it include anything that might address this? -----Original Message----- From: Stallard,David Sent: Thursday, October 27, 2011 9:24 AM To: '[email protected]' Subject: RE: 100% CPU in Archiva 1.3.5 To follow up on my previous message...you mentioned that frequent deployments could create the backlog. We do have a build server which is constantly doing builds of many different products, and each build uploads snapshots to Archiva, so my guess would be that maybe we had a period of heavier than normal builds so that Archiva was inundated with new snapshots to index. -----Original Message----- From: Stallard,David Sent: Thursday, October 27, 2011 9:17 AM To: '[email protected]' Subject: RE: 100% CPU in Archiva 1.3.5 Thanks, Brett. This morning our indexing queue is around 19,000 instead of 135,000, so it must be getting close to catching up with the indexing. I'm not sure how we got to having such a huge backlog; we pinpointed the day and hour that the CPU usage suddenly increased, and in the logs it doesn't look like we had a lot of new files uploaded around that particular time. Maybe at some point the indexing stopped for some reason and then later it started up again? I don't even know if that's possible, just thinking aloud. -----Original Message----- From: Brett Porter [mailto:[email protected]] On Behalf Of Brett Porter Sent: Thursday, October 27, 2011 1:39 AM To: [email protected] Subject: Re: 100% CPU in Archiva 1.3.5 It's working through a backlog of indexing operations, which are CPU and disk intensive. These may have been there from a scan, or from frequent deployments. Keep an eye on whether that count stays high over time. If so, you may need to prevent scanning certain repositories or artifact types that you don't need searched. - Brett On 27/10/2011, at 12:01 AM, Stallard,David wrote: > Although the CPU is still at 100%, Archiva has actually been > responsive for the last 24 hours. Here's what the System Status page > looks like at the moment: > > Queues > Queue Size > database-update 0 > indexing 135310 > repository-scanning 0 > > Repository Scans Currently in Progress No scans in progress. > > Caches > Cache Size Hits Misses Hit Ratio > operations 25 11 50 18% Flush > users 5 130523 106 100% Flush > resources 2 2 4 33% Flush > url-failures-cache 1000 0 0 0% Flush > userPermissions 2 132801 711 99% Flush effective-project-cache 1000 > 87048 30925 74% Flush roles 12 308 29 91% Flush permissions 31 926 62 > 94% Flush effectiveRoleSet 0 0 0 0% Flush userAssignments 0 2 4 33% > Flush ehcache 0 0 0 0% Flush keys 80 5 371 1% Flush > > Memory Usage > 291M/999M (Max: 999M) > > Current Time > Wed Oct 26 08:59:06 EDT 2011 > > -----Original Message----- > From: Brett Porter [mailto:[email protected]] On Behalf Of Brett > Porter > Sent: Wednesday, October 26, 2011 1:33 AM > To: [email protected] > Subject: Re: 100% CPU in Archiva 1.3.5 > > Are you able to visit the "system status" page before it becomes > unresponsive and see what is in progress, and the status of the heap? > > - Brett > > On 26/10/2011, at 7:48 AM, Stallard,David wrote: > >> We have been running Archiva 1.3.4 for several months without issue. >> Last week, CPU usage shot up to 100% or higher...we saw it go as high >> as 300% (this is a 4-CPU host). We were not able to tie this to any >> particular event. >> >> I discovered that 1.3.5 fixes a Dependency Tree issue which can cause >> high CPU usage, so yesterday we upgraded to 1.3.5. At first the >> problem seemed resolved, but over the next 15 hours the CPU usage >> continued to increase until now we are back to having CPU usage at > 100% consistently. >> When the hourly scans of internal and snapshots happen, this jumps up >> to around 200%. >> >> This morning Archiva became unresponsive, which is when I discovered >> that the high CPU is still an issue. I bounced Archiva; while this >> made the system responsive again, the CPU was immediately back at >> 100% > usage. >> >> The logs show no unusual activity. >> >> Any idea what might be happening? >> >> Thanks, >> David > > -- > Brett Porter > [email protected] > http://brettporter.wordpress.com/ > http://au.linkedin.com/in/brettporter > > > > > > -- Brett Porter [email protected] http://brettporter.wordpress.com/ http://au.linkedin.com/in/brettporter
