[Tracker-discuss] [issue504] Server error

Ezio Melotti added the comment: Thanks! -- status: chatting -> resolved ___ PSF Meta Tracker ___ _

[Tracker-discuss] [issue504] Server error

Izak Burger added the comment: The situation has been resolved. The RAID array is back in sync and all seems well again. No more klingon messages in syslog :-) ___ PSF Meta Tracker __

[Tracker-discuss] [issue504] Server error

Antoine Pitrou added the comment: Le dimanche 20 janvier 2013 à 07:35 +, Izak Burger a écrit : > At the moment I'm not having joy with smartmontools. The initial stats > showed some 28 bad sectors that were pending a swap, which isn't too > bad, but a full offline scan (which despite it's nam

Re: [Tracker-discuss] [issue504] Server error

On Sun, Jan 20, 2013 at 9:35 AM, Izak Burger wrote: > The files we lost were almost all log files, even in the other virtual > hosts on that machine. One of postgresql's WAL logs also failed but I > could recover it from a previous copy. By simply doing a few > successive rsyncs I got all the data

[Tracker-discuss] [issue504] Server error

Izak Burger added the comment: I worked on it quite a bit yesterday. I pushed the load average to 36 at times, which was almost entirely because of disk issues (Linux deems processes in the disk queue as runnable, so a high load average often points to a long disk queue). It should be better sinc

[Tracker-discuss] [issue504] Server error

Ezio Melotti added the comment: > Also, a bug I posted got an id but apparently was never committed to disk, > which hints to an underlying I/O error. I've seen this happening other times in the past, and I don't think it necessarily imply that it couldn't commit it to disk -- it might have bee

[Tracker-discuss] [issue504] Server error

Antoine Pitrou added the comment: Hi Izak, There are lots of problems with the tracker tonight, requests sometimes fail or take a lot of time. Ezio Melotti, who has shell access to the machine, says the CPUs are sometimes all waiting for I/O. Also, a bug I posted got an id but apparently was

[Tracker-discuss] [issue504] Server error

Antoine Pitrou added the comment: The requests are very slow on bugs.p.o currently, and Ezio Melotti points out that the 8 CPUs are at 50% in the "waiting" state. -- nosy: +pitrou priority: -> urgent ___ PSF Meta Tracker

[Tracker-discuss] [issue504] Server error

Izak Burger added the comment: It turns out the issue is a bit more complex than that. /dev/sda has been removed from the array. Normally just re-adding it causes the bad block to be swapped out, but /dev/sdb has sectors itself that might be suspect, so it cannot rebuild. It appears all these are

[Tracker-discuss] [issue504] Server error

New submission from Izak Burger: Hi all, I got an notification during the night that something is amiss on the psf virtual server hosting bugs.python.org. I've checked the filesystem, repaired it, and restarted the virtual server. The ground reason appears to be that we lost a disk in the RAID1.