Thanks for the patch, sorry I won't be able to test it now. fsck finished a couple of hours ago without errors (took about 11 hours total) and now I'm running the revision update to use version 2.13.
For now I restored the last daily backup to an EBS volume and will keep the server running from there. I'll stress the volume again (have some files to rsync now) and post about updates. Have you seen s3ql running on similar setups ? Serving webpages / files directly ? Cheers, Guilherme On Tuesday, February 17, 2015 at 3:25:11 PM UTC-2, Nikolaus Rath wrote: > > On Feb 16 2015, Guilherme Barile <[email protected] <javascript:>> wrote: > >>> with high I/O scenarios in the past (using the 1.x series), where the > >>> mount daemon would die under high load, but managed to solve them all > >>> with fsck.s3ql after rebooting. > >> > >> Actually that doesn't sound like a solution at all. mount.s3ql should > >> never crash, now matter how high the load is. Can you still reproduce > >> that? If so, it'd be great if you could post the backtrace. > >> > > This time it happened while I was performing 2 full backups with lots of > > small files, from my mounted s3qlfs to another s3 bucket via duplicity. > > I've experienced this when apache was under high load. When mount.s3ql > > stops, all other processes start to wait for io, increasing the load > > constantly, I can try to force this behaviour on another bucket after I > > restore this one. > > Please do! > > >> Btw, which version is it now? Your subject says 2.12. > > > > > > Filesystem was created on 2.11, I tried upgrading to 2.13 but couldn't > due > > to db revision upgrade, so I had to compile 2.12 to try fsck/upgrade my > > volume - /usr/local/bin/fsck.s3ql --debug s3://my-bucket/home > > > > > >> > Now fsck.s3ql is hanging at ..processed 99500 objects so far.. > >> > fsck.log (with --debug) shows a lot of HEAD requests, It seems to be > >> > running a verify for every block (about 1400000). > >> > >> That should not happen. Please post more context for the logfile. > What's > >> the last message before the HEAD requests are starting? > >> > > > > As this should not happen, I stopped fsck and ran it again to check the > > logs, the output doesn't show any errors > > > > https://gist.github.com/guigouz/7a6a624d97d12918b3f6 > > I think you may be running into a bug that has been fixed in S3QL > 2.13. Could you try the attached patch? It backports the relevant change > to 2.12. > > >>> I also tried > >>> downloading a metadata backup, but it hangs at the same point, here's > >>> an excerpt of the ongoing log: > >> > >> What do you mean with "at the same point"? Downloading metadata should > >> not issue any requests for data objects at all. > > > > fsck hangs at ..processed 99500 objects so far.., no matter which db > backup > > I use > > Ah, so there is no problem with downloading the metadata backup. > > > Best, > -Nikolaus > > -- > GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F > Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F > > »Time flies like an arrow, fruit flies like a Banana.« > -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
