On Sun, Jan 12, 2014 at 6:09 PM, Joe Van Dyk <[email protected]> wrote: > On Sunday, January 12, 2014 5:57:16 PM UTC-8, Daniel Farina wrote: >> >> On Sun, Jan 12, 2014 at 5:46 PM, Joe Van Dyk <[email protected]> wrote: >> > On Sunday, January 12, 2014 5:45:49 PM UTC-8, Joe Van Dyk wrote: >> >> >> >> On Sunday, January 12, 2014 5:24:41 PM UTC-8, Daniel Farina wrote: >> >>> >> >>> >> >>> On Jan 12, 2014 4:54 PM, "Joe Van Dyk" <[email protected]> wrote: >> >>> > >> >>> > When doing a backup-fetch, I get about 6.5-8 megabytes per second >> >>> > download from S3. I was expecting it to be a bit faster. Measured it >> >>> > through >> >>> > iftop and nethogs. >> >>> > >> >>> > I'm noticing that wal-e is using close to 100% of cpu, top shows it >> >>> > hovering at about 97%. >> >>> >> >>> Python 2.6 you say? This version cannot block 3DES encipherment >> >>> because >> >>> of a missing feature of the ssl module. This fix helped my >> >>> surprisingly >> >>> slow cases a lot. >> >>> >> >>> To confirm, try running Linux 'perf top' and see what symbols crop up. >> >>> The 3DES symbols in openssl were perspicuously named, which is how >> >>> this bug >> >>> was diagnosed. >> >> >> >> >> >> I upgraded to Python 2.7.6, no changes in speed. I'll check out perf >> >> tomorrow. >> >> Unfortunately. The 3DES blocking helped my cases a lot. >> >> > Should wal-e be using such a high cpu percentage? >> >> I haven't tuned WAL-E very much for CPU usage, but 6 megabytes per >> second is suspicious. Can you get a faster rate if you send /dev/zero >> to the disk? > > I can write at about 68 MB/s.
Not so good. I have measured WAL-E going faster in the past, but maybe there was a regression. Unfortunately you may have to take point figuring out why for a little while if you want to see progress Real Soon, because I'm pretty sure there's a nasty bug in backup-fetch right now and that's going to be holding my attention. Maybe one of the other list denizens/github issue-watchers with an eye for optimization can help. I think this level of performance is worth a Github bug report also, if only to widen the distribution in hopes someone will make time to profile. I bet there's some howlers in the code right now, as it has not been tuned for CPU usage modulo that 3DES change to my knowledge...ever. -- You received this message because you are subscribed to the Google Groups "wal-e" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
