What do you think about supporting multiple concurrency backends? We could provide support for both gevent and multiprocessing. Another option, and one that might be particularly suitable if you want to get rid of gevent, would be to look at using the concurrent.futures backport from Python 3.2 and instead support threads and multiprocessing. I'd be happy to help take a stab at this.
On Tue, Oct 14, 2014 at 1:45 PM, Daniel Farina <[email protected]> wrote: > On Tue, Oct 14, 2014 at 11:21 AM, Jason Yan <[email protected]> wrote: >> I was doing some benchmarks of backup-fetch runs on an hs1.8xlarge EC2 >> instance and noticed that I could only hit 500Mbps before wal-e became CPU >> bound (AFAICT due to SSL). I hacked up something quickly to test using >> multiprocessing (see https://github.com/tail/wal-e/tree/multiprocessing) and >> was able to peak at 2.25Gbps before hitting another bottleneck. I graphed a >> comparison of the two here: >> https://dl.dropboxusercontent.com/u/153/wal-e/multiprocessing_vs_gevent_1.png. >> >> I wanted to bring this up to the mailing list to see if there was any merits >> in pursuing this further since it would be a large change. I'm planning on >> testing backup-push and wal-fetch as well. I'm also curious what >> performance others are seeing. > > Long time no see! > > I would like to get rid of gevent entirely. I think it was a mistake > for me to have used it (of this I was convinced by Greg Stark). > > The main concern of something like multiprocessing is memory usage, > especially on systems with overcommit off, and therefore virtual > memory is fully charged. That said, there are things that can > probably be done by that, and some regression is acceptable for > simplifying away gevent and getting the performance methinks. > > Thank you for your insightful and rigorous model. -- You received this message because you are subscribed to the Google Groups "wal-e" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
