Could you use memcache instead of database?
On Mon, Apr 2, 2012 at 8:28 AM, Michel Pelletier <[email protected]> wrote: > I suspect the big issue is with your push() function, it unpickles a > list, appends a message to it, and then repickles it. As the list > gets bigger and bigger, this operation will get slower and slower > until it consumes almost all cpu time for your program. You'll need a > much more clever persistence strategy than this to get good > performance. You're pickling/unpickling/repickling the same messages > over and over again. > > I suggest starting with something simple, like writing the message to > a database. Also you don't need to unpickle your message and then > repickle it to write it, just write the raw off-the-wire message, then > later when you fetch it from storage, unpickle it at that point. It > seems to me your program doesn't need pickle at all, it could work > entirely with raw messages. > > -Michel > > On Mon, Apr 2, 2012 at 12:34 AM, Sean Ochoa <[email protected]> wrote: >> Hey all. >> >> In my attempt at trying to create a message queue with a persistence layer, >> I may have slowed things down. >> >> Here's my code so far: http://paste.pound-python.org/show/18411/ >> >> I'm wondering if someone could help me tune this thing so that I could put >> the persistence layer back in and still get good performance: 20,000 >> messages / sec or more. >> >> Any help is much appreciated! I'm still learning how to use zeromq. >> >> -- >> Sean >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
