> Andriy, I've run some tests with memcached and the overhead in grabbing > the python object (a list of objects actually) is too much. The objects > are fairly large, otherwise I would definitely agree with this solution.
1. Try split list of objects into chunks processed by distinct workers. This solution scales well horizontally, you have no duplication of data between workers. 2. The bigger object tree you transfer between boundaries, the higher impact on serialization. A custom data exchange protocol may be a solution, however that requires better understanding of data structures. Usually binary protocols offers better productivity over text based, however that cause also higher complexity. Raw data structures with on demand parsing may help. 3. Consider place data processing logic at backend side, to eliminate network boundary between worker and backend. Transfer only `display ready` data as result of such processing. Investigate if your backend allows a sort of hooks for data processing. I am pretty sure the problem you faced is not in technical area, it is just result of inefficient data processing. A solution to your problem is in a way you process data. With better understanding of data workflow and analysis of processes involved in processing there might be found a better algorithmic solution. Andriy _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
