There is a simple way of solving this issue. Changes that need to be made: 1) Make workers return slots directly to the splitter, not to the muxer.
2) Introduce a new kind of slots, passed between the muxer and workers. One slot grants the right to allocate a 1MB block for output buffer. Whenever a worker is about to allocate memory for output blocks, it needs to acquire a slot from the muxer. If there are no free slots, the worter must wait. However, to avoid possible deadlocks, the muxer will always grant slots to the worker that is processing the block the muxer is expecting to receive next (reord_needed). This algorithm is simple, but works. Of course there are better, more sophisticated algorithms for managing memory. One of them is to be implemented in lbzip2-2.0, which is under development. If anyone is interested I can prepare an appropriate patch. Just let me know. -- To UNSUBSCRIBE, email to [email protected] with a subject of "unsubscribe". Trouble? Contact [email protected]
