Tags: patch
Hi,
I think my attached patch takes care of this issue.
I've copied code from consumer_decompress() to consumer(), which limits
the output buffer to 20 * numCPU. It seems a bit of a hack to me, but,
probably no more than the rest of the program... but hey, it seems to work!
With this patch, the script attached to my previously email no longer
seems to trigger excessive memory usage. It'll be quite thoroughly
tested as I re-rerun my AMANDA backup with more than 1000 GiB of data.
Regards,
--
Steven Chamberlain
[email protected]
--- pbzip2-1.0.5.orig/pbzip2.cpp 2009-01-08 21:08:17.000000000 +0000
+++ pbzip2-1.0.5/pbzip2.cpp 2009-10-13 20:36:51.000000000 +0100
@@ -1517,7 +1517,16 @@
pthread_mutex_lock(OutMutex);
OutputBuffer[blockNum].buf = CompressedData;
OutputBuffer[blockNum].bufSize = outSize;
+ NumBufferedBlocks++;
pthread_mutex_unlock(OutMutex);
+ while (NumBufferedBlocks > (numCPU * 20))
+ {
+ #ifdef PBZIP_DEBUG
+ fprintf(stderr, "consumer: Throttling from FileWriter backlog: %d\n", NumBufferedBlocks);
+ #endif
+ usleep(50000);
+ }
+
if (FileData != NULL)
{
