On Tue, 2012-11-06 at 11:26:40 -0800, Mark Sapiro wrote: > Actually, AvoidDuplicates.py ccould serve as a good example, but it is > currently not actually used. It is experimental and is bot included in > the default GLOBAL_PIPELINE.
As you noted in your follow-up, the docstring does not at all describe what that handler actually does. I learned this when actually stepping through the code. :) > The major problem with keeping these data in-memory other than purging > "old" entries so that the dictionary doesn't grow too large, is that > in-memory data aren't shared between runners so if the incoming queue > is sliced, the multiple copies of IncomingRunner do not have access to > each other's data. > > In your case, the input to the hash on which runners are sliced > includes all the message headers and the listname so it is likely that > the "equivalent but different" listname messages will be in different > slices of the hash space. > > This is not a concern if IncomingRunner is not sliced. It is also not a > concern with a disk based cache as long as buffers are flushed after > writing because IncomingRunner locks the list whose message is being > processed which should prevent race conditions between different > slices of IncomingRunner. Then, would it make sense (or be overkill) to have the handler populate a dict of key, value = message-id, timestamp? And, store that dict in a pickle whose filename is derived from mlist.internal_name()? Obviously, this would result in a lot of pickles that are constantly opened, edited (and, periodically cleansed), and closed. Is the performance cost/benefit prohibitive? I would also be relying on the fact that a handler is never concurrently called for the same list -- is that understanding accurate? -- which avoids the scenario in which we are trying to simultaneously manipulate the same pickle. -- Sahil Tandon ------------------------------------------------------ Mailman-Users mailing list [email protected] http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
