Tomas Toft <[EMAIL PROTECTED]> writes: Hi everybody!
> Nope, memory usage has grown.
>
> viff-0.4:
> ~630,000kB
> viff-f993269c2660:
> ~690,000kB
>
> And this is per party (three of them in all).
Okay, thanks for testing! But maybe something else changed since
viff-0.4, so could you try revisions 563bcc37fe47 and f993269c2660?
It is quite easy in a Mercurial clone: if you have all local
modifications committed, then just do a 'hg update 563bcc37fe47' to
change the working copy to that revision. Do a 'hg update tip' to get
back to the tip.
> But I do see a difference: In 0.4 the memory usage was constantly
> max'ed out. In f993269c2660, two parties can finish early and then
> free essentially all their memory. Later they then reallocate ~0.5GB.
> So something good is definitely happening, but it doesn't solve my
> problem :-(
Hmm... I think we need to do some serious work on memory profiling (and
profiling in general). The new gc-test.py program shows a simple way to
do it under Linux -- it could be extended and built into VIFF.
I imagine some setting where memory usage will be sampled and dumped at
the end. The code would then do something like this:
enter_phase('starting x')
x = something big
x.addCallback(enter_phase, 'x ready')
x.addCallback(do_stuff_with_x)
enter_phase('starting y')
y = something else
y.addCallback(enter_phase, 'y ready')
which would result in 'phases' being defined and marked on the graph. If
several phases are active at the same time it could lead to several bars
like in this image:
http://www.bootchart.org/images/bootchart.initial.png
There they trace the boot sequence and each process is a separate bar.
It would be very cool if we could produce something similar!
> And just to describe my "gigantic" computation, it's actually not so
> big: I load 100,000 shares (of Z_7 :-) from disk, reconstruct the
> associated values, and print them.
Yeah, that sounds simple :-)
> Saying 4 bytes/field-element (say using 32-bit integers as could be
> done easily in C), and 400,000 field-elements (100,000 shares from
> each of the three parties and 100,000 reconstructed secrets) this
> amounts to around an astonishing 400kB (per party).
Right, 100,000 32-bit values would be 400 KB. If you allocated them as
integers using the array module, then they would take up close to 400 KB
in Python. But each Share holds a FieldElement which holds two integers.
So some of the massive overhead is simply due to more objects being
tossed around. Then there is a per object malloc overhead, reference
counters, list overallocation, etc... :-/
By the way, there is now an open ticket at Twisted about the memory
consumption of Deferred:
http://twistedmatrix.com/trac/ticket/3245
I made a patch and I hope they will eventually include it. Another guy
reported that using __slots__ cut the memory usage in half.
> One positive thing here is that I've gotten an idea of how to
> potentially remove /some/ of that overhead (honest-but-curious only).
Have you tried the code I sent you? I posted it here as well:
http://article.gmane.org/gmane.comp.python.twisted/16519
and got two comments that said that similar functionality already
existed in Twisted :-) There is a DeferredQueue and DeferredSemaphore in
twisted.internet.defer which could be used to limit concurrency.
> The observation is that we don't need to store all shares and
> reconstruct at the end. We can just keep track of the sum of the
> shares (times their lambda's) we've recieved. Whenever a new one
> arrives we add it to that accumulator. This might not make a big
> difference in a 3-party setting, but if many (e.g. 10) parties are
> involved, then this seems like a good idea to reduce memory
> requirements.
Do you mean that we can sort of recombine as we go? I can see how that
makes sense if we know which recombination vector to use -- and if
people are honest-but-curious (as you write) then that is no problem.
In the honest-but-curious setting we could even make it so that only t+1
players ever send their shares. If there are S such subsets, then we
could choose subset number "hash(program_counter) % S" for each
operation -- that would distribute the load evenly over all parties.
--
Martin Geisler
pgpLtAi6RH12Z.pgp
Description: PGP signature
_______________________________________________ viff-devel mailing list (http://viff.dk/) [email protected] http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
