Hi Pascal and Daniel,
I uploaded a heap dump when out of memory error happened[1]. Still the
most memory was consumed by Sat4j for resolving dependencies, P2
'Projector' did not hold too much BigInteger instances.
[1] http://dl.dropbox.com/u/4064358/java_pid17676.hprof.gz
Mengxin Zhu
On 02/18/2013 02:06 AM, Pascal Rapicault wrote:
Meng,
It would help if you could provide a heap dump (so we can feed it to
MAT). I think that the weights of the objects is one thing, but the
representation of the dependencies is another thing that takes memory.
As for the second optimization that Daniel mentions, I believe that
you should be able to implement it in the Projector.
Pascal
On 2013-02-17, at 9:23 AM, zhu kane <[email protected]
<mailto:[email protected]>> wrote:
Hi Daniel,
Thanks for your explanation! It's very good to know latest Sat4j
provides an optional way to reduce memory footprint.
I indeed created a test case for investigating it. The test data
makes the patch is large, you can download it[1], then apply it to
p2's source code.
A new test case 'MemoryConsuming' is introduced, free to use any
profile tool to observe the memory footprint.
You also mentioned second way to reduce memory footprint, is it
doable way by client code, such as p2?
[1]
http://dl.dropbox.com/u/4064358/0001-A-test-case-for-memory-footprint-of-SAT4J.patch.gz
Mengxin
On Sun, Feb 17, 2013 at 9:49 PM, Daniel Le Berre <[email protected]
<mailto:[email protected]>> wrote:
Le 17/02/2013 13:13, Mengxin Zhu a écrit :
> Hi Daniel and Pascal,
>
> I observed SAT4J library need lots of memory for resolving
dependencies.
>
> In our case we want to install 677 root IUs, and there are totally
> 17000 IUs in multiple repositories. More than 240M memory was
used in
> resolving process. Especially SAT4J library totally created
more than
> 210M int[] instances, and held them until the calculating
completed.
> Above memory usage data is obtained in VisualVM.
>
> Is it expected behavior to create and hold such large int array for
> calculating the dependencies? If it's unavoidable trade-off for the
> calculating, do we have a formula to estimate how much memory
will be
> required when resolving dependencies in certain number of IUs?
>
> Thank you in advance,
> Cheers,
> Mengxin Zhu
> _______________________________________________
> p2-dev mailing list
> [email protected] <mailto:[email protected]>
> https://dev.eclipse.org/mailman/listinfo/p2-dev
Hi Mengxin,
Sat4j uses arbitrary precision arithmetic (BigInteger) to manage the
weights used to order the UIs.
We use really big weights (e.g. with 30 digits!) to make sure
that those
IUs are applied whatever the consequence are (e.g. for patches).
In some cases, the information can be stored on long, to reduce
memory
usage. This is available in Sat4j 2.3.3 for instance.
sat4j 2.3.3 is currently being considered for inclusion with
Kepler. If
you can provide us with a test case, we could check if we could
reduce
the memory footprint
using long instead of BigInteger.
Another way to reduce the memory footprint would be to cache the
BigInteger object used: the objective function reuses many time
the same
weight.
Reusing the same BigInteger object each time should significantly
decrease the memory footprint too.
Cheers,
Daniel
_______________________________________________
p2-dev mailing list
[email protected] <mailto:[email protected]>
https://dev.eclipse.org/mailman/listinfo/p2-dev
_______________________________________________
p2-dev mailing list
[email protected] <mailto:[email protected]>
https://dev.eclipse.org/mailman/listinfo/p2-dev
_______________________________________________
p2-dev mailing list
[email protected]
https://dev.eclipse.org/mailman/listinfo/p2-dev
_______________________________________________
p2-dev mailing list
[email protected]
https://dev.eclipse.org/mailman/listinfo/p2-dev
