Robert Van Dam wrote:
Not to appear impatient but it seems as though my last message was
lost in the ether so I'll repost a condensed version. My apologies if
you did indeed get both.
First, if you use close to 2GB (more or less) of memory {Property.get
'gc.size'} and {Property.get 'gc.threshold'} can both become negative
(from signed int's I assume). I know that there is a pratical limit
of 2GB of active memory due to how garbage collection is performed but
'gc.threshold' can be above 2GB when 'gc.size' is not and if
'gc.threshold' is negative and garbage collection is attempted, then
Mozart will die with a "Virtual Memory Exhausted'. But this can
happen before the active size reaches the practical 2GB limit
(depending on how high you have 'gc.free' set to be) and so it should
not be failing.
This seems to be a bug that we could fix. Please send a bug report.
Second, I am using very large datasets to create very large, deeply
nested records (hence the above memory problems). I want to be able
to pickle these trees for later use. However, trees that are too big
kill the Oz emulator with the message 'Aborted'. With some testing I
believe I can occasional call Pickle.pack on a tree successfully
whereas Pickle.save on the same tree again aborts. Are there any
limitations to how big of a nested record can be pickled? Or is it
just that creating the bytestring is in essence creating a copy of the
tree and so I'm running out of memory (4GB on this machine)? If so,
then why the 'Aborted' message and not 'Virtual Memory Exhausted' as
at other times?
Rob
I presume that Pickle.pack and Pickle.save have somewhat different memory
utilisation that gives the effect you see. I think you are running
against the
limits of 32 bits, and these limits have not been "robustified" against
aborts
as they could be. The first fix should be that the emulator does not abort
but cancels the operation - this is a bug that we could fix (please send a
bug report to [EMAIL PROTECTED] explaining how to reproduce the problem
in a simple way). But this does not fix the real problem, which is that you
need to manipulate data larger than 32 bits. I see two possible solutions.
The first is that you manipulate your records through a data abstraction and
the data abstraction stores the records across multiple processes (using
Mozart's distribution support) and in multiple files. The second is to
use a
64-bit port of Mozart. This is more work: we did experiment with this in
the past (there was such a version running), but the current version is not
yet 64-bit ready because of other development that was done since (as
far as I know; I may be mistaken but one of the developers can correct me).
Peter
_________________________________________________________________________________
mozart-users mailing list
[email protected]
http://www.mozart-oz.org/mailman/listinfo/mozart-users
