Re: [computer-go] Serializing a very large object in Java

2007-02-11 Thread Roland Illig

Peter Drake wrote:

out.writeObject(root);


Storing the root node of the Monte Carlo search will get storeObject() 
called on all the siblings and children (assuming you are using an 
implementation similar to the one on http://senseis.xmp.net/?UCT.


With 10.000 nodes, the depth of the tree is a little less than with 
2.000.000 nodes. Since for each sibling (important) and each child (not 
so important), the writeObject() method is called, you get that stack 
overflow.


Since there are as many siblings per move as valid moves, the recursion 
depth grows quite quickly.


You can avoid that by not calling writeObject(this.sibling), but doing 
that in a while (...) loop.


Roland
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/


Re: [computer-go] Serializing a very large object in Java

2007-02-10 Thread Jacques BasaldĂșa

Peter Drake wrote (3 times):

 Exception in thread main ...
 . . .  and 91.449 bytes later  . . .
 ... (ObjectOutputStream.java:1369)

I studied the log file in depth and the problem is  . . .

. . . (you guessed) using Java  ;-)

Jacques.

BTW. I store this list. If you need your log file in the future, contact 
me cos I have 3 copies. ;-)


___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/


Re: [computer-go] Serializing a very large object in Java

2007-02-09 Thread wms
Peter, java serialization is not a good way to do persistent storage
of any kind, especially large data structures.

It has some pretty severe drawbacks:

- It is slow
- It breaks easily (ie, becomes unable to load older data sets) when
  you make even small changes in your code.
- It makes inefficient use of space
- The format is difficult to decode or manipulate in any way other
  than reading or writing with the exact .class file used to generate
  the serialization code.

Java serialization is excellent for RMI, but is pretty poor for any
other use. I've used serialization myself several times and I
regretted it every time (except for when I used it for RMI).
 
My advice would be to come up with your own data format and use
that.
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/


Re: [computer-go] Serializing a very large object in Java

2007-02-09 Thread terry mcintyre
Is this opening book database used for the UCT portion, or the playout portion 
of Orego? In the UCT portion, speed of access may not be that important; a 
database would probably be ideal. If used during the playout, then speed of 
access is more crucial.
 
Terry McIntyre
UNIX for hire
software development / systems administration / security

[EMAIL PROTECTED]
- Original Message 
From: Nick Apperson [EMAIL PROTECTED]

What type of data are you trying to serialize or rather store to disk?  Do you 
have pointers in the data.  Don't tell me you don't have pointers just because 
it is java.  Java has pointers, it just preteneds it doesn't.  Show us the 
datastructure and we can probably help you more.  If each entry has no pointers 
than your task should be pretty straightforward.  If you do have them, you are 
going to need to find a way to store those so that they are able to be 
redetermined.  In a language like java, where you can't do pointer arithmetic 
or anything of the sort this can get rather difficult.  Also, have you 
considered using a database instead of a flat file to store this data?


- Nick

On 2/9/07, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote:
Peter, java serialization is not a good way to do persistent storage
of any kind, especially large data structures.

It has some pretty severe drawbacks:

- It is slow
- It breaks easily (ie, becomes unable to load older data sets) when

  you make even small changes in your code.
- It makes inefficient use of space
- The format is difficult to decode or manipulate in any way other
  than reading or writing with the exact .class file used to generate

  the serialization code.

Java serialization is excellent for RMI, but is pretty poor for any
other use. I've used serialization myself several times and I
regretted it every time (except for when I used it for RMI).


My advice would be to come up with your own data format and use
that.
___
computer-go mailing list
computer-go@computer-go.org

http://www.computer-go.org/mailman/listinfo/computer-go/



___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/






 

Any questions? Get answers on any topic at www.Answers.yahoo.com.  Try it now.___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/

Re: [computer-go] Serializing a very large object in Java

2007-02-09 Thread Peter Drake

On Feb 9, 2007, at 10:48 AM, Nick Apperson wrote:

What type of data are you trying to serialize or rather store to  
disk?  Do you have pointers in the data.  Don't tell me you don't  
have pointers just because it is java.


I wouldn't dream of it. Java has pointers and I use the hell out of 'em.

Java has pointers, it just preteneds it doesn't.  Show us the  
datastructure and we can probably help you more.  If each entry has  
no pointers than your task should be pretty straightforward.  If  
you do have them, you are going to need to find a way to store  
those so that they are able to be redetermined.


Java's serialization mechanism does this automagically.

In a language like java, where you can't do pointer arithmetic or  
anything of the sort this can get rather difficult.  Also, have you  
considered using a database instead of a flat file to store this data?


I'm going to try Terry's idea of storing individual Nodes instead of  
storing the entire Node pool. Stay tuned...


Peter Drake
Assistant Professor of Computer Science
Lewis  Clark College
http://www.lclark.edu/~drake/

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/


Re: [computer-go] Serializing a very large object in Java

2007-02-09 Thread Peter Drake
The UCT portion. I'm storing/loading a pre-built UCT tree once at  
startup; the disk is not accessed during the game.


Peter Drake
Assistant Professor of Computer Science
Lewis  Clark College
http://www.lclark.edu/~drake/




On Feb 9, 2007, at 11:08 AM, terry mcintyre wrote:

Is this opening book database used for the UCT portion, or the  
playout portion of Orego? In the UCT portion, speed of access may  
not be that important; a database would probably be ideal. If used  
during the playout, then speed of access is more crucial.


Terry McIntyre
UNIX for hire
software development / systems administration / security

___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/


Re: [computer-go] Serializing a very large object in Java

2007-02-09 Thread Stuart A. Yeates

I serialised some very large Markov models (tens to low hundreds of
megabytes) for my PhD using java serialisation. A couple of hints:

*) they can be faster if you compress them (I used the standard Java
libraries). Disk access was the limiting factor in my case and
compression (I got 80% compression routinely) eased this bottleneck.

*) write functions to allow standard tail recursion optimisations to
performed by the compiler. I admit I never actually tested the
effectiveness of this, but it should improve with successive
generations of compiler. http://en.wikipedia.org/wiki/Tail_recursion

*) I only ever serialised classes of trivial structure which other
classes acted upon. I don't recall serialisation ever breaking. If the
classes you are serialising are the same classes you're changing every
time you make a minor change to your algorithm, that many change. Yes,
this breaks the vision of objects as both the data and the algorithms
acting on them.

*) If your data structure is very deep (which I imagine it is, if
you're seeing stack overflows), you may be better to store pointers to
every node in a hashtable (which is iterated though by the serialiser)
and serialise that.

*) The command line option -Xss controls the size of the stack (with
the Sun JVM). Use it just like the other memory options.

*) as pointed out elsewhere, this is probably not an ideal archival format.

cheers
stuart
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/


RE: [computer-go] Serializing a very large object in Java

2007-02-09 Thread Lucas, Simon M

 Three alternative options to Java's native serialisation:

  * Object database db4o: http://db4o.com  

  * WOX (Web Objects in XML) (my own)
 http://algoval.essex.ac.uk/wox/serial/readme.html

  * JSON (JavaScript Object Notation) - also has Java libraries.

   Simon Lucas


-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED]
Sent: 09 February 2007 18:40
To: computer-go@computer-go.org
Subject: Re: [computer-go] Serializing a very large object in Java

Peter, java serialization is not a good way to do persistent storage of
any kind, especially large data structures.
 

It has some pretty severe drawbacks:
 

- It is slow
- It breaks easily (ie, becomes unable to load older data sets) when
  you make even small changes in your code.
- It makes inefficient use of space
- The format is difficult to decode or manipulate in any way other
  than reading or writing with the exact .class file used to generate
  the serialization code.
 

Java serialization is excellent for RMI, but is pretty poor for any
other use. I've used serialization myself several times and I regretted
it every time (except for when I used it for RMI).
 
My advice would be to come up with your own data format and use that.
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/


Re: [computer-go] Serializing a very large object in Java

2007-02-09 Thread Ray Tayek

At 09:55 AM 2/9/2007, you wrote:

..., Java  has a stack overflow error.


i assume you have tried the java -Xss to set the stack size (type 
java -X for help on these)?


thanks

---
vice-chair http://ocjug.org/


___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/