Thanks Thomas! So I switched to h2-latest.jar and it fails sooner with an
IllegalStateException:
Caused by: java.lang.IllegalStateException: Negative position
-9223372036854646992 [1.3.173/6]
at
org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:709)
at org.h2.mvstore.MVStore.readPage(MVStore.java:1439)
at org.h2.mvstore.MVMap.readPage(MVMap.java:759)
at org.h2.mvstore.Page.getChildPage(Page.java:207)
at org.h2.mvstore.MVMap.binarySearch(MVMap.java:449)
at org.h2.mvstore.MVMap.binarySearch(MVMap.java:450)
at org.h2.mvstore.MVMap.get(MVMap.java:431)
Here is whats odd though, I basically have 3 large CSV files I'm parsing
right now (#1=500MB/4M rows, #2=2.2GB/26M rows, #3=1.3GB/15M rows), it
makes it through the first 2 fine, then I get the above exception about
100k rows into file #3.
File #3 was also the only file failing with the previously mentioned
BufferOverflowException in 1.3.173-release (it also seemed to have a slow
memory leak), So I'm wondering if I have some weird data in file #3 that is
causing issues? Again, it might be difficult to extract a unit test out of
this, but focusing on replicating file #3 might make it a bit easier.
Brian
On Wednesday, October 9, 2013 1:00:32 PM UTC-6, Thomas Mueller wrote:
>
> Hi,
>
> We fixed quite a number of bugs in the MVStore recently. I suggest to try
> again with the nightly build at
> http://h2database.com/html/build.html#automated - direct link
> http://www.h2database.com/automated/h2-latest.jar - or of course you
> could build it yourself.
>
> > I'm currently using MVStore as a large, temporary disk cache
>
> Noel and me recently implemented an off-heap storage for the MVStore, so
> it's using memory (not disk) outside the normal Java heap. This might be
> interesting for you. The documentation is not online yet (as it's not
> released yet), it is only available in source form yet at
> https://h2database.googlecode.com/svn/trunk/h2/src/docsrc/html/mvstore.html(see
> 'off-heap'). To use it, call:
>
> OffHeapStore offHeap = new OffHeapStore();
> MVStore s = new MVStore.Builder().
> fileStore(offHeap).open();
>
> I'm also thinking about combining the LIRS cache with the MVStore, so that
> you could build an off-heap LIRS cache. It shouldn't be complicated to
> implement (the cache would simply needs a map factory).
> Regards,
> Thomas
>
>
>
> On Wed, Oct 9, 2013 at 7:52 PM, Brian Bray <[email protected]<javascript:>
> > wrote:
>
>> Thomas,
>>
>> Here is a more elusive issue that I'm wondering if its a bug in
>> MVMap/MVStore.
>>
>> I'm currently using MVStore as a large, temporary disk cache, I have a
>> java program that scans about 10GB of raw CSV-ish files, and for each file,
>> plucks out a few fields I care about and stores it in a MVMap. At the end,
>> I merge the data from several MVMaps into a small JSON document which
>> represents all the data buried in the CSV files for a particular entity. I
>> then store that JSON document in H2 for my app to use.
>>
>> Its been working fairly well (and fast!), but there is one weird issue
>> I'm encountering, that throws the exception below after 6-8M rows have been
>> processed. Its going to be tough to extrapolate a test case, I was
>> wondering if you had any insight into this? It seems to go away when I
>> shorten some key sizes, but I don't know if I'm just delaying the problem
>> and eventually this would still happen?
>>
>> I'm using 1.3.173, basically every 50k records or so I call
>> MVstore.store() to flush to disk and eventually its throwing this:
>>
>> Caused by: java.nio.BufferOverflowException
>> at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
>> at java.nio.ByteBuffer.put(ByteBuffer.java:832)
>> at
>> org.h2.mvstore.type.ObjectDataType$SerializedObjectType.write(ObjectDataT
>> pe.java:1515)
>> at
>> org.h2.mvstore.type.ObjectDataType.write(ObjectDataType.java:113)
>> at org.h2.mvstore.Page.write(Page.java:799)
>> at org.h2.mvstore.Page.writeUnsavedRecursive(Page.java:860)
>> at org.h2.mvstore.Page.writeUnsavedRecursive(Page.java:855)
>> at org.h2.mvstore.Page.writeUnsavedRecursive(Page.java:855)
>> at org.h2.mvstore.Page.writeUnsavedRecursive(Page.java:855)
>> at org.h2.mvstore.MVStore.store(MVStore.java:921)
>> at org.h2.mvstore.MVStore.store(MVStore.java:813)
>>
>> Thanks,
>> Brian
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "H2 Database" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To post to this group, send email to [email protected]<javascript:>
>> .
>> Visit this group at http://groups.google.com/group/h2-database.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
--
You received this message because you are subscribed to the Google Groups "H2
Database" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/h2-database.
For more options, visit https://groups.google.com/groups/opt_out.
