Hi, I'm sorry I can't say what the problem is. It kind of looks like it's trying to read from a chunk while the chunk is not yet written. But I would need to know what your are doing exactly.
- Did you start with a new file? - How large is the file when the problem occurs? - How do you open the MVStore? - How to you process the data (concurrently, when do you call commit, do you call save yourself and if yes when,...)? - How large is a typical record? - What data types do you use? Regards, Thomas On Wed, Oct 9, 2013 at 11:41 PM, Brian Bray <[email protected]> wrote: > Thanks Thomas! So I switched to h2-latest.jar and it fails sooner with an > IllegalStateException: > > Caused by: java.lang.IllegalStateException: Negative position > -9223372036854646992 [1.3.173/6] > at > org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:709) > at org.h2.mvstore.MVStore.readPage(MVStore.java:1439) > at org.h2.mvstore.MVMap.readPage(MVMap.java:759) > at org.h2.mvstore.Page.getChildPage(Page.java:207) > at org.h2.mvstore.MVMap.binarySearch(MVMap.java:449) > at org.h2.mvstore.MVMap.binarySearch(MVMap.java:450) > at org.h2.mvstore.MVMap.get(MVMap.java:431) > > Here is whats odd though, I basically have 3 large CSV files I'm parsing > right now (#1=500MB/4M rows, #2=2.2GB/26M rows, #3=1.3GB/15M rows), it > makes it through the first 2 fine, then I get the above exception about > 100k rows into file #3. > > File #3 was also the only file failing with the previously mentioned > BufferOverflowException in 1.3.173-release (it also seemed to have a slow > memory leak), So I'm wondering if I have some weird data in file #3 that is > causing issues? Again, it might be difficult to extract a unit test out of > this, but focusing on replicating file #3 might make it a bit easier. > > Brian > > On Wednesday, October 9, 2013 1:00:32 PM UTC-6, Thomas Mueller wrote: > >> Hi, >> >> We fixed quite a number of bugs in the MVStore recently. I suggest to try >> again with the nightly build at http://h2database.com/html/** >> build.html#automated <http://h2database.com/html/build.html#automated> - >> direct link >> http://www.h2database.**com/automated/h2-latest.jar<http://www.h2database.com/automated/h2-latest.jar>- >> or of course you could build it yourself. >> >> > I'm currently using MVStore as a large, temporary disk cache >> >> Noel and me recently implemented an off-heap storage for the MVStore, so >> it's using memory (not disk) outside the normal Java heap. This might be >> interesting for you. The documentation is not online yet (as it's not >> released yet), it is only available in source form yet at >> https://h2database.**googlecode.com/svn/trunk/h2/** >> src/docsrc/html/mvstore.html<https://h2database.googlecode.com/svn/trunk/h2/src/docsrc/html/mvstore.html>(see >> 'off-heap'). To use it, call: >> >> OffHeapStore offHeap = new OffHeapStore(); >> MVStore s = new MVStore.Builder(). >> fileStore(offHeap).open(); >> >> I'm also thinking about combining the LIRS cache with the MVStore, so >> that you could build an off-heap LIRS cache. It shouldn't be complicated to >> implement (the cache would simply needs a map factory). >> Regards, >> Thomas >> >> >> >> On Wed, Oct 9, 2013 at 7:52 PM, Brian Bray <[email protected]> wrote: >> >>> Thomas, >>> >>> Here is a more elusive issue that I'm wondering if its a bug in >>> MVMap/MVStore. >>> >>> I'm currently using MVStore as a large, temporary disk cache, I have a >>> java program that scans about 10GB of raw CSV-ish files, and for each file, >>> plucks out a few fields I care about and stores it in a MVMap. At the end, >>> I merge the data from several MVMaps into a small JSON document which >>> represents all the data buried in the CSV files for a particular entity. I >>> then store that JSON document in H2 for my app to use. >>> >>> Its been working fairly well (and fast!), but there is one weird issue >>> I'm encountering, that throws the exception below after 6-8M rows have been >>> processed. Its going to be tough to extrapolate a test case, I was >>> wondering if you had any insight into this? It seems to go away when I >>> shorten some key sizes, but I don't know if I'm just delaying the problem >>> and eventually this would still happen? >>> >>> I'm using 1.3.173, basically every 50k records or so I call >>> MVstore.store() to flush to disk and eventually its throwing this: >>> >>> Caused by: java.nio.**BufferOverflowException >>> at java.nio.HeapByteBuffer.put(**HeapByteBuffer.java:183) >>> at java.nio.ByteBuffer.put(**ByteBuffer.java:832) >>> at org.h2.mvstore.type.**ObjectDataType$** >>> SerializedObjectType.write(**ObjectDataTpe.java:1515) >>> at org.h2.mvstore.type.**ObjectDataType.write(** >>> ObjectDataType.java:113) >>> at org.h2.mvstore.Page.write(**Page.java:799) >>> at org.h2.mvstore.Page.**writeUnsavedRecursive(Page.**java:860) >>> at org.h2.mvstore.Page.**writeUnsavedRecursive(Page.**java:855) >>> at org.h2.mvstore.Page.**writeUnsavedRecursive(Page.**java:855) >>> at org.h2.mvstore.Page.**writeUnsavedRecursive(Page.**java:855) >>> at org.h2.mvstore.MVStore.store(**MVStore.java:921) >>> at org.h2.mvstore.MVStore.store(**MVStore.java:813) >>> >>> Thanks, >>> Brian >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "H2 Database" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to h2-database...@**googlegroups.com. >>> To post to this group, send email to [email protected]. >>> >>> Visit this group at >>> http://groups.google.com/**group/h2-database<http://groups.google.com/group/h2-database> >>> . >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "H2 Database" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/h2-database. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "H2 Database" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/h2-database. For more options, visit https://groups.google.com/groups/opt_out.
