Hi,
First of all I wanted to thank the Neo4j team for the great support with
this mail list.
I'm answering here to Johan Svensson reply to the other thread with the same
name,
(*since this is the original I started before it got replicated somehow*).
>Pablo, you had a problem with batch inserter
On Thu, Feb 17, 2011 at 3:10 PM, Johan Svensson wrote:
> Hello,
>
> I am having a hard time to follow what the problems really are since
> conversation is split up in several threads.
>
Totally right and sorry about that, I'll start a new thread for my use case.
Cheers
--
Massimo
http://meridi
Hello,
I am having a hard time to follow what the problems really are since
conversation is split up in several threads.
Pablo, you had a problem with batch inserter throwing an exception
upon shutdown that I suspected was due to not enough available disk
space. Then there was the the "to many op
On Thu, Feb 17, 2011 at 12:54 PM, Pablo Pareja wrote:
> Hi Massimo,
>
> It's too bad you are running into the same kind of situations, (specially
> when
> the conclusion you came up to is that Neo4j just degrades...).
> However, did you already try dividing "the" big insertion process into
> smal
Hi Massimo,
It's too bad you are running into the same kind of situations, (specially
when
the conclusion you came up to is that Neo4j just degrades...).
However, did you already try dividing "the" big insertion process into
smaller
steps?
I mean, do you think Neo4j degradation is just proportiona
On Wed, Feb 16, 2011 at 11:12 AM, Mark Harwood wrote:
> However, the underlying Neo4J database doesn't seem to be able to cope
> with inserting these volumes of data on my available hardware and I
> don't have (and would hope not to need) 10s of gigabytes of RAM to
> throw at the problem.
I'm us
Hi David,
there's another thread I started with the same name: "Help with exception
using BatchInserter". (*that's the one we are*
*referring to)*
I guess Mark started this one by mistake instead of replying to the other.
Cheers,
Pablo
On Wed, Feb 16, 2011 at 6:55 PM, David Montag <
david.mon...@
Hi guys,
I don't have the history of this thread. What issues are you running into?
David
On Wed, Feb 16, 2011 at 2:37 AM, Pablo Pareja wrote:
> Hi Mark,
>
> Thanks for your contribution.
> I'm using a m2.4xlarge EC2 AWS instance type (*68.4 GB of memory, 26
> EC2 Compute
> Units (8 virtual co
Hi Mark,
Thanks for your contribution.
I'm using a m2.4xlarge EC2 AWS instance type (*68.4 GB of memory, 26
EC2 Compute
Units (8 virtual cores with 3.25 EC2 Compute Units each)).*
Besides I'm parsing a 17G xml file for populating the DB and normally
problems only arise when I have
already inserted
Hi Pablo,
>>Regarding the boiled down version of my code I guess I could prepare it but
>>it's quite a big project
Here's a boiled-down batch load demo I did earlier based on public
Wikipedia data: http://code.google.com/p/graphdb-load-tester/
It includes what I believe to be a faster Lucene bat
Hi,
I managed to change the ulimit in my machine and didn't get the "Too many
open files" error
again. However, even though no exceptions were thrown, this morning when I
tested some
queries against the DB results were quite weird and completely unexpected.
Most nodes that should have been inserte
If that many files are kept open at one point in time there's probably a bug
in the code. I don't think that increasing number of files your OS allows is
a correct solution, but may work as a temporary solution. I'd love to get a
boiled down (or the code as is) version of you code so that I can rep
Hi!
2011-02-15 13:19, Pablo Pareja:
> (Too many open files)
This could be that you have old processes around that still keep lots of
files open. Or that you simply have to few file handles available to
begin with. Just google for "Too many open files" + your operating
system, and you'll find i
Here you have the entire stack trace I just got (*now it's a different
error...*):
java.io.FileNotFoundException:
/mnt/bio4j_volume/bio4jdb/lucene/organism_scientific_name_index/_1a9e.fdx
(Too many open files)
org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.index(LuceneIndexBatchInserterImpl.j
Ok, I remember there was a java.io.IOException afterwards, unfortunately I
didn't copy
that part thinking it was the consequence of the previous one.
Anyways I think it said something about an index file that existed but
couldn't be accessed for an unknown reason,
is that possible?
I'll try to repr
There should have been a cause at the end of that trace, looking something
like:
caused by: java.io.IOException ...
That is what I'm interested in.
-tobias
On Tue, Feb 15, 2011 at 10:12 AM, Pablo Pareja wrote:
> I already put the stack trace in the first message:
>
> Unable to write record[24
I already put the stack trace in the first message:
Unable to write record[244986398] @[32583190934]
org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112)
org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindo
I had a second look at this, and realized that those memory mapping settings
aren't used from the batch inserter. Everything is allocated in the Java
Heap. 63G out of 64G RAM is still a bit much, but not as big of a problem as
I first thought.
This also means that there could be another problem be
Ups... I thought memory mapping was using the Java heap space.
Thanks
Pablo
On Mon, Feb 14, 2011 at 10:06 AM, Tobias Ivarsson <
tobias.ivars...@neotechnology.com> wrote:
> Your problem is likely related to the fact that 10G+10G+10G+10G + 63G >
> 64G.
>
> Memory mapping uses memory outside of the
Your problem is likely related to the fact that 10G+10G+10G+10G + 63G > 64G.
Memory mapping uses memory outside of the Java heap. All memory you use,
memory mapping plus Java heap must fit in the available RAM.
Cheers,
Tobias
On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja wrote:
> Hi all,
> I
Hi all,
I keep getting this exception and don't know what the reason could be:
Unable to write record[244986398] @[32583190934]
org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112)
org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.for
21 matches
Mail list logo
