Hi,
33M nodes, 144M relationships. I've imported using "neo4j-importer" and
it's volume on the disk is 9GB.
I've run it on my Mac with -Xmx12g -Xms12g. It seems that it's working if I
don't open any new *outputFileStream.*
I'm getting error after dataset initialization before any scanning over
nodes and when I want to open an output file to write on.
File storeDir = new File(dataGraphPath);
dataGraph = new GraphDatabaseFactory().
newEmbeddedDatabaseBuilder(storeDir)
.setConfig(GraphDatabaseSettings.
pagecache_memory, "6g")
.newGraphDatabase();
File fout = new File(middleFile);
FileOutputStream fos = new FileOutputStream(fout);
I didn't try that. Should I remove that folder first then run it?
Exception in thread "main" java.io.FileNotFoundException: focusSetIDS.txt (
Too many open files in system)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at src.simpleTests.FocusSetCounterIDS.run(FocusSetCounterIDS.java:95
)
at src.simpleTests.FocusSetCounterIDS.main(FocusSetCounterIDS.java:
59)
On Thursday, October 6, 2016 at 10:47:39 PM UTC-7, Michael Hunger wrote:
>
> How big is the dataset and when / how do you import it?
>
> Did you try to delete the schema/labels folder (it is rebuilt)
>
> On Fri, Oct 7, 2016 at 2:23 AM, Mohammad Hossain Namaki <mhn....@gmail.com
> <javascript:>> wrote:
>
>>
>> Version 3.0.1
>>
>> Sorry, I didn't get it, what do you mean by reusing the previous labels?
>> I'm getting the error after initialization of the data graph when I want
>> to create an output text file to write something inside of it.
>>
>>
>>
>> Thanks for property suggestion. However, I don't know about "label
>> combinations".
>>
>> I've attached the complete stack trace and debug.log of the dataset.
>>
>>
>>
>>
>> On Thursday, October 6, 2016 at 5:08:03 PM UTC-7, Michael Hunger wrote:
>>>
>>> Could you re-use previous labels?
>>> How big is your graph?
>>>
>>> Which exact Neo4j version do you use? 3.0.6 ?
>>>
>>> If you don't care about the time you could also use a property instead
>>> of a label or use label combinations.
>>>
>>> I'll see if there is any other way around it.
>>>
>>> Michael
>>>
>>> On Fri, Oct 7, 2016 at 1:45 AM, Mohammad Hossain Namaki <
>>> mhn....@gmail.com> wrote:
>>>
>>>> Hi, Thanks for the reply.
>>>> I use labels for different graph pattern matching algorithm e.g.
>>>> Subgraph isomorphism, bi-simulation, ......
>>>> I use getLabels().iterator().next().name() but I don't use
>>>> dataGraph.getAllLabels() or dataGraph.getAllLabelsInUse(). I can wait
>>>> for getting the label from disk no need to be in memory(because I have my
>>>> own neighborhood indexing which works just by demand on the part of the
>>>> graph that I'm focusing on)
>>>>
>>>>
>>>>
>>>>
>>>> On Thursday, October 6, 2016 at 4:38:32 PM UTC-7, Michael Hunger wrote:
>>>>>
>>>>> What are you using the labels for?
>>>>>
>>>>> On Thu, Oct 6, 2016 at 7:58 PM, Mohammad Hossain Namaki <
>>>>> mhn....@gmail.com> wrote:
>>>>>
>>>>>> Thanks.
>>>>>> So, there is no way that I can turn-off this indexing? My conjecture
>>>>>> is Neo4j is a disk-based database management system and it should use
>>>>>> bounded resources. If I have more resources it can use more and give me
>>>>>> better performance but if I don't have, it shouldn't.
>>>>>> Is there anyway to configure this in Java API?
>>>>>>
>>>>>> I'm just aware of "pagecache_memory":
>>>>>>
>>>>>> File storeDir = new File(dataGraphPath);
>>>>>> dataGraph = new
>>>>>> GraphDatabaseFactory().newEmbeddedDatabaseBuilder(storeDir)
>>>>>> .setConfig(GraphDatabaseSettings.pagecache_memory,
>>>>>> "6g").newGraphDatabase();
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thursday, October 6, 2016 at 2:13:15 AM UTC-7, Michael Hunger
>>>>>> wrote:
>>>>>>>
>>>>>>> perhaps you can use a docker container instead which allows you to
>>>>>>> configure the open file limit?
>>>>>>>
>>>>>>> For storing the lookup from label -> list of nodes Neo4j uses an
>>>>>>> index under the hood.
>>>>>>>
>>>>>>> Michael
>>>>>>>
>>>>>>> On Wed, Oct 5, 2016 at 5:24 AM, Mohammad Hossain Namaki <
>>>>>>> mhn....@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> I've imported a huge dataset into Neo4j. 33M nodes and 144M
>>>>>>>> relationships. Thanks to Neo4j makers, "neo4j-importer" was very
>>>>>>>> efficient.
>>>>>>>> However, I'm getting (Too many open files) errors.
>>>>>>>>
>>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>> Caused by: java.nio.file.FileSystemException:
>>>>>>>> /fastscratch/mnamaki/idsForExp/idsAttackDB/schema/
>>>>>>>> *label/lucene/labelStore/1*: *Too many open files*
>>>>>>>>
>>>>>>>>
>>>>>>>> I've read some question/answer regarding this. However, I'm not the
>>>>>>>> administrator of the system that I run my java code on there. I'm
>>>>>>>> using
>>>>>>>> Neo4j 3.0 using java API.
>>>>>>>>
>>>>>>>> So, I cannot increase the hard-limit of no-files which is 10240 in
>>>>>>>> the linux server that I have.
>>>>>>>>
>>>>>>>>
>>>>>>>> Is there any way that I can shut down this feature of Neo4j? I
>>>>>>>> didn't "index" anything explicitly and just used a command to import
>>>>>>>> nodes
>>>>>>>> and relationships there.
>>>>>>>>
>>>>>>>> Or
>>>>>>>>
>>>>>>>> Is there any way that I can handle this without admin priviledge?
>>>>>>>>
>>>>>>>>
>>>>>>>> How can I understand how many number of files this dataset is
>>>>>>>> required?
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Neo4j" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to neo4j+un...@googlegroups.com.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Neo4j" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to neo4j+un...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Neo4j" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to neo4j+un...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Neo4j" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to neo4j+un...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
