I am using LOAD CSV to update nodes of an existing label. The label
contains 4M existing nodes. The CSV file contains 600k rows; some new, some
updates.
My Centos 6.x box has 36G of ram. Using the 3.0.1 tuning guide, I have set
pagecache to 24G and heap_initial_size/heap_max_size to 8G.
Attached is my cql file. When executing this cql file neo4j consumes heap
space until space is exhausted. The query plan (attached) appears to
indicate there are no issues so I would expect period commits to re-use
heap. I have adjusted pagecache and heap sizes up and down but every load
attempts fails when heap is exhausted.
Any ideas or thoughts?
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
USING PERIODIC COMMIT 100
LOAD CSV WITH HEADERS FROM
'file:/export/warehouse/DW/accounts_XX_08-23-2016_03-03.txt' as row
FIELDTERMINATOR '\t'
merge (x:XXaccounts {AccountsID:row.AccountsID})
on create set x+=row on match set x=row;
Compiler CYPHER 3.0
Planner COST
Runtime INTERPRETED
+---------------------------------+----------------+------+---------+-----------+----------------------------+
| Operator | Estimated Rows | Rows | DB Hits | Variables
| Other |
+---------------------------------+----------------+------+---------+-----------+----------------------------+
| +ProduceResults | 1 | 0 | 0 |
| |
| |
+----------------+------+---------+-----------+----------------------------+
| +EmptyResult | | 0 | 0 |
| |
| |
+----------------+------+---------+-----------+----------------------------+
| +Apply | 1 | 0 | 0 | row, x
| |
| |\
+----------------+------+---------+-----------+----------------------------+
| | +AntiConditionalApply | 1 | 0 | 0 | row, x
| |
| | |\
+----------------+------+---------+-----------+----------------------------+
| | | +SetNodePropertyFromMap | 1 | 0 | 0 | row, x
| |
| | | |
+----------------+------+---------+-----------+----------------------------+
| | | +MergeCreateNode | 1 | 0 | 0 | x -- row
| |
| | | |
+----------------+------+---------+-----------+----------------------------+
| | | +Argument | 1 | 0 | 0 | row
| |
| | |
+----------------+------+---------+-----------+----------------------------+
| | +ConditionalApply | 1 | 0 | 0 | row, x
| |
| | |\
+----------------+------+---------+-----------+----------------------------+
| | | +SetNodePropertyFromMap | 1 | 0 | 0 | row, x
| |
| | | |
+----------------+------+---------+-----------+----------------------------+
| | | +Argument | 1 | 0 | 0 | row -- x
| |
| | |
+----------------+------+---------+-----------+----------------------------+
| | +Optional | 1 | 0 | 0 | x
| |
| | |
+----------------+------+---------+-----------+----------------------------+
| | +NodeUniqueIndexSeek(Locking) | 1 | 0 | 0 | x
| :XXaccounts(AccountsID) |
| |
+----------------+------+---------+-----------+----------------------------+
| +Limit | 0 | 0 | 0 | row
| Literal(0) |
| |
+----------------+------+---------+-----------+----------------------------+
| +LoadCSV | 1 | 1 | 0 | row
| |
+---------------------------------+----------------+------+---------+-----------+----------------------------+
Total database accesses: 0
