Hi Jose,
On Saturday, November 29, 2014 1:12:52 AM UTC+2, José F. Morales wrote:
>
>
>> 1. On nodes and their labels. First of all, I strongly suggest you to
>> separate your nodes into different .csv files by label. So you won't have a
>> column *`label`* in your .csv but rather set of files:
>>
>> nodes_LabelA.csv
>> ...
>> nodes_LabelZ.csv
>>
>> whatever your labels are. (Consider label to be kinda of synonym for
>> `class` in object-oriented programming or `table` in RDBMS). That's due the
>> fact that labels in Cypher are somewhat specific entities and you probably
>> won't be allowed to make them parameterized into variables inside your LOAD
>> CSV statement.
>>
>>
> OK, so you have modified your original idea of putting the db into two
> files 1 nodes , 1 relationships. Now here you say, put all the nodes into
> 1 file/ label. The way I have worked with it, I created 1 file for a
> class of nodes I'll call CLT_SOURCE and another file for a class of nodes
> called CLT_TARGET.
>
Ok, but how many valid distinct combinations of your 10 node labels may
exist? I was speaking about a simple case where you have some limited
number of possible node labels (or their combinations), say less than 10.
You are recommending that with the nodes, I take two steps...
> 1) Combine my CLT_SOURCE and CLT_TARGET nodes,
>
Not nessesary "combine" but just give each node a unique (temporary)
*my_node_id
*see my "10+M tree" example below.
> 2) then I split that file into files that correspond to the node:
> *my_node_id, * 1 label, and then properties P1...Pn. Since I have 10
> Labels/node, I should have 10 files named..... Nodes_LabelA...
> Nodes_LabelJ. Thus...
>
You may have as much labels per node you wish, but it is all about how many
valid distinct combinations of labels you have. (One single label is a
combination itself, obviously).
If you have some limited quantity of valid label combination it's one
story. But if we are talking about order of 10! possible valid
combinations, the story is somewhat more interesting :) Which setup is
yours?
> File: CLT_Nodes-LabelA columns: *my_node_id,* label A, property
> P1..., property P4
> ...
> File: CLT_Nodes-LabelJ columns: *my_node_id,* label B, property
> P1..., property P4
>
>
> Q1: What are the rules about what can be used for *my_node_id? *I have
> usually seen them as a letter integer combination. Is that the convention?
> Sometimes I've seen a letter being used with a specific class of nodes
> a1..a100 for one class and b1..b100 for another. I learned the hard way
> that you have to give each node a unique ID. I used CLT_1...CLT_n for my
> CLT_SOURCE nodes and CLT_TARGET_1...CLT_TARGET_n for my TARGET nodes.
> It worked with the smaller db I made. Anything wrong using the convention
> n1...n100?
>
I'm not aware of any conventions here, the only thing I know for sure is
that *schema index works much(!) faster on plain integers than on Unicode
strings*. That's the only difference which I consider significant. So my
personal preference is to have *my_node_id* to be a unique integer. Once
when importing a 10+ millions nodes into a tree with variable height [1..7]
where each level of nodes was in a separate file (because of level's own
unique label and unique set of properties) I just selected a schema for
numbering them like
:Skewer:Level1 my_node_id = 10000000 + file1.csv line number
:Skewer:Level2 my_node_id = 20000000 + file2.csv line number
...
:Skewer:Level7 my_node_id = 70000000 + file7.csv line number
so relationship file (all relationships were of a same single type) has
become a simple 2 column .csv like this with 10+ millions of lines
10000017,20000362
10000017,20000547
10000017,40083215
10000018,30000397
...
After successful importing of 7 node files (and have nodes ready in db and
indexed on their unique *my_node_id* under the label :Skewer) I split
relationships.csv into 1000+ files with 10000 lines each and wrote a dumb
shell script which loaded them with `neo4j-shell -c` file by file doing
`sleep 60` between files (to give neo4j a minute to complete each batch
transaction) than started it Friday evening and got my tree ready on Monday
morning :)
If you prefer alphanumerics for my_node_id it's completely up to you :)
Anyway, after successful import you may prefer to remove those temporary
ids completely from the database, just to conserve space where properties
are stored.
> 2. Then consider one additional "technological" label, let's name it
>> `:Skewer` because it will "penetrate" all your nodes of every different
>> label (class) like a kebab skewer.
>>
>> Before you start (or at least before you start importing relationships) do
>>
>> CREATE CONSTRAINT ON (every_node:Skewer) ASSERT every_node.my_node_id IS
>> UNIQUE;
>>
>>
> Q2: Should I do scenario 1 or 2?
>
> Scenario 1: add two labels to each file? One from my original nodes and
> one as "Skewer"
>
> File 1: CLT_Nodes-LabelA columns: *my_node_id,* label A, *Skewer*,
> property P1..., property P4
> ...
> File 2: CLT_Nodes-LabelJ columns: *my_node_id,* label J, *Skewer*,
> property P1..., property P4
>
> OR
>
> Scenario 2: Include an eleventh file thus....
>
> File 11: CLT_Nodes-LabelK columns: *my_node_id,* *Skewer*, property
> P1..., property P4
>
> From below, I think you mean Scenario 1.
>
Yes and you don't need to add a column for :Skewer label into a file, the
LOAD CSV statement should assign it.
> Q3: “Skewer” is just an integer right? It corresponds in a way to
> my_node_id
>
No, it's a label! so in Cypher your node (suppose it has 2 labels :LabelA
and :LabelJ ) is described like
MATCH (n:LabelA:LabelJ:Skewer {my_node_id: 123454, p1: 'something', p2:
'something
else', p3: 'etc.'})
Here is some sort of cypher….
>
> //Creating the nodes
>
>
>
> USING PERIODIC COMMIT 1000
>
> LOAD CSV WITH HEADERS FROM “…/././…. CLT_NODES_LabelA.csv" AS csvline
>
> MERGE (my_node_id:Skewer: LabelA {property1: csvline.property1})
>
> ON CREATE SET
>
> n.Property2 = csvline.Property2,
>
> n.Property3 = csvline.Property3,
>
> n.Property4 = csvline.Property4;
>
>
> ….
> LOAD CSV WITH HEADERS FROM “…/././…. CLT_NODES_LabelJ.csv" AS csvline
>
>
>
> MERGE (my_node_id:Skewer: LabelJ {property1: csvline.property1})
>
> ON CREATE SET
>
> n.Property2 = csvline.Property2,
>
> n.Property3 = csvline.Property3,
>
> n.Property4 = csvline.Property4;
>
>
>
>
> Q4: So does repeating the LOAD CSV with each file CLT_NODES_LabelA…J
> combine the various labels and their respective values with their
> corresponding nodes?
>
Label is not a variable, it does not have a value. It's just a label,
consider "tag".
Also *my_node_id* IS a variable so it does have a value.
Looking at your 2 code snippets - in case you hope that the first one will
create a node with LabelA and the second one will assign LabelJ to a node
which was created earlier, you are wrong. But... if you remove labels from
MERGE, it will work but look here with attention:
LOAD CSV WITH HEADERS FROM "…/././…. CLT_NODES_LabelA.csv" AS csvline
MERGE (new_node_A:Skewer: {my_node_id: csvline.node_unique_number, property1
: csvline.property1})
// only my_node_id and property1 values will be taken into account! no
labels, no other properties are taken care of
// AFAIR we do not need `ON CREATE SET` here, do you really care is it a
new node or it was created earlier?
SET
new_node_A : LabelA,
new_node_A.Property2 = csvline.Property2,
new_node_A.Property3 = csvline.Property3,
new_node_A.Property4 = csvline.Property4;
LOAD CSV WITH HEADERS FROM "…/././…. CLT_NODES_LabelJ.csv" AS csvline
MERGE (new_node_J:Skewer: {my_node_id: csvline.node_unique_number, property1
: csvline.property1})
// only my_node_id and property1 values will be taken into account! no
labels, no other properties are taken care of
// AFAIR we do not need `ON CREATE SET` here, do you really care is it a
new node or it was created earlier?
SET
new_node_J : LabelJ,
new_node_J.Property2 = csvline.Property2,
new_node_J.Property3 = csvline.Property3,
new_node_J.Property4 = csvline.Property4;
What you get if doing things this way:
1. When doing LabelA .csv you will create whatever uniquely numbered
nodes were not already in the database, fill their properties (or maybe
overwrite them?) and label the node (be it new or existing one) with LabelA
- no matter what other labels did node (possibly) have,
2. When doing LabelJ .csv you *again *will create whatever uniquely
numbered nodes were not already in the database, *again* either fill or
overwrite propertiers, and *again* label the node (be it new or existing
one) with LabelJ - no matter what other labels did node (possibly) have,
3. so if you created some node with first file and labeled it LabelA, if
the same unique *my_node_id *occur both in first and second files, your
node will get 2 labels LabelA and LabelJ.
> Q5: Since I think of my data in terms of the two classes of nodes in my
> Data model …[CLT_SOURCE —> CLT_TARGET ; CLT_TARGET —> CLT_SOURCE], after
> loading the nodes, how then I get two classes of nodes?
>
Make them 2 labels: CLTSource and CLTTarget respectively.
> Q6: Is there a step missing that explains how the code below got to have a
> “source_node” and a “dest_node” that appears to correspond to my CLT_SOURCE
> and CLT_TARGET nodes?
>
// suppose we coded relationships as 2 my_node_id's of nodes
LOAD CSV FROM "...somewhere..." AS csvline
MATCH (s:CLTSource:Skewer {my_node_id: TOINT(csvline[0)})
USING INDEX s:Skewer(my_node_id)
WITH s
MATCH (t:CLTTarget:Skewer {my_node_id: TOINT(csvline[1)})
USING INDEX t:Skewer(my_node_id)
MERGE (s)-[r:MY_RELATIONSHIP_TYPE]->(t)
SET
r.prop1 = 'smth';
4. Now when you are done with nodes and start doing LOAD CSV for
>> relationships, you may give the MATCH statement, which looks up your pair
>> of nodes, a hint for fast lookup, like
>>
>> LOAD CSV ...from somewhere... AS csvline
>> MATCH (source_node:Skewer {my_node_id: ToInt(csvline[0]}), (dest_node:
>> Skewer {my_node_id: ToInt(csvline[1]})
>> CREATE (source_node)-[r:MY_REL_TYPE {rel_prop_00: csvline[2], ...,
>> rel_prop_NN: csvline[ZZ]}]->(dest_node);
>>
>>
> Q6: This LOAD CSV command (line 1) looks into the separate REL.csv file
> you mentioned first right?
>
Yep
> Q7: csvline is some sort of temp file that is a series of lines of the cvs
> file?
>
This is a variable - collection which is filled by column values of .csv
line by line. You can use it either as an array referring fields by their
index (my preferred way) - or, if you use `WITH HEADERS` mode, you can use
it as a keyed map.
See http://neo4j.com/docs/2.1.6/cypherdoc-importing-csv-files-with-cypher.html
Q8: Do you imply in line 2 that the REL.csv file has headers that include
> source_node, dest_node ?
>
No I don't use headers so I refer to csvline fields by their index
("collection mode")
> Q9: While I see how Skewer is a label, how is my_node_id a property
> (line 2) ?
>
Because it IS a property of a node, and you build constraint & index *on
this exact property* inside the scope of a label :Skewer
> Q10: How does my_node_id relate to either ToInt(csvline[0]} or
> ToInt(csvline[1]} (line 2) ?
>
For .csv with relationships, csvline[0] is a value of *my_node_id *property
of the *source* node, csvline[1] is a value of *my_node_id *property of the
*target* node, and TOINT() type conversion is used because my personal
preference is to use integers for ids.
> Is it that ToInt(csvline[0]} refers to the a line of the REL.csv file?
>
> Does csvline[0] refer to a column in REL.csv as do csvline[2] and
> csvline[ZZ] (line 3) ?
>
I think you can combine import of multiple .CSV files in a single LOAD CSV
statement but I didn't ever try this mode.
WBR,
Andrii
> Adding *`:Skewer` *label in MATCH will tell Cypher to (implicitly) use
>> your index on *my_node_id* which was created when you created your
>> constraint. Or you may try to explicitly give it a hint to use the index,
>> with USING INDEX... clause after MATCH before CREATE. Btw some earlier
>> versions of Neo4j refused to use index in LOAD CSV for some reason, I hope
>> this problem is gone with 2.1.5.
>>
>> OK
>
>
>> 5. While importing, be careful to *explicitly specify type conversions
>> for each property which is not a string*. I have seen numerous occasions
>> when people missed ToInt(csvline[i]) or ToFloat(csvline[j]) - and Cypher
>> silently stored their (supposed) numerics as strings. It's Ok, dude, you
>> say it :) This led to confusion afterwards when say numerical comparisons
>> doesn't MATCH and so on (though it's easy to correct with a single Cypher
>> command, but anyway).
>>
>> Think I did that re. type conversion. Only applies to properties for my
> data.
>
> Sorry for so many questions. I am really interested in figuring this out!
>
> Thanks loads,
> Jose
>
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
