Hi Guys, I've just fixed an issue on edge creation. Can you pull the project and rebuild it? Current version of ETL module is 1.1-SNAPSHOT.
Lvc@ On 23 September 2014 11:48, Olakunle Olaniyi <[email protected]> wrote: > ok, I tried it again, i think it only creates an edge from alien to person > where there was no match and it had to create that vertex. A sample result > set is as below. > Any ideas what might be wrong ? > > > orientdb {db=test}> select from alien > > ----+------+----+----------+-------- > # |@RID |aid |aname |in_hates > ----+------+----+----------+-------- > 0 |#12:0 |1 |Alien - 1 |null > 1 |#12:1 |2 |Alien - 2 |null > 2 |#12:2 |3 |Alien - 3 |null > 3 |#12:3 |4 |Alien - 4 |null > 4 |#12:4 |5 |Alien - 5 |null > 5 |#12:5 |6 |Alien - 6 |null > 6 |#12:6 |7 |Alien - 7 |null > 7 |#12:7 |8 |Alien - 8 |null > 8 |#12:8 |9 |Alien - 9 |null > 9 |#12:9 |10 |Alien - 10|null > 10 |#12:10|11 |Alien - 11|null > 11 |#12:11|12 |Alien - 12|null > 12 |#12:12|13 |Alien - 13|null > 13 |#12:13|14 |Alien - 14|null > 14 |#12:14|15 |Alien - 15|null > 15 |#12:15|16 |Alien - 16|null > 16 |#12:16|17 |Alien - 17|null > 17 |#12:17|18 |Alien - 18|null > 18 |#12:18|19 |Alien - 19|null > 19 |#12:19|20 |Alien - 20|null > 20 |#12:20|1 |Alien - 1 |null > 21 |#12:21|2 |Alien - 2 |null > 22 |#12:22|3 |Alien - 3 |null > 23 |#12:23|4 |Alien - 4 |null > 24 |#12:24|5 |Alien - 5 |null > 25 |#12:25|6 |Alien - 6 |null > 26 |#12:26|7 |Alien - 7 |null > 27 |#12:27|8 |Alien - 8 |null > 28 |#12:28|9 |Alien - 9 |null > 29 |#12:29|10 |Alien - 10|null > 30 |#12:30|11 |Alien - 11|null > 31 |#12:31|12 |Alien - 12|null > 32 |#12:32|13 |Alien - 13|null > 33 |#12:33|14 |Alien - 14|null > 34 |#12:34|15 |Alien - 15|null > 35 |#12:35|16 |Alien - 16|null > 36 |#12:36|17 |Alien - 17|null > 37 |#12:37|18 |Alien - 18|null > 38 |#12:38|19 |Alien - 19|null > 39 |#12:39|20 |Alien - 20|null > 40 |#12:40|22 |Alien - 2 |[size=1] > 41 |#12:41|23 |Alien - 3 |[size=1] > 42 |#12:42|24 |Alien - 4 |[size=1] > 43 |#12:43|25 |Alien - 5 |[size=1] > 44 |#12:44|26 |Alien - 6 |[size=1] > 45 |#12:45|27 |Alien - 7 |[size=1] > 46 |#12:46|28 |Alien - 8 |[size=1] > 47 |#12:47|29 |Alien - 9 |[size=1] > 48 |#12:48|30 |Alien - 10|[size=1] > 49 |#12:49|31 |Alien - 11|[size=1] > 50 |#12:50|32 |Alien - 12|[size=1] > 51 |#12:51|33 |Alien - 13|[size=1] > 52 |#12:52|34 |Alien - 14|[size=1] > 53 |#12:53|35 |Alien - 15|[size=1] > 54 |#12:54|36 |Alien - 16|[size=1] > 55 |#12:55|37 |Alien - 17|[size=1] > 56 |#12:56|38 |Alien - 18|[size=1] > 57 |#12:57|39 |Alien - 19|[size=1] > 58 |#12:58|40 |Alien - 20|[size=1] > ----+------+----+----------+-------- > > > > > orientdb {db=test}> select from person > > > ----+------+----+-----------+--------- > # |@RID |pid |pname |out_hates > ----+------+----+-----------+--------- > 0 |#11:0 |1 |Person - 1 |null > 1 |#11:1 |2 |Person - 2 |null > 2 |#11:2 |3 |Person - 3 |null > 3 |#11:3 |4 |Person - 4 |null > 4 |#11:4 |5 |Person - 5 |null > 5 |#11:5 |6 |Person - 6 |null > 6 |#11:6 |7 |Person - 7 |null > 7 |#11:7 |8 |Person - 8 |null > 8 |#11:8 |9 |Person - 9 |null > 9 |#11:9 |10 |Person - 10|null > 10 |#11:10|11 |Person - 11|null > 11 |#11:11|12 |Person - 12|null > 12 |#11:12|13 |Person - 13|null > 13 |#11:13|14 |Person - 14|null > 14 |#11:14|15 |Person - 15|null > 15 |#11:15|16 |Person - 16|null > 16 |#11:16|17 |Person - 17|null > 17 |#11:17|18 |Person - 18|null > 18 |#11:18|19 |Person - 19|null > 19 |#11:19|20 |Person - 20|null > 20 |#11:20|22 |null |[size=1] > 21 |#11:21|23 |null |[size=1] > 22 |#11:22|24 |null |[size=1] > 23 |#11:23|25 |null |[size=1] > 24 |#11:24|26 |null |[size=1] > 25 |#11:25|27 |null |[size=1] > 26 |#11:26|28 |null |[size=1] > 27 |#11:27|29 |null |[size=1] > 28 |#11:28|30 |null |[size=1] > 29 |#11:29|31 |null |[size=1] > 30 |#11:30|32 |null |[size=1] > 31 |#11:31|33 |null |[size=1] > 32 |#11:32|34 |null |[size=1] > 33 |#11:33|35 |null |[size=1] > 34 |#11:34|36 |null |[size=1] > 35 |#11:35|37 |null |[size=1] > 36 |#11:36|38 |null |[size=1] > 37 |#11:37|39 |null |[size=1] > 38 |#11:38|40 |null |[size=1] > ----+------+----+-----------+--------- > > > On Tue, Sep 23, 2014 at 6:00 AM, 'Curtis Mosters' via OrientDB < > [email protected]> wrote: > >> I tested ETL 2 months ago and there I had the same problem, that the ETL >> did not create any edges. >> >> Will test again when 2.0 is officially out. >> >> Am Montag, 22. September 2014 23:48:18 UTC+2 schrieb Olakunle Olaniyi: >>> >>> Hello , >>> I have 2 json files (person.json and alien.json) and they are as below : >>> >>> =======person.json================ >>> { >>> "config": { >>> "log": "debug" >>> }, >>> "extractor" : { >>> "jdbc": { "driver": "org.postgresql.Driver", >>> "url": "jdbc:postgresql://localhost/testdb", >>> "userName": "postgres", >>> "userPassword": "password1", >>> "query": "select * from person" } >>> }, >>> "transformers" : [ >>> { "vertex": { "class": "Person"} } >>> ], >>> "loader" : { >>> "orientdb": { >>> "dbURL": "plocal:/home/orientdb/db/databases/db_test", >>> dbUser: "admin", >>> dbPassword: "admin", >>> dbAutoDropIfExists: false, >>> dbAutoCreate: true, >>> tx: false, >>> wal: false, >>> batchCommit: 1000, >>> dbType: "graph", >>> indexes: [{class:"Person", fields:["pid:string"], >>> type:"UNIQUE_HASH_INDEX" }] >>> } >>> } >>> } >>> >>> >>> >>> >>> >>> ======alien.json===================== >>> { >>> "config": { >>> "log": "debug" >>> }, >>> "extractor" : { >>> "jdbc": { "driver": "org.postgresql.Driver", >>> "url": "jdbc:postgresql://localhost/testdb", >>> "userName": "postgres", >>> "userPassword": "password1", >>> "query": "select * from alien" } >>> }, >>> "transformers" : [ >>> { "vertex": { "class": "Alien"} }, >>> { "edge": { "class": "hasLink", "direction" : "in", >>> "joinFieldName": "aid", >>> "lookup":"Person.pid", "unresolvedLinkAction":"CREATE"} } >>> ], >>> "loader" : { >>> "orientdb": { >>> "dbURL": "plocal:/home/orientdb/db/databases/db_test", >>> dbUser: "admin", >>> dbPassword: "admin", >>> dbAutoDropIfExists: false, >>> dbAutoCreate: true, >>> tx: false, >>> wal: false, >>> batchCommit: 1000, >>> dbType: "graph", >>> indexes: [{class:"Alien", fields:["aid:string"], >>> type:"UNIQUE_HASH_INDEX" }] >>> } >>> } >>> } >>> >>> On Mon, Sep 22, 2014 at 7:13 PM, Luca Garulli <[email protected]> wrote: >>> >>>> Hi, >>>> Seems there is no Loader configured. Could you post also your JSON file? >>>> >>>> Lvc@ >>>> >>>> On 22 September 2014 18:42, Olakunle Olaniyi <[email protected]> wrote: >>>> >>>>> Hi Luca, >>>>> I'd like to start by saying thanks for the explanation on this so far. >>>>> I am running orientdb 2.0-M1 and this is not working for me as well. It >>>>> shows the joinvalue alright and they are correct but the edges are not be >>>>> created. Have an explanation why this is so? Your response will be much >>>>> appreciated. >>>>> >>>>> Olakay >>>>> >>>>> P.S : the log could be see below : >>>>> >>>>> >>>>> OrientDB etl v.2.0-M1 (build @BUILD@) www.orientechnologies.com >>>>> BEGIN ETL PROCESSOR >>>>> [1:vertex] DEBUG Transformer input: {aid:1,name:Qualitrail - 1} >>>>> [1:vertex] DEBUG Transformer output: v(Alien)[#12:10] >>>>> [1:edge] DEBUG Transformer input: v(Alien)[#12:10] >>>>> [1:edge] DEBUG WARNING: index Person.mid not found. Lookups could be >>>>> really slow >>>>> [1:edge] DEBUG joinValue=1, lookupResult=Person#11:0{mid:1,name:Qualitrail >>>>> - 1,in_hasLink:[size=1]} v2 >>>>> [1:edge] DEBUG Transformer output: v(Alien)[#12:10] >>>>> [3:vertex] DEBUG Transformer input: {aid:2,name:Qualitrail - 2} >>>>> [3:vertex] DEBUG Transformer output: v(Alien)[#12:11] >>>>> [3:edge] DEBUG Transformer input: v(Alien)[#12:11] >>>>> [3:edge] DEBUG joinValue=2, lookupResult=Person#11:1{mid:2,name:Qualitrail >>>>> - 2} v1 >>>>> [3:edge] DEBUG Transformer output: v(Alien)[#12:11] >>>>> [5:vertex] DEBUG Transformer input: {aid:3,name:Qualitrail - 3} >>>>> [5:vertex] DEBUG Transformer output: v(Alien)[#12:12] >>>>> [5:edge] DEBUG Transformer input: v(Alien)[#12:12] >>>>> [5:edge] DEBUG joinValue=3, lookupResult=Person#11:2{mid:3,name:Qualitrail >>>>> - 3} v1 >>>>> [5:edge] DEBUG Transformer output: v(Alien)[#12:12] >>>>> [7:vertex] DEBUG Transformer input: {aid:4,name:Qualitrail - 4} >>>>> [7:vertex] DEBUG Transformer output: v(Alien)[#12:13] >>>>> [7:edge] DEBUG Transformer input: v(Alien)[#12:13] >>>>> [7:edge] DEBUG joinValue=4, lookupResult=Person#11:3{mid:4,name:Qualitrail >>>>> - 4} v1 >>>>> [7:edge] DEBUG Transformer output: v(Alien)[#12:13] >>>>> [9:vertex] DEBUG Transformer input: {aid:5,name:Qualitrail - 5} >>>>> [9:vertex] DEBUG Transformer output: v(Alien)[#12:14] >>>>> [9:edge] DEBUG Transformer input: v(Alien)[#12:14] >>>>> [9:edge] DEBUG joinValue=5, lookupResult=Person#11:4{mid:5,name:Qualitrail >>>>> - 5} v1 >>>>> [9:edge] DEBUG Transformer output: v(Alien)[#12:14] >>>>> [11:vertex] DEBUG Transformer input: {aid:6,name:Qualitrail - 6} >>>>> [11:vertex] DEBUG Transformer output: v(Alien)[#12:15] >>>>> [11:edge] DEBUG Transformer input: v(Alien)[#12:15] >>>>> [11:edge] DEBUG joinValue=6, >>>>> lookupResult=Person#11:5{mid:6,name:Qualitrail >>>>> - 6} v1 >>>>> [11:edge] DEBUG Transformer output: v(Alien)[#12:15] >>>>> [13:vertex] DEBUG Transformer input: {aid:7,name:Qualitrail - 7} >>>>> [13:vertex] DEBUG Transformer output: v(Alien)[#12:16] >>>>> [13:edge] DEBUG Transformer input: v(Alien)[#12:16] >>>>> [13:edge] DEBUG joinValue=7, >>>>> lookupResult=Person#11:6{mid:7,name:Qualitrail >>>>> - 7} v1 >>>>> [13:edge] DEBUG Transformer output: v(Alien)[#12:16] >>>>> [15:vertex] DEBUG Transformer input: {aid:8,name:Qualitrail - 8} >>>>> [15:vertex] DEBUG Transformer output: v(Alien)[#12:17] >>>>> [15:edge] DEBUG Transformer input: v(Alien)[#12:17] >>>>> [15:edge] DEBUG joinValue=8, >>>>> lookupResult=Person#11:7{mid:8,name:Qualitrail >>>>> - 8} v1 >>>>> [15:edge] DEBUG Transformer output: v(Alien)[#12:17] >>>>> [17:vertex] DEBUG Transformer input: {aid:9,name:Qualitrail - 9} >>>>> [17:vertex] DEBUG Transformer output: v(Alien)[#12:18] >>>>> [17:edge] DEBUG Transformer input: v(Alien)[#12:18] >>>>> [17:edge] DEBUG joinValue=9, >>>>> lookupResult=Person#11:8{mid:9,name:Qualitrail >>>>> - 9} v1 >>>>> [17:edge] DEBUG Transformer output: v(Alien)[#12:18] >>>>> [19:vertex] DEBUG Transformer input: {aid:10,name:Qualitrail - 10} >>>>> [19:vertex] DEBUG Transformer output: v(Alien)[#12:19] >>>>> [19:edge] DEBUG Transformer input: v(Alien)[#12:19] >>>>> [19:edge] DEBUG joinValue=10, >>>>> lookupResult=Person#11:9{mid:10,name:Qualitrail >>>>> - 10} v1 >>>>> [19:edge] DEBUG Transformer output: v(Alien)[#12:19] >>>>> END ETL PROCESSOR >>>>> >>>>> >>>>> On Thursday, July 3, 2014 6:06:45 PM UTC+2, Lvc@ wrote: >>>>>> >>>>>> Hi guys, >>>>>> I'm glad to announce a new project under the Orient Technologies >>>>>> umbrella: *OrientDB-ETL*! >>>>>> >>>>>> This is the URL of WiKi: >>>>>> *https://github.com/orientechnologies/orientdb-etl/wiki >>>>>> <https://github.com/orientechnologies/orientdb-etl/wiki>* >>>>>> >>>>>> Thanks to OrientDB-ETL project, moving data from/to OrientDB is super >>>>>> fast and require no more code writing: only a JSON file. >>>>>> >>>>>> So far these are the available components: >>>>>> >>>>>> - *Extractors*: >>>>>> - File >>>>>> - *Transformers*: >>>>>> - CSV, to convert rows in documents >>>>>> - Link, to transform JOIN in links >>>>>> - Merge, to merge the parsed document with an existent >>>>>> - Vertex, to convert a document in Vertex >>>>>> - Edge, to create edges >>>>>> - Skip, to skip records based on expressions >>>>>> - Code, to execute arbitrary Javascript code >>>>>> - *Loaders*: >>>>>> - OrientDB, to save into a OrientDB database >>>>>> >>>>>> OrientDB-ETL module is in beta status, and will be final with >>>>>> OrientDB v2.0 (September). To use it against OrientDB 2.0-SNAPSHOT follow >>>>>> the "Installation" instructions. >>>>>> >>>>>> Below an example to import *DBPedia* *csv* files ( >>>>>> http://wiki.dbpedia.org/DBpediaAsTables): >>>>>> >>>>>> { >>>>>> config: { >>>>>> verbose: true, >>>>>> fileDirectory: "/temp/databases/dbpedia_csv/", >>>>>> fileName: "Person.csv.gz" >>>>>> }, >>>>>> begin: [ >>>>>> { let: { name: "$filePath", value: "$fileDirectory.append( >>>>>> $fileName )"} }, >>>>>> { let: { name: "$className", value: "$fileName.substring( 0, >>>>>> $fileName.indexOf('.') )"} } >>>>>> ], >>>>>> extractor : { >>>>>> line: { path: "$filePath", lock : true } >>>>>> }, >>>>>> transformers : [ >>>>>> { csv: { separator: ",", nullValue: "NULL", skipFrom: 1, skipTo: 3 >>>>>> } }, >>>>>> { merge: { joinFieldName:"URI", lookup:"V.URI" } }, >>>>>> { vertex: { class: "$className"} } >>>>>> ], >>>>>> loader : { >>>>>> orientdb: { >>>>>> dbURL: "plocal:/temp/databases/dbpedia", >>>>>> dbUser: "admin", >>>>>> dbPassword: "admin", >>>>>> dbAutoCreate: true, >>>>>> tx: false, >>>>>> batchCommit: 1000, >>>>>> dbType: "graph", >>>>>> indexes: [{class:"V", fields:["URI:string"], type:"UNIQUE" }] >>>>>> } >>>>>> } >>>>>> } >>>>>> >>>>>> For more information: https://github.com/orientechnologies/orientdb- >>>>>> etl/wiki/Import-from-DBPedia. >>>>>> >>>>>> Lvc@ >>>>>> >>>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "OrientDB" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "OrientDB" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>> topic/orient-database/Xp38b55AbpU/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Olakunle >>> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "OrientDB" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/orient-database/Xp38b55AbpU/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Olakunle > > -- > > --- > You received this message because you are subscribed to the Google Groups > "OrientDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
