Well, the "skipDuplicates" sort of works - a duplicate Id was encountered,
but rather than simply being skipped, I received an ORecordDuplicated
exception, which brought the entire data load to a screeching halt. This is
perhaps more aggressive behavior than just a simple "skip".
Here is the stack trace:
E:\SearchPlatform\orientdb-community-2.0.3\bin>oetl
..\Loader\Country_StateProvince.control.json
OrientDB etl v.2.0.3 (build @BUILD@) www.orientechnologies.com
BEGIN ETL PROCESSOR
Error in Pipeline execution:
com.orientechnologies.orient.core.storage.ORecordDuplicatedException:
Cannot index record
Country_StateProvince{Id:58AC0F54-011C-E411-8534-00155D016D32,Name:MASSACHUSETTS,Code:MA,DateModified:Mon
Mar 09 10:30:30 EDT 2015,ParentId:64AB0F54-011C-E411-8534-00155D016D32}:
found duplicated key '58AC0F54-011C-E411-8534-00155D016D32' in index
'Base.Id' previously assigned to the record #32:4 RID=#32:4
com.orientechnologies.orient.core.storage.ORecordDuplicatedException:
Cannot index record
Country_StateProvince{Id:58AC0F54-011C-E411-8534-00155D016D32,Name:MASSACHUSETTS,Code:MA,DateModified:Mon
Mar 09 10:30:30 EDT 2015,ParentId:64AB0F54-011C-E411-8534-00155D016D32}:
found duplicated key '58AC0F54-011C-E411-8534-00155D016D32' in index
'Base.Id' previously assigned to the record #32:4 RID=#32:4
at
com.orientechnologies.orient.core.index.OIndexTxAwareOneValue.checkEntry(OIndexTxAwareOneValue.java:221)
at
com.orientechnologies.orient.core.index.OClassIndexManager.checkIndexedPropertiesOnCreation(OClassIndexManager.java:322)
at
com.orientechnologies.orient.core.index.OClassIndexManager.checkIndexes(OClassIndexManager.java:581)
at
com.orientechnologies.orient.core.index.OClassIndexManager.onRecordBeforeCreate(OClassIndexManager.java:423)
at
com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:218)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.callbackHooks(ODatabaseDocumentTx.java:966)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:1686)
at
com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:94)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2277)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2111)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:117)
at
com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.createRecord(OBinaryNetworkProtocolAbstract.java:353)
at
com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.createRecord(ONetworkProtocolBinary.java:1373)
at
com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:350)
at
com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:220)
at
com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69)
ETL process halted:
com.orientechnologies.orient.etl.OETLProcessHaltedException:
com.orientechnologies.orient.core.storage.ORecordDuplicatedException:
Cannot index record
Country_StateProvince{Id:58AC0F54-011C-E411-8534-00155D016D32,Name:MASSACHUSETTS,Code:MA,DateModified:Mon
Mar 09 10:30:30 EDT 2015,ParentId:64AB0F54-011C-E411-8534-00155D016D32}:
found duplicated key '58AC0F54-011C-E411-8534-00155D016D32' in index
'Base.Id' previously assigned to the record #32:4 RID=#32:4
Is this a known issue?
Thanks,
Patrick
On Wednesday, March 11, 2015 at 4:20:59 AM UTC-4, Lvc@ wrote:
>
> Hi Patrick,
> See skipDuplicates option in
> http://www.orientechnologies.com/docs/last/orientdb-etl.wiki/Transformer.html#vertex
>
> Lvc@
>
>
> On 10 March 2015 at 23:02, Patrick Hoeffel <[email protected]
> <javascript:>> wrote:
>
>> I'm doing batch loading of data into my OrientDB database. On initial
>> load the data is all new, so no issues, but if I need to update data later,
>> in bulk, is there a way to use the OETL tool to specify "IF EXISTS (PK)
>> UPDATE, ELSE INSERT;"
>> I've looked through the documentation, but can't seem to find an answer.
>>
>> Thanks,
>>
>> Patrick
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "OrientDB" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
---
You received this message because you are subscribed to the Google Groups
"OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
