I set the "tx" and "batchCommit" parameters telling the etl-process to commit every 10.000 records, and now it works. Basically, I was trying to import +500.000 record from a RDBMS to Orient performing the commit only at the end of the process. How can I calculate "batchCommit" parameter value in order to have the best performance?
Giulio Il giorno giovedì 17 luglio 2014 13:33:01 UTC+2, Giulio Grosso ha scritto: > > The oracle db version is 10g (10.2.0.5.0) and I'm using the last release > of jdbc driver available and downloadable from oracle official website! :/ > I'll let you know about old-gen objects... > > Giulio > > Il giorno giovedì 17 luglio 2014 12:31:12 UTC+2, Lvc@ ha scritto: >> >> You can setup the Heap with -Xmx setting. ETL is not heap consuming, so >> it's pretty weird. Can you profile where the old-gen objects remain >> allocated? I'm thinking in a bug of JDBC driver. >> >> Lvc@ >> >> >> >> On 17 July 2014 12:24, Giulio Grosso <[email protected]> wrote: >> >>> Yes, using the etl-tool with default configuration I got an OOM >>> Exception: GC overhead limit exceeded. >>> The etl sample process that I'm trying to execute consists on migrating >>> data from an Oracle db table (about 500.000 records on 15 columns) to an >>> orientdb instance (v. 1.7.5). >>> >>> So do I have to work on JVM and orientdb memory configuration? >>> >>> Giulio >>> >>> Il giorno giovedì 17 luglio 2014 12:00:03 UTC+2, Lvc@ ha scritto: >>>> >>>> Hi Giulio, >>>> You can use the same JVM and OrientDB settings to limit memory. Do you >>>> have any OOM exception? >>>> >>>> Lvc@ >>>> >>>> >>>> On 17 July 2014 11:04, Giulio Grosso <[email protected]> wrote: >>>> >>>>> Hi again, >>>>> how can I limit the memory usage during the etl process? >>>>> Have I to work on database configuration or on etl-tool one? >>>>> >>>>> Thanks in advance. >>>>> >>>>> Giulio >>>>> >>>>> Il giorno mercoledì 16 luglio 2014 15:38:02 UTC+2, Giulio Grosso ha >>>>> scritto: >>>>> >>>>>> Hi everybody, >>>>>> I'm trying to use new oritentdb-etl tool. >>>>>> I have correctly configured the environment (orientdb community >>>>>> version 1.7.5) following all the steps here >>>>>> <https://github.com/orientechnologies/orientdb-etl/wiki>. >>>>>> >>>>>> Unfortunately I get the following exception: >>>>>> >>>>>> Exception in thread "main" com.orientechnologies.orient.c >>>>>> ore.exception.OSerializationException: Error on unmarshalling J >>>>>> SON content for record: config: { >>>>>> verbose: true, >>>>>> }, >>>>>> extractor : { >>>>>> jdbc: { driver: "com.oracle.jdbc.Driver", >>>>>> url: "jdbc:oracle:thin:@//xx.xxx.xxx.xx:xxxx/xxx", >>>>>> userName: "xxxxxxxx", >>>>>> userPassword: "xxxxxxxx", >>>>>> query: "select * from People" } >>>>>> }, >>>>>> transformers : [ >>>>>> { vertex: { "class": "People"} } >>>>>> ], >>>>>> loader : { >>>>>> orientdb: { >>>>>> dbURL: "plocal:/temp/databases/etl-test-xxx", >>>>>> dbAutoCreate: true >>>>>> } >>>>>> } >>>>>> at com.orientechnologies.orient.c >>>>>> ore.serialization.serializer.record.string.ORecordSerializerJSON. >>>>>> fromString(ORe >>>>>> cordSerializerJSON.java:293) >>>>>> at com.orientechnologies.orient.core.record.ORecordAbstract. >>>>>> fromJSON(ORecordAbstract.java:202) >>>>>> at com.orientechnologies.orient.etl.OETLProcessor.main( >>>>>> OETLProcessor.java:169) >>>>>> Caused by: com.orientechnologies.orient.core.exception. >>>>>> OSerializationException: Error on unmarshalling JSON content: wro >>>>>> ng format. Use <field> : <value> >>>>>> at com.orientechnologies.orient.c >>>>>> ore.serialization.serializer.record.string.ORecordSerializerJSON. >>>>>> fromString(ORe >>>>>> cordSerializerJSON.java:165) >>>>>> at com.orientechnologies.orient.c >>>>>> ore.serialization.serializer.record.string.ORecordSerializerJSON. >>>>>> getValueAsReco >>>>>> rd(ORecordSerializerJSON.java:609) >>>>>> at com.orientechnologies.orient.c >>>>>> ore.serialization.serializer.record.string.ORecordSerializerJSON. >>>>>> getValueAsObje >>>>>> ctOrMap(ORecordSerializerJSON.java:579) >>>>>> at com.orientechnologies.orient.c >>>>>> ore.serialization.serializer.record.string.ORecordSerializerJSON. >>>>>> getValue(OReco >>>>>> rdSerializerJSON.java:412) >>>>>> at com.orientechnologies.orient.c >>>>>> ore.serialization.serializer.record.string.ORecordSerializerJSON. >>>>>> fromString(ORe >>>>>> cordSerializerJSON.java:242) >>>>>> ... 2 more >>>>>> >>>>>> Json config file seems to be ok, where's the error? >>>>>> >>>>>> Thanks in advance. >>>>>> >>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "OrientDB" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "OrientDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
