[orientdb] Out of memory during import via REST
Hi, in our project, we want to integrate public genomic databases and link them together with orient. we are trying to import 50+ mio. documents into one class and set links to a second class in an graph db (version 1.5.0). We are attempting that from that with the REST-Interface and POST-Commands. During really big imports we are seeing some memory leak and the orient-server stops the import with an OutOfMemoryError: 2013-12-23 20:35:41:105 FINE Free memory is low 346,92Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:35:58:940 FINE [OMMapManagerNew] flushing pages in memory... [OMMapManagerNew] 2013-12-23 20:35:58:941 FINE [OMMapManagerNew] flushed 2/360 blocks [OMMapManagerNew] 2013-12-23 20:36:16:789 FINE Free memory is low 346,86Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:36:34:630 FINE [OMMapManagerNew] flushing pages in memory... [OMMapManagerNew] 2013-12-23 20:36:34:631 FINE Free memory is low 346,83Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:36:34:631 FINE [OMMapManagerNew] flushed 2/360 blocks [OMMapManagerNew] 2013-12-23 20:36:52:475 FINE Free memory is low 347,28Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:38:10:125 FINE [OMMapManagerNew] flushing pages in memory... [OMMapManagerNew]Sent run-time exception to the client /192.xxx.x.xx:x: java.lang.OutOfMemoryError: GC overhead limit exceeded GC overhead limit exceeded - com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.parse(OStringSerializerHelper.java:196) - com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.smartSplit(OStringSerializerHelper.java:171) - com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.smartSplit(OStringSerializerHelper.java:161) - com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.fromString(ORecordSerializerSchemaAware2CSV.java:386) - com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.fromStream(ORecordSerializerStringAbstract.java:79) - com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.deserializeFields(ORecordSchemaAwareAbstract.java:150) - com.orientechnologies.orient.core.record.impl.ODocument.deserializeFields(ODocument.java:1386) - com.orientechnologies.orient.core.record.impl.ODocument.checkForFields(ODocument.java:1276) - com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:712) - com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:623) - com.orientechnologies.orient.core.sql.OCommandExecutorSQLCreateLink.execute(OCommandExecutorSQLCreateLink.java:283) - com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:57) - com.orientechnologies.orient.core.storage.OStorageEmbedded.executeCommand(OStorageEmbedded.java:99) - com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:88) - com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:59) - com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1141) - com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:302) - com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:123) - com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:45) Sometimes we encounter the same problem after big import while the execution of the CREATE LINK command. Has anybody encountered this problem before and could stop orient from leaking memory? With regards, Konrad Lötzsch. -- --- You received this message because you are subscribed to the Google Groups OrientDB group. To unsubscribe from this group and stop receiving emails from it, send an email to orient-database+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [orientdb] Out of memory during import via REST
Hi Konrad, could you try the same using OrientDB 1.6 with plocal? Lvc@ On 2 January 2014 12:39, Konrad Lötzsch konrad.loetz...@gmail.com wrote: Hi, in our project, we want to integrate public genomic databases and link them together with orient. we are trying to import 50+ mio. documents into one class and set links to a second class in an graph db (version 1.5.0). We are attempting that from that with the REST-Interface and POST-Commands. During really big imports we are seeing some memory leak and the orient-server stops the import with an OutOfMemoryError: 2013-12-23 20:35:41:105 FINE Free memory is low 346,92Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:35:58:940 FINE [OMMapManagerNew] flushing pages in memory... [OMMapManagerNew] 2013-12-23 20:35:58:941 FINE [OMMapManagerNew] flushed 2/360 blocks [OMMapManagerNew] 2013-12-23 20:36:16:789 FINE Free memory is low 346,86Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:36:34:630 FINE [OMMapManagerNew] flushing pages in memory... [OMMapManagerNew] 2013-12-23 20:36:34:631 FINE Free memory is low 346,83Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:36:34:631 FINE [OMMapManagerNew] flushed 2/360 blocks [OMMapManagerNew] 2013-12-23 20:36:52:475 FINE Free memory is low 347,28Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:38:10:125 FINE [OMMapManagerNew] flushing pages in memory... [OMMapManagerNew]Sent run-time exception to the client /192.xxx.x.xx:x: java.lang.OutOfMemoryError: GC overhead limit exceeded GC overhead limit exceeded - com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.parse(OStringSerializerHelper.java:196) - com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.smartSplit(OStringSerializerHelper.java:171) - com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.smartSplit(OStringSerializerHelper.java:161) - com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.fromString(ORecordSerializerSchemaAware2CSV.java:386) - com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.fromStream(ORecordSerializerStringAbstract.java:79) - com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.deserializeFields(ORecordSchemaAwareAbstract.java:150) - com.orientechnologies.orient.core.record.impl.ODocument.deserializeFields(ODocument.java:1386) - com.orientechnologies.orient.core.record.impl.ODocument.checkForFields(ODocument.java:1276) - com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:712) - com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:623) - com.orientechnologies.orient.core.sql.OCommandExecutorSQLCreateLink.execute(OCommandExecutorSQLCreateLink.java:283) - com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:57) - com.orientechnologies.orient.core.storage.OStorageEmbedded.executeCommand(OStorageEmbedded.java:99) - com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:88) - com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:59) - com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1141) - com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:302) - com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:123) - com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:45) Sometimes we encounter the same problem after big import while the execution of the CREATE LINK command. Has anybody encountered this problem before and could stop orient from leaking memory? With regards, Konrad Lötzsch. -- --- You received this message because you are subscribed to the Google Groups OrientDB group. To unsubscribe from this group and stop receiving emails from it, send an email to orient-database+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- --- You received this message because you are subscribed to the Google Groups OrientDB group. To unsubscribe from this group and stop receiving emails from it, send an email to orient-database+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [orientdb] Out of memory during import via REST
Hi, You use local storage , this issue does not exist in plocal storage, could you use 1.6.3 version with *plocal *storage and do import once again. You can find information about plocal storage here https://github.com/orientechnologies/orientdb/wiki/plocal-storage-components On Thu, Jan 2, 2014 at 1:39 PM, Konrad Lötzsch konrad.loetz...@gmail.comwrote: Hi, in our project, we want to integrate public genomic databases and link them together with orient. we are trying to import 50+ mio. documents into one class and set links to a second class in an graph db (version 1.5.0). We are attempting that from that with the REST-Interface and POST-Commands. During really big imports we are seeing some memory leak and the orient-server stops the import with an OutOfMemoryError: 2013-12-23 20:35:41:105 FINE Free memory is low 346,92Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:35:58:940 FINE [OMMapManagerNew] flushing pages in memory... [OMMapManagerNew] 2013-12-23 20:35:58:941 FINE [OMMapManagerNew] flushed 2/360 blocks [OMMapManagerNew] 2013-12-23 20:36:16:789 FINE Free memory is low 346,86Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:36:34:630 FINE [OMMapManagerNew] flushing pages in memory... [OMMapManagerNew] 2013-12-23 20:36:34:631 FINE Free memory is low 346,83Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:36:34:631 FINE [OMMapManagerNew] flushed 2/360 blocks [OMMapManagerNew] 2013-12-23 20:36:52:475 FINE Free memory is low 347,28Mb of 9,39Gb (3%), calling listeners to free memory... [OMemoryWatchDog] 2013-12-23 20:38:10:125 FINE [OMMapManagerNew] flushing pages in memory... [OMMapManagerNew]Sent run-time exception to the client /192.xxx.x.xx:x: java.lang.OutOfMemoryError: GC overhead limit exceeded GC overhead limit exceeded - com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.parse(OStringSerializerHelper.java:196) - com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.smartSplit(OStringSerializerHelper.java:171) - com.orientechnologies.orient.core.serialization.serializer.OStringSerializerHelper.smartSplit(OStringSerializerHelper.java:161) - com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.fromString(ORecordSerializerSchemaAware2CSV.java:386) - com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.fromStream(ORecordSerializerStringAbstract.java:79) - com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.deserializeFields(ORecordSchemaAwareAbstract.java:150) - com.orientechnologies.orient.core.record.impl.ODocument.deserializeFields(ODocument.java:1386) - com.orientechnologies.orient.core.record.impl.ODocument.checkForFields(ODocument.java:1276) - com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:712) - com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:623) - com.orientechnologies.orient.core.sql.OCommandExecutorSQLCreateLink.execute(OCommandExecutorSQLCreateLink.java:283) - com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:57) - com.orientechnologies.orient.core.storage.OStorageEmbedded.executeCommand(OStorageEmbedded.java:99) - com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:88) - com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:59) - com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1141) - com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:302) - com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:123) - com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:45) Sometimes we encounter the same problem after big import while the execution of the CREATE LINK command. Has anybody encountered this problem before and could stop orient from leaking memory? With regards, Konrad Lötzsch. -- --- You received this message because you are subscribed to the Google Groups OrientDB group. To unsubscribe from this group and stop receiving emails from it, send an email to orient-database+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- Best regards, Andrey Lomakin. Orient Technologies the Company behind OrientDB -- --- You received this message because you are subscribed to the Google Groups OrientDB group. To unsubscribe from this group and stop receiving emails from it, send an email to
