Hi Luca, Thanks for the response, but I'm afraid it had no effect, I still get this stacktrace :
2015-08-19 08:38:42:308 INFO [local_orient0_future_net_uk] received updated status local_orient0_future_net_uk.techradarMap=ONLINE [OHazelcastPlugin]Error on updating record #11:173 (cluster: com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster@7e9b0eac) java.lang.NullPointerException at com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiLocal.load(OSBTreeBonsaiLocal.java:455) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerShared.loadTree(OSBTreeCollectionManagerShared.java:95) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerAbstract.loadSBTree(OSBTreeCollectionManagerAbstract.java:98) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerProxy.loadSBTree(OSBTreeCollectionManagerProxy.java:52) at com.orientechnologies.orient.core.storage.impl.local.paginated.ORidBagUpdateSerializationOperation.loadTree(ORidBagUpdateSerializationOperation.java:73) at com.orientechnologies.orient.core.storage.impl.local.paginated.ORidBagUpdateSerializationOperation.execute(ORidBagUpdateSerializationOperation.java:54) at com.orientechnologies.orient.core.storage.impl.local.paginated.ORecordSerializationContext.executeOperations(ORecordSerializationContext.java:99) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:1567) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.updateRecord(OAbstractPaginatedStorage.java:728) at com.orientechnologies.orient.server.distributed.ODistributedStorage.updateRecord(ODistributedStorage.java:599) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:1757) at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:102) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2316) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2150) at com.orientechnologies.orient.server.distributed.task.OUpdateRecordTask.execute(OUpdateRecordTask.java:95) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.executeOnLocalNode(OHazelcastPlugin.java:753) at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.onMessage(ODistributedWorker.java:298) at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.run(ODistributedWorker.java:121) The Java command line (from ps) looks like this : java -server -Xmx512m -Djna.nosys=true -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9 -DridBag.embeddedToSbtreeBonsaiThreshold=2147483647 -Dstorage.diskCache.bufferSize=3072 -Djava.util.logging.config.file=/data/orientdb/config/orientdb-server-log.properties -Dorientdb.config.file=/etc/orientdb/server-config.xml -Dorientdb.www.path=/data/orientdb/www -Dorientdb.build.number=UNKNOWN@r; 2015-08-10 18:25:49+0000 -cp /data/orientdb/lib/orientdb-server-2.0.14.jar:/data/orientdb/lib/* -Ddistributed=true com.orientechnologies.orient.server.OServerMain I had to set 2147483647 as I get a Number Format Exception when setting it to Integer.MAX_VALUE. Do you have any other suggestions? Thanks, Rich On Tuesday, 18 August 2015 19:23:41 UTC+1, l.garulli wrote: > > Hi Rich, > > The problem with Tree Bonsai is that is not ready to run on distributed > architecture yet (we have an open issue on that). So you should set > "ridBag.embeddedToSbtreeBonsaiThreshold" to Integer.MAX_VALUE to avoid > using it. > > Best Regards, > > Luca Garulli > Founder & CEO > OrientDB <http://orientdb.com/> > > > On 18 August 2015 at 11:06, Richard Jones <[email protected] <javascript:> > > wrote: > >> Hello, >> >> Quite a long problem, apologies. >> >> I've got a two-node distribution setup, my database has data (imported >> through ETL offline and synched to the other node when ODB is started), but >> I need to import daily updates (deltas) to that data, and can't get any way >> to work. Here are the ways I've tried to import a delta of 273 changes >> (each of which will update multiple vertices and edges). The import is a >> bit messy, there are less than 100 changes that actually need to be made, >> with lots of duplicates and changes being made and undone sequentially, so >> it's quite understandable that there's a transactional problem, but I would >> hope there's a way to overcome it. >> >> Firstly, I tried an ETL import to a remote:localhost:2424/database URL >> (online), writeQuorum 2, tx true and batchCommit 1000 (defaults) - I get a >> writeQuorum failure in the ETL output, and the server log for the node that >> I'm not running ETL on has this : >> >> Error on updating record #11:390 (cluster: >> com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster@365e3fb7) >> java.lang.NullPointerException >> at >> com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiLocal.load(OSBTreeBonsaiLocal.java:455) >> at >> com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerShared.loadTree(OSBTreeCollectionManagerShared.java:95) >> at >> com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerAbstract.loadSBTree(OSBTreeCollectionManagerAbstract.java:98) >> at >> com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerProxy.loadSBTree(OSBTreeCollectionManagerProxy.java:52) >> at >> com.orientechnologies.orient.core.storage.impl.local.paginated.ORidBagUpdateSerializationOperation.loadTree(ORidBagUpdateSerializationOperation.java:73) >> at >> com.orientechnologies.orient.core.storage.impl.local.paginated.ORidBagUpdateSerializationOperation.execute(ORidBagUpdateSerializationOperation.java:54) >> at >> com.orientechnologies.orient.core.storage.impl.local.paginated.ORecordSerializationContext.executeOperations(ORecordSerializationContext.java:99) >> at >> com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:1567) >> at >> com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.updateRecord(OAbstractPaginatedStorage.java:728) >> at >> com.orientechnologies.orient.server.distributed.ODistributedStorage.updateRecord(ODistributedStorage.java:599) >> at >> com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:1757) >> at >> com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:102) >> at >> com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2316) >> at >> com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2150) >> at >> com.orientechnologies.orient.server.distributed.task.OUpdateRecordTask.execute(OUpdateRecordTask.java:95) >> at >> com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.executeOnLocalNode(OHazelcastPlugin.java:753) >> at >> com.orientechnologies.orient.server.hazelcast.ODistributedWorker.onMessage(ODistributedWorker.java:298) >> at >> com.orientechnologies.orient.server.hazelcast.ODistributedWorker.run(ODistributedWorker.java:121) >> >> Which appears to be trying to get a page out of cache. Any suggestions as >> to what might be the problem here? I was going to set the embedded/OSB >> threshold very high to force everything to be embedded edges, but according >> to the various warnings (e.g. >> http://orientdb.com/docs/last/Concurrency.html#concurrency-on-adding-edges) >> when in distributed mode the edges are always embedded so this has no >> effect. I have to wonder why the OSBTreeBonsai code is being executed, but >> I must misunderstand what it's being used for. Or maybe it's because I >> imported the initial dataset offline, so the edges were created as OSBTree >> before the database was somehow flagged as distributed? This may be the >> case because if I import the delta into an empty database (so there's been >> no offline ETL import), then I get a different exception >> (OWOWCache.java:865, see below). >> >> The end result of this import is that I have one vertex (11:390 mentioned >> in the error) with different versions (out by one) on each node, so the >> database is unusable and I don't know how to fix it. For each different >> method of importing the DB I reset to the offline-import before trying >> another way. >> >> I then tried to shut down one ODB server and perform an offline ETL of >> the delta. This works fine, but when I start up the server with the new >> data, it then receives a copy of the (old) data from the other node, >> presumably because the offline node is rejoining an active cluster and so >> needs to get new data? NB - there are no inserts/updates (or even selects) >> happening other then my ETL imports, so the node that was online has not >> received any changes. How can I make the node that receives the offline ETL >> import the 'master' so it syncs its database to the other node? >> >> One way I've managed to import the data is turn off one node, run the >> online ETL on the remaining node, then bring the other node back online. >> This causes the node that was offline to receive a copy of the updated >> database. But this isn't ideal, as it's difficult to orchestrate and >> automate, and it means I'll only have one DB node running (which will be >> busy) for the period of the import. >> >> If I try to import with writeQuorum 1, then the import completes but on >> the node I'm not running ETL on, the log contains amny errors like the >> stack trace above, and also this : >> >> Error on updating record #55:10 (cluster: >> com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster@6d66a58c) >> java.lang.NullPointerException >> at >> com.orientechnologies.orient.core.index.hashindex.local.cache.OWOWCache.fileNameById(OWOWCache.java:865) >> at >> com.orientechnologies.orient.core.index.hashindex.local.cache.OReadWriteDiskCache.fileNameById(OReadWriteDiskCache.java:236) >> at >> com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperation.fileNameById(OAtomicOperation.java:281) >> at >> com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerShared.loadTree(OSBTreeCollectionManagerShared.java:89) >> at >> com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerAbstract.loadSBTree(OSBTreeCollectionManagerAbstract.java:98) >> at >> com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerProxy.loadSBTree(OSBTreeCollectionManagerProxy.java:52) >> at >> com.orientechnologies.orient.core.storage.impl.local.paginated.ORidBagUpdateSerializationOperation.loadTree(ORidBagUpdateSerializationOperation.java:73) >> at >> com.orientechnologies.orient.core.storage.impl.local.paginated.ORidBagUpdateSerializationOperation.execute(ORidBagUpdateSerializationOperation.java:54) >> at >> com.orientechnologies.orient.core.storage.impl.local.paginated.ORecordSerializationContext.executeOperations(ORecordSerializationContext.java:99) >> at >> com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:1567) >> at >> com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.updateRecord(OAbstractPaginatedStorage.java:728) >> at >> com.orientechnologies.orient.server.distributed.ODistributedStorage.updateRecord(ODistributedStorage.java:599) >> at >> com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:1757) >> at >> com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:102) >> at >> com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2316) >> at >> com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2150) >> at >> com.orientechnologies.orient.server.distributed.task.OUpdateRecordTask.execute(OUpdateRecordTask.java:95) >> at >> com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.executeOnLocalNode(OHazelcastPlugin.java:753) >> at >> com.orientechnologies.orient.server.hazelcast.ODistributedWorker.onMessage(ODistributedWorker.java:298) >> at >> com.orientechnologies.orient.server.hazelcast.ODistributedWorker.run(ODistributedWorker.java:121) >> >> The database is then (understandably) completely out of sync and unusable. >> >> I also get the OSBTreeBonsaiLocal.java:455 stack trace if I have a >> database with writeQuorum 2, and an ETL import with tx true and batchCommit >> 1, which I would expect to be slow but safe. >> >> I'm a bit stuck at the moment, so unless anyone has any suggestions on >> the NullPointerExceptions or on how to allow the node I have run the >> offline ETL import on to sync its data to the other node, then I can't >> really see a way past this problem. Is ETL the wrong tool for this problem? >> Should I be batching up inserts and running them via another client API? >> >> I can try to provide a minimal test case if anyone is interested and the >> stack traces are not enough. >> >> Thanks, >> Rich >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "OrientDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
