Carlo, On Thu, Nov 7, 2013 at 3:33 AM, <[email protected]> wrote:
> Hi, > > We've successfully setup replication (m16 snapshot) and it's working great > on our test instances with ~10K entries. > With ~85K entries, we cannot get a consumer (slave) instance initially > loaded without running out of memory. > this error is on consumer correct? > We're on 32 bit so the most we can set the JVM memory to is ~ 1.34 GB > > 2 questions: > Is it possible to batch the replication changes 5000 at a time to give the > jvm time to catch up? > should be possible with the config setting you mentioned below > We boot up an empty instance and it starts replicating the 85k entries. It > runs out of memory before it's done. > We're not sure what the state of the database is once this occurs. > Again, when there are only 10k, it completes no problem. We tried setting > ads-replSearchSizeLimit to 5k but this doesn't seem to make any difference. > > this is strange, the value is indeed used while sending the replication rquest > Also, Is it possible to configure a consumer to begin replicating from a > certain time? For example, if I wanted the consumer to > Pull everything in from 2 pm yesterday? We want to replay changes from a > certain point to the slaves. > > the server already takes care of this based on the previous state stored (using what is known as a 'cookie') so after restart a consumer should be able to pickup from where it left > > Thanks, > > Here's how our consumers are configured. > > dn: > ads-replConsumerId=cpro,ou=replConsumers,ads-serverId=ldapServer,ou=servers,ads-directoryServiceId=default,ou=config > objectClass: top > objectClass: ads-base > objectClass: ads-replConsumer > ads-replSearchSizeLimit: 5000 > ads-replAttributes: * > ads-replConsumerId: cpro > ads-replRefreshInterval: 60000 > ads-replUserPassword: secret > ads-replStrictCertValidation: FALSE > ads-replUserDn: uid=admin,ou=system > ads-replUseTls: FALSE > ads-replProvPort: 10389 > ads-replProvHostName: localhost > ads-replRefreshNPersist: TRUE > ads-replSearchScope: sub > ads-replSearchTimeOut: 0 > ads-searchBaseDN: o=cpro > ads-replSearchFilter: (objectClass=*) > ads-enabled: TRUE > ads-replAliasDerefMode: never > > Java OOM > jvm 1 | Exception in thread "Thread-4" > jvm 1 | java.lang.OutOfMemoryError: Java heap space > jvm 1 | at java.lang.Class.getDeclaredMethods0(Native Method) > jvm 1 | at > java.lang.Class.privateGetDeclaredMethods(Class.java:2436) > jvm 1 | at java.lang.Class.getDeclaredMethod(Class.java:1937) > jvm 1 | at > java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1344) > jvm 1 | at > java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:50) > jvm 1 | at > java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:446) > jvm 1 | at java.security.AccessController.doPrivileged(Native > Method) > jvm 1 | at > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:411) > jvm 1 | at > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308) > jvm 1 | at > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407) > jvm 1 | at > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308) > jvm 1 | at > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407) > jvm 1 | at > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308) > jvm 1 | at > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407) > jvm 1 | at > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308) > jvm 1 | at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1114) > jvm 1 | at > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) > jvm 1 | at jdbm.btree.BTree.writeExternal(BTree.java:580) > jvm 1 | at > java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429) > jvm 1 | at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398) > jvm 1 | at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) > jvm 1 | at > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) > jvm 1 | at > jdbm.helper.Serialization.serialize(Serialization.java:74) > jvm 1 | at > jdbm.helper.DefaultSerializer.serialize(DefaultSerializer.java:81) > jvm 1 | at > jdbm.recman.BaseRecordManager.update(BaseRecordManager.java:274) > jvm 1 | at > jdbm.recman.CacheRecordManager.updateCacheEntries(CacheRecordManager.java:417) > jvm 1 | at > jdbm.recman.CacheRecordManager.commit(CacheRecordManager.java:349) > jvm 1 | at > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.sync(JdbmTable.java:977) > jvm 1 | at > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.commit(JdbmTable.java:1183) > jvm 1 | at > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.remove(JdbmTable.java:829) > jvm 1 | at > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.drop(JdbmIndex.java:435) > jvm 1 | at > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.drop(JdbmIndex.java:1) > jvm 1 | Server daemon died! > this definitely looking like a JDBM cache related issue, can you file a bug please > jvm 1 | Exception in thread "pool-4-thread-1" > jvm 1 | java.lang.OutOfMemoryError: Java heap space > jvm 1 | at java.util.HashMap.newKeyIterator(HashMap.java:840) > jvm 1 | at java.util.HashMap$KeySet.iterator(HashMap.java:874) > jvm 1 | at java.util.HashSet.iterator(HashSet.java:153) > jvm 1 | at > java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1005) > jvm 1 | at > java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1004) > jvm 1 | at > org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<init>(NioProcessor.java:321) > jvm 1 | at > org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<init>(NioProcessor.java:311) > jvm 1 | at > org.apache.mina.transport.socket.nio.NioProcessor.allSessions(NioProcessor.java:93) > jvm 1 | at > org.apache.mina.core.polling.AbstractPollingIoProcessor.notifyIdleSessions(AbstractPollingIoProcessor.java:760) > jvm 1 | at > org.apache.mina.core.polling.AbstractPollingIoProcessor.access$900(AbstractPollingIoProcessor.java:67) > jvm 1 | at > org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1135) > jvm 1 | at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > jvm 1 | at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > jvm 1 | at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > jvm 1 | at java.lang.Thread.run(Thread.java:662) > jvm 1 | java.lang.OutOfMemoryError: Java heap space > jvm 1 | at java.lang.String.toCharArray(String.java:2725) > jvm 1 | at > sun.nio.cs.SingleByteDecoder.<init>(SingleByteDecoder.java:27) > jvm 1 | at sun.nio.cs.MS1252$Decoder.<init>(MS1252.java:72) > jvm 1 | at sun.nio.cs.MS1252.newDecoder(MS1252.java:39) > jvm 1 | at > java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:116) > jvm 1 | at > java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:108) > jvm 1 | at java.lang.StringCoding.decode(StringCoding.java:167) > jvm 1 | at java.lang.StringCoding.decode(StringCoding.java:185) > jvm 1 | at java.lang.String.<init>(String.java:570) > jvm 1 | at > org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3759) > jvm 1 | at > org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084) > jvm 1 | at java.lang.Thread.run(Thread.java:662) > jvm 1 | [15:47:16] WARN > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Java heap > space > jvm 1 | java.lang.OutOfMemoryError: Java heap space > wrapper | JVM appears hung: Timed out waiting for signal from JVM. > wrapper | JVM did not exit on request, terminated > wrapper | Launching a JVM... > jvm 2 | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org > jvm 2 | Copyright 1999-2006 Tanuki Software, Inc. All Rights > Reserved. > jvm 2 | > is the below error thrown when a restart was attempted by you after the process was killed with OOM? > jvm 2 | [15:54:56] ERROR > [org.apache.directory.server.core.authz.GroupCache] > - Exception while initializing the groupCache: {} > jvm 2 | > org.apache.directory.api.ldap.model.exception.LdapOperationErrorException > jvm 2 | at > org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1120) > jvm 2 | at > org.apache.directory.server.xdbm.search.evaluator.EqualityEvaluator.evaluate(EqualityEvaluator.java:97) > jvm 2 | at > org.apache.directory.server.xdbm.search.evaluator.AndEvaluator.evaluate(AndEvaluator.java:110) > jvm 2 | at > org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:167) > jvm 2 | at > org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:1) > jvm 2 | at > org.apache.directory.server.core.api.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:377) > jvm 2 | at > org.apache.directory.server.core.authz.GroupCache.initialize(GroupCache.java:164) > jvm 2 | at > org.apache.directory.server.core.authz.GroupCache.<init>(GroupCache.java:122) > jvm 2 | at > org.apache.directory.server.core.authz.AciAuthorizationInterceptor.init(AciAuthorizationInterceptor.java:286) > jvm 2 | at > org.apache.directory.server.core.DefaultDirectoryService.initInterceptors(DefaultDirectoryService.java:688) > jvm 2 | at > org.apache.directory.server.core.DefaultDirectoryService.initialize(DefaultDirectoryService.java:1836) > jvm 2 | at > org.apache.directory.server.core.DefaultDirectoryService.startup(DefaultDirectoryService.java:1247) > jvm 2 | at > org.apache.directory.server.ApacheDsService.initDirectoryService(ApacheDsService.java:323) > jvm 2 | at > org.apache.directory.server.ApacheDsService.start(ApacheDsService.java:182) > jvm 2 | at > org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.start(ApacheDsTanukiWrapper.java:72) > jvm 2 | at > org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788) > jvm 2 | Caused by: > org.apache.directory.api.ldap.model.exception.LdapOperationErrorException > jvm 2 | at > org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1196) > jvm 2 | at > org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1116) > jvm 2 | ... 15 more > > -- Kiran Ayyagari http://keydap.com
