On Thu, Nov 7, 2013 at 10:50 PM, <[email protected]> wrote: > Kiran, thanks. > Yes the out of memory issue is on consumer, the provider is fine and > memory does not grow hardly at all. > > On or around line 1148 of ServiceBuilder the SyncReplConfiguration size > limit is not being set from the replBean. > When I add the line below > > config.setSearchSizeLimit( > replBean.getReplSearchSizeLimit() ); > > ahhhhh this is a bad mistake, thanks for finding it
> > The consumer only processes 5000 entries. That takes care of the JVM OOM > but the consumer does not appear to repeatedly process another 5000 at a > time. Should it, can it? > it should, can you file a bug please. otoh, change the mode to refreshOnly by setting ads-replRefreshNPersist to 'false' and try one all the 85k gets replicated you can change the value to 'true' ads-replRefreshNPersist: FALSE Thanks. > > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf > Of Kiran Ayyagari > Sent: Wednesday, November 06, 2013 11:43 PM > To: [email protected] > Subject: Re: Options to get a replica consumer initially loaded with large > db > > Carlo, > > On Thu, Nov 7, 2013 at 3:33 AM, <[email protected]> wrote: > > > Hi, > > > > We've successfully setup replication (m16 snapshot) and it's working > > great on our test instances with ~10K entries. > > With ~85K entries, we cannot get a consumer (slave) instance > > initially loaded without running out of memory. > > > this error is on consumer correct? > > > We're on 32 bit so the most we can set the JVM memory to is ~ 1.34 GB > > > > 2 questions: > > Is it possible to batch the replication changes 5000 at a time to give > > the jvm time to catch up? > > > should be possible with the config setting you mentioned below > > > We boot up an empty instance and it starts replicating the 85k > > entries. It runs out of memory before it's done. > > We're not sure what the state of the database is once this occurs. > > Again, when there are only 10k, it completes no problem. We tried > > setting ads-replSearchSizeLimit to 5k but this doesn't seem to make any > difference. > > > > this is strange, the value is indeed used while sending the > > replication > rquest > > > Also, Is it possible to configure a consumer to begin replicating > > from a certain time? For example, if I wanted the consumer to Pull > > everything in from 2 pm yesterday? We want to replay changes from a > > certain point to the slaves. > > > > the server already takes care of this based on the previous state > > stored > (using what is known as a 'cookie') > so after restart a consumer should be able to pickup from where it left > > > > > Thanks, > > > > Here's how our consumers are configured. > > > > dn: > > ads-replConsumerId=cpro,ou=replConsumers,ads-serverId=ldapServer,ou=se > > rvers,ads-directoryServiceId=default,ou=config > > objectClass: top > > objectClass: ads-base > > objectClass: ads-replConsumer > > ads-replSearchSizeLimit: 5000 > > ads-replAttributes: * > > ads-replConsumerId: cpro > > ads-replRefreshInterval: 60000 > > ads-replUserPassword: secret > > ads-replStrictCertValidation: FALSE > > ads-replUserDn: uid=admin,ou=system > > ads-replUseTls: FALSE > > ads-replProvPort: 10389 > > ads-replProvHostName: localhost > > ads-replRefreshNPersist: TRUE > > ads-replSearchScope: sub > > ads-replSearchTimeOut: 0 > > ads-searchBaseDN: o=cpro > > ads-replSearchFilter: (objectClass=*) > > ads-enabled: TRUE > > ads-replAliasDerefMode: never > > > > Java OOM > > jvm 1 | Exception in thread "Thread-4" > > jvm 1 | java.lang.OutOfMemoryError: Java heap space > > jvm 1 | at java.lang.Class.getDeclaredMethods0(Native Method) > > jvm 1 | at > > java.lang.Class.privateGetDeclaredMethods(Class.java:2436) > > jvm 1 | at java.lang.Class.getDeclaredMethod(Class.java:1937) > > jvm 1 | at > > > java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1344) > > jvm 1 | at > > java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:50) > > jvm 1 | at > > java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:446) > > jvm 1 | at java.security.AccessController.doPrivileged(Native > > Method) > > jvm 1 | at > > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:411) > > jvm 1 | at > > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308) > > jvm 1 | at > > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407) > > jvm 1 | at > > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308) > > jvm 1 | at > > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407) > > jvm 1 | at > > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308) > > jvm 1 | at > > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407) > > jvm 1 | at > > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308) > > jvm 1 | at > > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1114) > > jvm 1 | at > > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) > > jvm 1 | at jdbm.btree.BTree.writeExternal(BTree.java:580) > > jvm 1 | at > > > java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429) > > jvm 1 | at > > > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398) > > jvm 1 | at > > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) > > jvm 1 | at > > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) > > jvm 1 | at > > jdbm.helper.Serialization.serialize(Serialization.java:74) > > jvm 1 | at > > jdbm.helper.DefaultSerializer.serialize(DefaultSerializer.java:81) > > jvm 1 | at > > jdbm.recman.BaseRecordManager.update(BaseRecordManager.java:274) > > jvm 1 | at > > > jdbm.recman.CacheRecordManager.updateCacheEntries(CacheRecordManager.java:417) > > jvm 1 | at > > jdbm.recman.CacheRecordManager.commit(CacheRecordManager.java:349) > > jvm 1 | at > > > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.sync(JdbmTable.java:977) > > jvm 1 | at > > > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.commit(JdbmTable.java:1183) > > jvm 1 | at > > > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.remove(JdbmTable.java:829) > > jvm 1 | at > > > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.drop(JdbmIndex.java:435) > > jvm 1 | at > > > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.drop(JdbmIndex.java:1) > > jvm 1 | Server daemon died! > > > this definitely looking like a JDBM cache related issue, can you file a > bug please > > > jvm 1 | Exception in thread "pool-4-thread-1" > > jvm 1 | java.lang.OutOfMemoryError: Java heap space > > jvm 1 | at java.util.HashMap.newKeyIterator(HashMap.java:840) > > jvm 1 | at java.util.HashMap$KeySet.iterator(HashMap.java:874) > > jvm 1 | at java.util.HashSet.iterator(HashSet.java:153) > > jvm 1 | at > > > java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1005) > > jvm 1 | at > > > java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1004) > > jvm 1 | at > > > org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<init>(NioProcessor.java:321) > > jvm 1 | at > > > org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<init>(NioProcessor.java:311) > > jvm 1 | at > > > org.apache.mina.transport.socket.nio.NioProcessor.allSessions(NioProcessor.java:93) > > jvm 1 | at > > > org.apache.mina.core.polling.AbstractPollingIoProcessor.notifyIdleSessions(AbstractPollingIoProcessor.java:760) > > jvm 1 | at > > > org.apache.mina.core.polling.AbstractPollingIoProcessor.access$900(AbstractPollingIoProcessor.java:67) > > jvm 1 | at > > > org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1135) > > jvm 1 | at > > > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > > jvm 1 | at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > > jvm 1 | at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > > jvm 1 | at java.lang.Thread.run(Thread.java:662) > > jvm 1 | java.lang.OutOfMemoryError: Java heap space > > jvm 1 | at java.lang.String.toCharArray(String.java:2725) > > jvm 1 | at > > sun.nio.cs.SingleByteDecoder.<init>(SingleByteDecoder.java:27) > > jvm 1 | at sun.nio.cs.MS1252$Decoder.<init>(MS1252.java:72) > > jvm 1 | at sun.nio.cs.MS1252.newDecoder(MS1252.java:39) > > jvm 1 | at > > java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:116) > > jvm 1 | at > > java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:108) > > jvm 1 | at java.lang.StringCoding.decode(StringCoding.java:167) > > jvm 1 | at java.lang.StringCoding.decode(StringCoding.java:185) > > jvm 1 | at java.lang.String.<init>(String.java:570) > > jvm 1 | at > > > org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3759) > > jvm 1 | at > > org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084) > > jvm 1 | at java.lang.Thread.run(Thread.java:662) > > jvm 1 | [15:47:16] WARN > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Java > > heap space > > jvm 1 | java.lang.OutOfMemoryError: Java heap space > > wrapper | JVM appears hung: Timed out waiting for signal from JVM. > > wrapper | JVM did not exit on request, terminated wrapper | > > Launching a JVM... > > jvm 2 | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org > > jvm 2 | Copyright 1999-2006 Tanuki Software, Inc. All Rights > > Reserved. > > jvm 2 | > > > is the below error thrown when a restart was attempted by you after the > process was killed with OOM? > > > jvm 2 | [15:54:56] ERROR > > [org.apache.directory.server.core.authz.GroupCache] > > - Exception while initializing the groupCache: {} > > jvm 2 | > > org.apache.directory.api.ldap.model.exception.LdapOperationErrorException > > jvm 2 | at > > > org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1120) > > jvm 2 | at > > > org.apache.directory.server.xdbm.search.evaluator.EqualityEvaluator.evaluate(EqualityEvaluator.java:97) > > jvm 2 | at > > > org.apache.directory.server.xdbm.search.evaluator.AndEvaluator.evaluate(AndEvaluator.java:110) > > jvm 2 | at > > > org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:167) > > jvm 2 | at > > > org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:1) > > jvm 2 | at > > > org.apache.directory.server.core.api.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:377) > > jvm 2 | at > > > org.apache.directory.server.core.authz.GroupCache.initialize(GroupCache.java:164) > > jvm 2 | at > > > org.apache.directory.server.core.authz.GroupCache.<init>(GroupCache.java:122) > > jvm 2 | at > > > org.apache.directory.server.core.authz.AciAuthorizationInterceptor.init(AciAuthorizationInterceptor.java:286) > > jvm 2 | at > > > org.apache.directory.server.core.DefaultDirectoryService.initInterceptors(DefaultDirectoryService.java:688) > > jvm 2 | at > > > org.apache.directory.server.core.DefaultDirectoryService.initialize(DefaultDirectoryService.java:1836) > > jvm 2 | at > > > org.apache.directory.server.core.DefaultDirectoryService.startup(DefaultDirectoryService.java:1247) > > jvm 2 | at > > > org.apache.directory.server.ApacheDsService.initDirectoryService(ApacheDsService.java:323) > > jvm 2 | at > > > org.apache.directory.server.ApacheDsService.start(ApacheDsService.java:182) > > jvm 2 | at > > > org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.start(ApacheDsTanukiWrapper.java:72) > > jvm 2 | at > > > org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788) > > jvm 2 | Caused by: > > org.apache.directory.api.ldap.model.exception.LdapOperationErrorException > > jvm 2 | at > > > org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1196) > > jvm 2 | at > > > org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1116) > > jvm 2 | ... 15 more > > > > > > > -- > Kiran Ayyagari > http://keydap.com > -- Kiran Ayyagari http://keydap.com
