[
https://issues.apache.org/jira/browse/DIRSERVER-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198396#comment-17198396
]
Rashid Mahmood commented on DIRSERVER-2255:
-------------------------------------------
We are facing same problem in Multi-Master setup in our Reference Env. Next
target is our production environment
We have 2 seperate data centers backed by load balancer. Each data center has a
Maste-Slave(Machine1 Master and Machine 2 slave) setup with 20k enrtries.
We need a Master-Master setup between Data Centers, i enabled the consumer on
each Data Center's Master instance to poll the other Data Center's Master
instance. Those Master instances had already 20K Entries. Then i noticed
Replication was not working, any change made on one Master is not replicated on
other Master. Memory usage on each Master was reaching to 2GB. Took a heap dump
on one instance and found SyncReplSearchListener occupied almost 90% of memory.
What could be the reason???
Couple of more questions:
1). Are both Masters trying to synch all entries? I think so maybe because both
newly enabled consumers on each Master are not sending any cookie to the target
Master, it makes Master think of sending all entries to consumer.
2). How many consumers one Master can serve, is there any benchmark?
3). Is there any guideline with best practices in setting up Multi-Master for
Production System?
4). We have a cron job which makes daily backup of our Data Partition, in case
of broken DB on one Master/Slave; should we restore from backup or just create
a fresh installation? To make restore with minimun Repk overhead we need to
make backup of the cookie entry so that Repl-Server send only those
modifications which happend after DB corruption, right???
Our Environment:
Linux Centos 7
ApacheDS 2.0.0.AM24
All consumers are with ads-replrefreshnpersist: TRUE configured.
!9-instances-SyncReplSearchListener .png!
> java.lang.OutOfMemoryError: Java heap space
> -------------------------------------------
>
> Key: DIRSERVER-2255
> URL: https://issues.apache.org/jira/browse/DIRSERVER-2255
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: replication
> Affects Versions: 2.0.0.AM25
> Environment: Linux CentOS 7
> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
> Reporter: li
> Priority: Major
> Attachments: 1.png, 9-instances-SyncReplSearchListener .png,
> conf.zip, wrapper.conf
>
>
> INFO | jvm 1 | 2018/10/29 12:01:45 | Exception in thread "Thread-4"
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.util.Arrays.copyOf(Arrays.java:3236)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.nio.charset.CharsetEncoder.replaceWith(CharsetEncoder.java:301)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:207)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:233)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:558)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:554)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.lang.StringCoding.encode(StringCoding.java:348)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.lang.String.getBytes(String.java:941)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.api.util.Strings.getBytesUtf8(Strings.java:1626)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.api.ldap.model.entry.Value.<init>(Value.java:261)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.apply(DefaultAttribute.java:1828)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.api.ldap.model.entry.DefaultEntry.readExternal(DefaultEntry.java:2443)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.ldap.replication.ReplicaEventMessageSerializer.deserialize(ReplicaEventMessageSerializer.java:126)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.deserialize(BPage.java:1188)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.deserialize(BPage.java:81)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.recman.BaseRecordManager.fetch(BaseRecordManager.java:329)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.loadBPage(BPage.java:949)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.childBPage(BPage.java:938)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.insert(BPage.java:380)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BTree.insert(BTree.java:302)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.put(JdbmTable.java:541)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.ldap.replication.provider.ReplicaEventLog.log(ReplicaEventLog.java:176)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.ldap.replication.provider.SyncReplSearchListener.entryModified(SyncReplSearchListener.java:314)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.core.event.EventInterceptor.fire(EventInterceptor.java:169)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.core.event.EventInterceptor.modify(EventInterceptor.java:333)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:508)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.core.subtree.SubentryInterceptor.modify(SubentryInterceptor.java:1249)
> INFO | jvm 1 | 2018/10/29 12:01:47 | Exception in thread
> "pool-13-thread-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.entry.Value.clone(Value.java:425)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.name.Ava.getValue(Ava.java:728)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.name.Rdn.<init>(Rdn.java:455)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.name.Dn.<init>(Dn.java:197)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.entry.DefaultEntry.normalizeDn(DefaultEntry.java:519)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.entry.DefaultEntry.<init>(DefaultEntry.java:300)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.ldap.client.api.LdapNetworkConnection.messageReceived(LdapNetworkConnection.java:2757)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1019)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1141)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:437)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:256)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1141)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:643)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:539)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1242)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1231)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> java.lang.Thread.run(Thread.java:748)
> INFO | jvm 1 | 2018/10/29 12:50:50 | Exception in thread
> "pool-7-thread-5835" java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> java.io.BufferedReader.<init>(BufferedReader.java:105)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> java.io.BufferedReader.<init>(BufferedReader.java:116)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.schema.parsers.OpenLdapSchemaParser.parseAttributeType(OpenLdapSchemaParser.java:2599)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.schema.parsers.AttributeTypeDescriptionSchemaParser.parse(AttributeTypeDescriptionSchemaParser.java:81)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.schema.syntaxCheckers.AttributeTypeDescriptionSyntaxChecker.isValidSyntax(AttributeTypeDescriptionSyntaxChecker.java:163)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.entry.Value.<init>(Value.java:285)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.createStringValue(DefaultAttribute.java:449)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.add(DefaultAttribute.java:1104)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.shared.SchemaService.generateAttributeTypes(SchemaService.java:144)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.shared.SchemaService.generateSchemaSubentry(SchemaService.java:258)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.shared.SchemaService.getSubschemaEntry(SchemaService.java:432)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.schema.SchemaInterceptor.search(SchemaInterceptor.java:1519)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.authz.DefaultAuthorizationInterceptor.search(DefaultAuthorizationInterceptor.java:354)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.authz.AciAuthorizationInterceptor.search(AciAuthorizationInterceptor.java:1247)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.authn.AuthenticationInterceptor.search(AuthenticationInterceptor.java:1394)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.normalization.NormalizationInterceptor.search(NormalizationInterceptor.java:414)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.DefaultOperationManager.search(DefaultOperationManager.java:1781)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.shared.DefaultCoreSession.search(DefaultCoreSession.java:1193)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.doSimpleSearch(SearchRequestHandler.java:810)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handleIgnoringReferrals(SearchRequestHandler.java:1164)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handleWithReferrals(SearchRequestHandler.java:1262)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handle(SearchRequestHandler.java:212)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handle(SearchRequestHandler.java:92)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:207)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:56)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:243)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:223)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1019)
> STATUS | wrapper | 2018/10/29 12:56:37 | TERM trapped. Shutting down.
> INFO | jvm 1 | 2018/10/29 12:56:46 | Error in WrapperListener.stop
> callback. java.lang.OutOfMemoryError: Java heap space
> INFO | jvm 1 | 2018/10/29 12:56:46 | java.lang.OutOfMemoryError: Java
> heap space
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> java.util.HashMap.resize(HashMap.java:704)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> java.util.HashMap.putVal(HashMap.java:663)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> java.util.HashMap.put(HashMap.java:612)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> jdbm.recman.RecordFile.commit(RecordFile.java:314)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> jdbm.recman.PageManager.close(PageManager.java:329)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> jdbm.recman.BaseRecordManager.close(BaseRecordManager.java:164)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.apache.directory.server.ldap.replication.provider.ReplicaEventLog.stop(ReplicaEventLog.java:226)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.apache.directory.server.ldap.replication.provider.SyncReplRequestHandler.stop(SyncReplRequestHandler.java:274)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.apache.directory.server.ldap.LdapServer.stop(LdapServer.java:668)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.apache.directory.server.ApacheDsService.stop(ApacheDsService.java:610)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.stop(ApacheDsTanukiWrapper.java:205)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.tanukisoftware.wrapper.WrapperManager$13.run(WrapperManager.java:3134)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
