Hi Chetan, sure. Below is the dump on startup. I highlighted in yellow the place where it loops over existing nodes and spends all those minutes. Thank you for any hits/feedback!
main [1] (RUNNABLE) java.net.SocketInputStream.socketRead0 line: not available [native method] java.net.SocketInputStream.socketRead line: 116 java.net.SocketInputStream.read line: 170 java.net.SocketInputStream.read line: 141 com.mongodb.connection.SocketStream.read line: 85 com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers line: 504 com.mongodb.connection.InternalStreamConnection.receiveMessage line: 222 com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage line: 105 com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage line: 438 com.mongodb.connection.CommandProtocol.execute line: 112 com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute line: 168 com.mongodb.connection.DefaultServerConnection.executeProtocol line: 289 com.mongodb.connection.DefaultServerConnection.command line: 176 com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol line: 216 com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol line: 207 com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol line: 113 com.mongodb.operation.FindOperation$1.call line: 488 com.mongodb.operation.FindOperation$1.call line: 483 com.mongodb.operation.OperationHelper.withConnectionSource line: 241 com.mongodb.operation.OperationHelper.withConnection line: 214 com.mongodb.operation.FindOperation.execute line: 483 com.mongodb.operation.FindOperation.execute line: 80 com.mongodb.Mongo.execute line: 818 com.mongodb.Mongo$2.execute line: 805 com.mongodb.DBCollection.findOne line: 777 com.mongodb.DBCollection.findOne line: 747 org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.findUncached line: 483 org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.findUncachedWithRetry line: 455 org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.find line: 400 org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.find line: 352 org.apache.jackrabbit.oak.plugins.document.NodeDocument.getPreviousDocument line: 1327 org.apache.jackrabbit.oak.plugins.document.NodeDocument.getPreviousDocs line: 1294 org.apache.jackrabbit.oak.plugins.document.NodeDocument.getCommitValue line: 1898 org.apache.jackrabbit.oak.plugins.document.NodeDocument.resolveCommitValue line: 1725 org.apache.jackrabbit.oak.plugins.document.NodeDocument.createLastRevs line: 1699 org.apache.jackrabbit.oak.plugins.document.NodeDocument.getNodeAtRevision line: 949 org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.readNode line: 1111 org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$4.call line: 839 org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$4.call line: 832 com.google.common.cache.LocalCache$LocalManualCache$1.load line: 4742 com.google.common.cache.LocalCache$LoadingValueReference.loadFuture line: 3527 com.google.common.cache.LocalCache$Segment.loadSync line: 2319 com.google.common.cache.LocalCache$Segment.lockedGetOrLoad line: 2282 com.google.common.cache.LocalCache$Segment.get line: 2197 com.google.common.cache.LocalCache.get line: 3937 com.google.common.cache.LocalCache$LocalManualCache.get line: 4739 org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.getNode line: 832 org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$8.apply line: 1072 org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$8.apply line: 1068 com.google.common.collect.Iterators$8.transform line: 799 com.google.common.collect.TransformedIterator.next line: 48 com.google.common.collect.TransformedIterator.next line: 48 org.apache.jackrabbit.oak.plugins.document.DocumentNodeState$ChildNodeEntryIterator.next line: 657 org.apache.jackrabbit.oak.plugins.document.DocumentNodeState$ChildNodeEntryIterator.next line: 626 com.google.common.collect.Iterators$7.computeNext line: 651 com.google.common.collect.AbstractIterator.tryToComputeNext line: 143 com.google.common.collect.AbstractIterator.hasNext line: 138 com.google.common.collect.Iterators$5.hasNext line: 547 org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.squeeze line: 125 org.apache.jackrabbit.oak.plugins.nodetype.write.InitialContent.initialize line: 118 org.apache.jackrabbit.oak.spi.lifecycle.CompositeInitializer.initialize line: 48 org.apache.jackrabbit.oak.spi.lifecycle.OakInitializer.initialize line: 42 org.apache.jackrabbit.oak.Oak.createNewContentRepository line: 628 org.apache.jackrabbit.oak.Oak.createContentRepository line: 616 org.apache.jackrabbit.oak.jcr.Jcr.createContentRepository line: 367 org.apache.jackrabbit.oak.jcr.Jcr.createRepository line: 375 com.gs.gwe.cntsvcs.utils.RepositoryHelpers.createRepository line: 144 com.gs.gwe.cntsvcs.ContentServiceApp$1.getDocumentRepository line: 267 com.gs.gwe.cntsvcs.ContentServiceApp$1$$FastClassByGuice$$37665752.invoke line: not available com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision line: 264 com.google.inject.internal.ProviderMethod$Factory.provision line: 401 com.google.inject.internal.ProviderMethod$Factory.get line: 376 com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call line: 46 com.google.inject.internal.InjectorImpl.callInContext line: 1092 com.google.inject.internal.ProviderToInternalFactoryAdapter.get line: 40 com.google.inject.internal.SingletonScope$1.get line: 194 com.google.inject.internal.InternalFactoryToProviderAdapter.get line: 41 com.google.inject.internal.InjectorImpl$2$1.call line: 1019 com.google.inject.internal.InjectorImpl.callInContext line: 1085 com.google.inject.internal.InjectorImpl$2.get line: 1015 com.google.inject.internal.InjectorImpl.getInstance line: 1050 com.gs.gwe.cntsvcs.ContentServiceApp.run line: 117 com.gs.gwe.cntsvcs.ContentServiceApp.run line: 61 io.dropwizard.cli.EnvironmentCommand.run line: 40 io.dropwizard.cli.ConfiguredCommand.run line: 77 io.dropwizard.cli.Cli.run line: 70 io.dropwizard.Application.run line: 80 com.gs.gwe.cntsvcs.ContentServiceApp.main line: 67 sun.reflect.NativeMethodAccessorImpl.invoke0 line: not available [native method] sun.reflect.NativeMethodAccessorImpl.invoke line: 62 sun.reflect.DelegatingMethodAccessorImpl.invoke line: 43 java.lang.reflect.Method.invoke line: 497 com.intellij.rt.execution.application.AppMain.main line: 134 On Fri, Feb 24, 2017 at 2:04 AM, Chetan Mehrotra <[email protected]> wrote: > Can you provide a thread dump around startup time where you see Oak is > reading all child nodes? > Chetan Mehrotra > > > On Fri, Feb 24, 2017 at 2:26 AM, Eugene Prystupa > <[email protected]> wrote: > > Thanks, Michael. > > > > I should have included more details in the original email. > > We are on 1.4.10 version of Jackrabbit Oak, we are using Mongo backend. > > > > > > On Thu, Feb 23, 2017 at 3:40 PM, Michael Dürig <[email protected]> > wrote: > > > >> > >> > >> On 23.02.17 19:11, Eugene Prystupa wrote: > >> > >>> We are seeing severe delays on start-up (20 minutes+) when repository > is > >>> created (new Jcr(oak).createRepository()). > >>> > >> > >> Regardless of the content structure, 20 min. seems off. What back-end > are > >> you on? Which version of Oak is this? > >> > >> Michael > >> > > > > > > > > -- > > Thanks, > > Eugene > -- Thanks, Eugene
