[jira] [Resolved] (OAK-2758) Performance: Consider caches for MutableTree#getTree and MemoryNodeBuilder#getChildNode
[
https://issues.apache.org/jira/browse/OAK-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Richard resolved OAK-2758.
---
Resolution: Invalid
> Performance: Consider caches for MutableTree#getTree and
> MemoryNodeBuilder#getChildNode
> ---
>
> Key: OAK-2758
> URL: https://issues.apache.org/jira/browse/OAK-2758
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.2
>Reporter: Joel Richard
>Priority: Critical
> Labels: performance
> Attachments: OAK-2758-SecureNodeBuilder-lastChildNode-cache.patch,
> OAK-2758-SegmentNodeState-lastChildNode-cache.patch, experimental_caches.patch
>
>
> While I was analysing Sling's rendering performance, I noticed that it would
> help a lot to implement a cache for the ResourceResolver. For some pages it
> almost doubled the rendering performance. This made me wondering whether
> Oak's read performance could still be improved.
> I noticed that a lot of time is spent in MutableTree#getTree and
> MemoryNodeBuilder#getChildNode and have implemented a specific cache for
> them. These two caches improve the read performance up to 4 times and do not
> break any oak-core tests.
> Here the benchmark results for ReadDeepTreeTest:
> {code}
> Fixtures: Oak-Tar
> Admin User: false
> Runtime: 5
> Num Items: 1000
> Concurrency: 1,2,4
> Random User: true
> Profiling: false
> --
> Executing benchmarks as admin: false on Oak-Tar
> ---
> # ReadDeepTreeTest , C,min,10%,50%,90%,max,
> N
> Oak-Tar , 1, 16, 16, 17, 19, 22,
>290
> Oak-Tar , 2, 23, 29, 44, 68,115,
>216
> Oak-Tar , 4, 24, 43, 97,154,232,
>207
> {code}
> The same results with my changes:
> {code}
> # ReadDeepTreeTest , C,min,10%,50%,90%,max,
> N
> Oak-Tar , 1, 4, 4, 5, 5, 15,
> 1038
> Oak-Tar , 2, 10, 14, 16, 20, 60,
>577
> Oak-Tar , 4, 13, 27, 32, 40, 69,
>605
> {code}
> I have also implemented another cache for properties, but it didn't really
> help and broke some tests.
> The experimental patch is attached. It's not meant to be applied, but just to
> point out areas with potential for improvement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3600) Cache property index definitions
[ https://issues.apache.org/jira/browse/OAK-3600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996326#comment-14996326 ] Joel Richard commented on OAK-3600: --- [~tmueller], can you please share your thoughts about this? > Cache property index definitions > > > Key: OAK-3600 > URL: https://issues.apache.org/jira/browse/OAK-3600 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, query >Affects Versions: 1.3.9 >Reporter: Joel Richard > Labels: performance > > At the moment, most of the time of the execution plan calculation (often 70%) > is spent in PropertyIndex.getCost. Therefore, it would make sense to cache > the property index definitions in a map which avoids all > unnecessary traversal/repository operations. > This cache could either be attached to the session or expire after a few > seconds because the problem is not that createPlan is slow itself, but is > just called too often. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3600) Cache property index definitions
Joel Richard created OAK-3600: - Summary: Cache property index definitions Key: OAK-3600 URL: https://issues.apache.org/jira/browse/OAK-3600 Project: Jackrabbit Oak Issue Type: Improvement Components: core, query Affects Versions: 1.3.9 Reporter: Joel Richard At the moment, most of the time of the execution plan calculation (often 70%) is spent in PropertyIndex.getCost. Therefore, it would make sense to cache the property index definitions in a map which avoids all unnecessary traversal/repository operations. This cache could either be attached to the session or expire after a few seconds because the problem is not that createPlan is slow itself, but is just called too often. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3535) Update jackrabbit version to 2.11.2
Joel Richard created OAK-3535: - Summary: Update jackrabbit version to 2.11.2 Key: OAK-3535 URL: https://issues.apache.org/jira/browse/OAK-3535 Project: Jackrabbit Oak Issue Type: Task Reporter: Joel Richard Fix For: 1.3.9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2679) Query engine: faster cost calculation
[ https://issues.apache.org/jira/browse/OAK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907746#comment-14907746 ] Joel Richard commented on OAK-2679: --- [~mmarth], it turned out that it is very hard to implement a cache for the execution plans because the values and even uncommitted changes have an influence on which index must be used. On the other hand, the first results of this optimisation look very promising and almost as good as a cache. If necessary, it can be improved further by creating a cached index definition data structure (instead of traversing over NodeStates) which is updated with an observation listener (as far as I remember that is the most expensive task right now). > Query engine: faster cost calculation > - > > Key: OAK-2679 > URL: https://issues.apache.org/jira/browse/OAK-2679 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, query >Reporter: Thomas Mueller >Assignee: Thomas Mueller > Labels: performance > Fix For: 1.3.7 > > Attachments: 0001-OAK-2679-Reduce-execution-plan-overhead_0.2.patch, > OAK-2679-2.patch, OAK-2679.patch, executionplancache.patch > > > If there are many indexes, preparing a query can take a long time, in > relation to executing the query. > The query execution plans can be cached. The cache should be invalidated if > there are new indexes, or indexes are changed; a simple solution might be to > use a timeout, and / or a manual cache clean via JMX or so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3447) Parallelize recursive usage of compareAgainstBaseState
Joel Richard created OAK-3447: - Summary: Parallelize recursive usage of compareAgainstBaseState Key: OAK-3447 URL: https://issues.apache.org/jira/browse/OAK-3447 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.3.6 Environment: All, especially document Reporter: Joel Richard In order to improve the performance of compareAgainstBaseState, it would help to parallelize the recursive usage of compareAgainstBaseState. The idea is that each sub tree which has a different revision would then be processed in parallel (although it would probably suffice to only fork the process when the nodes are not cached). This should significantly reduce the time which is lost while waiting for an external database assumed that there are at least two changes between the base revisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-1312) Bundle nodes into a document
[ https://issues.apache.org/jira/browse/OAK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906328#comment-14906328 ] Joel Richard commented on OAK-1312: --- One approach to decide which nodes should be saved together in a document could be to leverage the node type and mixin information: If the type or mixins of a node allow or require certain child nodes, then these child nodes would be stored together with the parent node in a single document. Since a child node isn't read before it's parent anyway, nodes could still be retrieved in the old way. Example: Since a node with access control needs to have the rep:AccessControllable mixin and the mixin suggests that a node with this mixin might have the rep:policy child node, these two nodes would be stored together. > Bundle nodes into a document > > > Key: OAK-1312 > URL: https://issues.apache.org/jira/browse/OAK-1312 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, mongomk >Reporter: Marcel Reutegger > > For very fine grained content with many nodes and only few properties per > node it would be more efficient to bundle multiple nodes into a single > MongoDB document. Mostly reading would benefit because there are less > roundtrips to the backend. At the same time storage footprint would be lower > because metadata overhead is per document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3166) Apply adjustments for newly exported JackrabbitSession#getItemOrNull
[ https://issues.apache.org/jira/browse/OAK-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-3166: -- Fix Version/s: 1.3.7 > Apply adjustments for newly exported JackrabbitSession#getItemOrNull > > > Key: OAK-3166 > URL: https://issues.apache.org/jira/browse/OAK-3166 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: jcr >Affects Versions: 1.3.3 >Reporter: Joel Richard >Assignee: Dominique Jäggi > Fix For: 1.3.7 > > Attachments: > 0001-OAK-3166-Apply-adjustments-for-newly-exported-Jackra.patch > > > Once the patch for JCR-3870 has been applied, there are a few cosmetic > adjustments necessary and test coverage should be increased. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3361) Reduce PerfLogger isDebugEnabled overhead
Joel Richard created OAK-3361: - Summary: Reduce PerfLogger isDebugEnabled overhead Key: OAK-3361 URL: https://issues.apache.org/jira/browse/OAK-3361 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.3.4 Reporter: Joel Richard If a node is cached, 1/4 of the time which is used to call DocumentNodeStore.getNode is spent in PerfLogger.start and PerfLogger.end just for checking whether or not debug logging is enabled (this is likely much less if no TurboFilters are used). To reduce the overhead of the PerfLogger, it should not check if debug is enabled in end() if start is below 0 anyway. Moreover, it would help to check only every second if debug is really enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3361) Reduce PerfLogger isDebugEnabled overhead
[ https://issues.apache.org/jira/browse/OAK-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-3361: -- Attachment: 0001-OAK-3361-Reduce-PerfLogger-isDebugEnabled-overhead.patch [~chetanm], would you mind reviewing the attached patch? > Reduce PerfLogger isDebugEnabled overhead > - > > Key: OAK-3361 > URL: https://issues.apache.org/jira/browse/OAK-3361 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Affects Versions: 1.3.4 >Reporter: Joel Richard > Labels: performance > Attachments: > 0001-OAK-3361-Reduce-PerfLogger-isDebugEnabled-overhead.patch > > > If a node is cached, 1/4 of the time which is used to call > DocumentNodeStore.getNode is spent in PerfLogger.start and PerfLogger.end > just for checking whether or not debug logging is enabled (this is likely > much less if no TurboFilters are used). > To reduce the overhead of the PerfLogger, it should not check if debug is > enabled in end() if start is below 0 anyway. Moreover, it would help to check > only every second if debug is really enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2679) Query engine: cache execution plans
[ https://issues.apache.org/jira/browse/OAK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-2679: -- Attachment: 0001-OAK-2679-Reduce-execution-plan-overhead_0.2.patch I have attached an updated version of the patch (0001-OAK-2679-Reduce-execution-plan-overhead_0.2.patch) which makes the PropertyIndex plan cache more effective. Query engine: cache execution plans --- Key: OAK-2679 URL: https://issues.apache.org/jira/browse/OAK-2679 Project: Jackrabbit Oak Issue Type: Improvement Components: core, query Reporter: Thomas Mueller Assignee: Thomas Mueller Labels: performance Fix For: 1.3.5 Attachments: 0001-OAK-2679-Reduce-execution-plan-overhead_0.2.patch, OAK-2679.patch, executionplancache.patch If there are many indexes, preparing a query can take a long time, in relation to executing the query. The query execution plans can be cached. The cache should be invalidated if there are new indexes, or indexes are changed; a simple solution might be to use a timeout, and / or a manual cache clean via JMX or so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2679) Query engine: cache execution plans
[ https://issues.apache.org/jira/browse/OAK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-2679: -- Attachment: (was: 0001-OAK-2679-Reduce-execution-plan-overhead.patch) Query engine: cache execution plans --- Key: OAK-2679 URL: https://issues.apache.org/jira/browse/OAK-2679 Project: Jackrabbit Oak Issue Type: Improvement Components: core, query Reporter: Thomas Mueller Assignee: Thomas Mueller Labels: performance Fix For: 1.3.5 Attachments: OAK-2679.patch, executionplancache.patch If there are many indexes, preparing a query can take a long time, in relation to executing the query. The query execution plans can be cached. The cache should be invalidated if there are new indexes, or indexes are changed; a simple solution might be to use a timeout, and / or a manual cache clean via JMX or so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3153) Make it possible to disable recording of stack trace in SessionStats
[ https://issues.apache.org/jira/browse/OAK-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-3153: -- Attachment: (was: 0001-OAK-3153-Make-it-possible-to-disable-recording-of-st.patch) Make it possible to disable recording of stack trace in SessionStats Key: OAK-3153 URL: https://issues.apache.org/jira/browse/OAK-3153 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.3.3 Reporter: Joel Richard Labels: performance For the rendering of some pages we have to create a lot of sessions. Around 9% of the rendering time is spent inside of RepositoryImpl.login. Half of this time is spent creating the exception in SessionStats. Therefore, it would be useful if the recording of the exception could be disabled to improve the performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3153) Make it possible to disable recording of stack trace in SessionStats
[ https://issues.apache.org/jira/browse/OAK-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-3153: -- Attachment: 0001-OAK-3153-Make-it-possible-to-disable-recording-of-st.patch Make it possible to disable recording of stack trace in SessionStats Key: OAK-3153 URL: https://issues.apache.org/jira/browse/OAK-3153 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.3.3 Reporter: Joel Richard Labels: performance Attachments: 0001-OAK-3153-Make-it-possible-to-disable-recording-of-st.patch For the rendering of some pages we have to create a lot of sessions. Around 9% of the rendering time is spent inside of RepositoryImpl.login. Half of this time is spent creating the exception in SessionStats. Therefore, it would be useful if the recording of the exception could be disabled to improve the performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2679) Query engine: cache execution plans
[
https://issues.apache.org/jira/browse/OAK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Richard updated OAK-2679:
--
Attachment: 0001-OAK-2679-Reduce-execution-plan-overhead.patch
Attached patch (0001-OAK-2679-Reduce-execution-plan-overhead.patch) which does
not use a cache, but instead reduces the execution plan overhead with the
following changes:
{quote}
* Add QueryIndex.getMinimalCost which allows to skip the remaining index cost
comparison if it finds an index whose cost is = the minimum cost of the next
more expensive index
* Implement getMinimalCost for all indexes
* Do not read index definitions in PropertyIndex.getCost if the filter does not
have property restrictions and selector restrictions
* Stop looking for better index definition in PropertyIndex once it finds a
definition with the absolute minimum cost
* Cache for 30 seconds if there are no ordered property or Solr indexes
{quote}
More about it on the mailing list in a minute.
Query engine: cache execution plans
---
Key: OAK-2679
URL: https://issues.apache.org/jira/browse/OAK-2679
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core, query
Reporter: Thomas Mueller
Assignee: Thomas Mueller
Labels: performance
Fix For: 1.3.5
Attachments: 0001-OAK-2679-Reduce-execution-plan-overhead.patch,
OAK-2679.patch, executionplancache.patch
If there are many indexes, preparing a query can take a long time, in
relation to executing the query.
The query execution plans can be cached. The cache should be invalidated if
there are new indexes, or indexes are changed; a simple solution might be to
use a timeout, and / or a manual cache clean via JMX or so.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-2679) Query engine: cache execution plans
[
https://issues.apache.org/jira/browse/OAK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697050#comment-14697050
]
Joel Richard edited comment on OAK-2679 at 8/14/15 2:16 PM:
Attached patch (0001-OAK-2679-Reduce-execution-plan-overhead.patch) which does
not use a cache, but instead reduces the execution plan overhead with the
following changes:
{quote}
* Add QueryIndex.getMinimalCost which allows to skip the remaining index cost
comparison if it finds an index whose cost is = the minimum cost of the next
more expensive index
* Implement getMinimalCost for all indexes
* Do not read index definitions in PropertyIndex.getCost if the filter does not
have property restrictions and selector restrictions
* Stop looking for better index definition in PropertyIndex once it finds a
definition with the absolute minimum cost
* Cache for 30 seconds if there are no ordered property or Solr indexes
{quote}
More about it on the mailing list in a minute (subject Patch for OAK-2679 to
reduce query execution plan overhead).
was (Author: joelrich):
Attached patch (0001-OAK-2679-Reduce-execution-plan-overhead.patch) which does
not use a cache, but instead reduces the execution plan overhead with the
following changes:
{quote}
* Add QueryIndex.getMinimalCost which allows to skip the remaining index cost
comparison if it finds an index whose cost is = the minimum cost of the next
more expensive index
* Implement getMinimalCost for all indexes
* Do not read index definitions in PropertyIndex.getCost if the filter does not
have property restrictions and selector restrictions
* Stop looking for better index definition in PropertyIndex once it finds a
definition with the absolute minimum cost
* Cache for 30 seconds if there are no ordered property or Solr indexes
{quote}
More about it on the mailing list in a minute.
Query engine: cache execution plans
---
Key: OAK-2679
URL: https://issues.apache.org/jira/browse/OAK-2679
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core, query
Reporter: Thomas Mueller
Assignee: Thomas Mueller
Labels: performance
Fix For: 1.3.5
Attachments: 0001-OAK-2679-Reduce-execution-plan-overhead.patch,
OAK-2679.patch, executionplancache.patch
If there are many indexes, preparing a query can take a long time, in
relation to executing the query.
The query execution plans can be cached. The cache should be invalidated if
there are new indexes, or indexes are changed; a simple solution might be to
use a timeout, and / or a manual cache clean via JMX or so.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-3153) Make it possible to disable recording of stack trace in SessionStats
[ https://issues.apache.org/jira/browse/OAK-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680195#comment-14680195 ] Joel Richard edited comment on OAK-3153 at 8/10/15 2:37 PM: [~tmueller], I have attached a patch. Can you please review and apply it? The patch disables the recording of the init stack trace until 2000 concurrent sessions are open. The threshold can be configured/disabled by setting the oak.sessionStats.initStackTraceThreshold system property. *Edit:* I have just seen that you asked for a threshold of 1000 above. Can you please have a look at the code anyway and if there are some other things you would like me to change I will make all changes at once. was (Author: joelrich): [~tmueller], I have attached a patch. Can you please review and apply it? The patch disables the recording of the init stack trace until 2000 concurrent sessions are open. The threshold can be configured/disabled by setting the oak.sessionStats.initStackTraceThreshold system property. Make it possible to disable recording of stack trace in SessionStats Key: OAK-3153 URL: https://issues.apache.org/jira/browse/OAK-3153 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.3.3 Reporter: Joel Richard Labels: performance Attachments: 0001-OAK-3153-Make-it-possible-to-disable-recording-of-st.patch For the rendering of some pages we have to create a lot of sessions. Around 9% of the rendering time is spent inside of RepositoryImpl.login. Half of this time is spent creating the exception in SessionStats. Therefore, it would be useful if the recording of the exception could be disabled to improve the performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3153) Make it possible to disable recording of stack trace in SessionStats
[ https://issues.apache.org/jira/browse/OAK-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-3153: -- Attachment: 0001-OAK-3153-Make-it-possible-to-disable-recording-of-st.patch [~tmueller], I have attached a patch. Can you please review and apply it? The patch disables the recording of the init stack trace until 2000 concurrent sessions are open. The threshold can be configured/disabled by setting the oak.sessionStats.initStackTraceThreshold system property. Make it possible to disable recording of stack trace in SessionStats Key: OAK-3153 URL: https://issues.apache.org/jira/browse/OAK-3153 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.3.3 Reporter: Joel Richard Labels: performance Attachments: 0001-OAK-3153-Make-it-possible-to-disable-recording-of-st.patch For the rendering of some pages we have to create a lot of sessions. Around 9% of the rendering time is spent inside of RepositoryImpl.login. Half of this time is spent creating the exception in SessionStats. Therefore, it would be useful if the recording of the exception could be disabled to improve the performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3153) Make it possible to disable recording of stack trace in SessionStats
[ https://issues.apache.org/jira/browse/OAK-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653223#comment-14653223 ] Joel Richard commented on OAK-3153: --- [~tmueller], we could also just use an AtomicInteger which is incremented when a session is created and decremented when it is closed. To reduce the volatility, only long living sessions could be counted. Make it possible to disable recording of stack trace in SessionStats Key: OAK-3153 URL: https://issues.apache.org/jira/browse/OAK-3153 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.3.3 Reporter: Joel Richard Labels: performance For the rendering of some pages we have to create a lot of sessions. Around 9% of the rendering time is spent inside of RepositoryImpl.login. Half of this time is spent creating the exception in SessionStats. Therefore, it would be useful if the recording of the exception could be disabled to improve the performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3153) Make it possible to disable recording of stack trace in SessionStats
[ https://issues.apache.org/jira/browse/OAK-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653245#comment-14653245 ] Joel Richard commented on OAK-3153: --- Then we could just introduce a threshold option which enables the exception recording once exceeded. By setting it to 0 the current behaviour could be kept. What do you think? Make it possible to disable recording of stack trace in SessionStats Key: OAK-3153 URL: https://issues.apache.org/jira/browse/OAK-3153 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.3.3 Reporter: Joel Richard Labels: performance For the rendering of some pages we have to create a lot of sessions. Around 9% of the rendering time is spent inside of RepositoryImpl.login. Half of this time is spent creating the exception in SessionStats. Therefore, it would be useful if the recording of the exception could be disabled to improve the performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3153) Make it possible to disable recording of stack trace in SessionStats
[
https://issues.apache.org/jira/browse/OAK-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14651538#comment-14651538
]
Joel Richard commented on OAK-3153:
---
I have discussed this issue with [~tmueller] and he didn't like the idea to
make it possible disabling it in general because then you cannot easily find
session leaks anymore. One option would be to keep track of the number of long
living sessions and if they keep increasing enable this option automatically.
However, I don't know if it's worth that much effort. [~tmueller], what do you
think?
The ideal solution would be an API like
{code}
repository.login(credentials, new WithSession() {
doDork(Session session) {
}
})
{code}
but this is very unlikely an option and it wouldn't even solve our particular
problem because there we use SlingRepository.
Make it possible to disable recording of stack trace in SessionStats
Key: OAK-3153
URL: https://issues.apache.org/jira/browse/OAK-3153
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Affects Versions: 1.3.3
Reporter: Joel Richard
Labels: performance
For the rendering of some pages we have to create a lot of sessions. Around
9% of the rendering time is spent inside of RepositoryImpl.login. Half of
this time is spent creating the exception in SessionStats. Therefore, it
would be useful if the recording of the exception could be disabled to
improve the performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-2679) Query engine: cache execution plans
[ https://issues.apache.org/jira/browse/OAK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647632#comment-14647632 ] Joel Richard edited comment on OAK-2679 at 8/3/15 6:46 AM: --- [~mmarth], here are a few numbers about the time which is lost at the moment with finding the best execution plan: Static files: 21% of the request (query time is 22%) Page 1: 9.5% of the request (query time is 10.5%) Page 2: 5% of the request (query time is 5.5%) Login: 65% of the session login time was (Author: joelrich): [~mmarth], here are a few numbers about the time which is lost at the moment with finding the best execution plan: Static files: 21% of the request (query time is 22%) Page 1: 9.5% of the request (query time is 10.5%) Page 2: 5% of the request (query time is 5.5%) Login: 65% of the login time Query engine: cache execution plans --- Key: OAK-2679 URL: https://issues.apache.org/jira/browse/OAK-2679 Project: Jackrabbit Oak Issue Type: Improvement Components: core, query Reporter: Thomas Mueller Assignee: Thomas Mueller Labels: performance Fix For: 1.3.5 Attachments: OAK-2679.patch, executionplancache.patch If there are many indexes, preparing a query can take a long time, in relation to executing the query. The query execution plans can be cached. The cache should be invalidated if there are new indexes, or indexes are changed; a simple solution might be to use a timeout, and / or a manual cache clean via JMX or so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2679) Query engine: cache execution plans
[ https://issues.apache.org/jira/browse/OAK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14651519#comment-14651519 ] Joel Richard commented on OAK-2679: --- Should the execution plan for some query index type change more often, we could also add a getCacheStrategy method to the QueryIndex interface which allows to control whether the execution plan can be cached at all and if so for how long/until when. Query engine: cache execution plans --- Key: OAK-2679 URL: https://issues.apache.org/jira/browse/OAK-2679 Project: Jackrabbit Oak Issue Type: Improvement Components: core, query Reporter: Thomas Mueller Assignee: Thomas Mueller Labels: performance Fix For: 1.3.5 Attachments: OAK-2679.patch, executionplancache.patch If there are many indexes, preparing a query can take a long time, in relation to executing the query. The query execution plans can be cached. The cache should be invalidated if there are new indexes, or indexes are changed; a simple solution might be to use a timeout, and / or a manual cache clean via JMX or so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-2679) Query engine: cache execution plans
[ https://issues.apache.org/jira/browse/OAK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647632#comment-14647632 ] Joel Richard edited comment on OAK-2679 at 8/3/15 6:48 AM: --- [~mmarth], here are a few numbers about the time which is lost at the moment with finding the best execution plan: Static files: 21% of the request (query time is 22%) Page 1: 9.5% of the request (query time is 10.5%) Page 2: 5% of the request (query time is 5.5%) Login: 65% of the repository login time was (Author: joelrich): [~mmarth], here are a few numbers about the time which is lost at the moment with finding the best execution plan: Static files: 21% of the request (query time is 22%) Page 1: 9.5% of the request (query time is 10.5%) Page 2: 5% of the request (query time is 5.5%) Login: 65% of the session login time Query engine: cache execution plans --- Key: OAK-2679 URL: https://issues.apache.org/jira/browse/OAK-2679 Project: Jackrabbit Oak Issue Type: Improvement Components: core, query Reporter: Thomas Mueller Assignee: Thomas Mueller Labels: performance Fix For: 1.3.5 Attachments: OAK-2679.patch, executionplancache.patch If there are many indexes, preparing a query can take a long time, in relation to executing the query. The query execution plans can be cached. The cache should be invalidated if there are new indexes, or indexes are changed; a simple solution might be to use a timeout, and / or a manual cache clean via JMX or so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-3154) Improve SimpleNodeAggregator performance with a NodeState cache
[ https://issues.apache.org/jira/browse/OAK-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard resolved OAK-3154. --- Resolution: Won't Fix Improve SimpleNodeAggregator performance with a NodeState cache --- Key: OAK-3154 URL: https://issues.apache.org/jira/browse/OAK-3154 Project: Jackrabbit Oak Issue Type: Improvement Components: query Affects Versions: 1.3.3 Reporter: Joel Richard Labels: performance I have profiled a query where 16% of the query fetching time is spent inside of SimpleNodeAggregator.isNodeType. In my case, a lot of nodes which are read have overlapping paths. Because the nodes seem to be iterated alphabetically, it would be possible to cache the previous NodeState chain in an array and reuse as much as possible if the previous and current path overlap. This would significantly reduce the query fetching time in cases where a lot of paths are similar. Since the NodeState cache array can be reused for the whole query execution, the possible overhead of it should be negligible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3154) Improve SimpleNodeAggregator performance with a NodeState cache
[ https://issues.apache.org/jira/browse/OAK-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14651837#comment-14651837 ] Joel Richard commented on OAK-3154: --- The following methods could benefit from a similar cache: org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.IndexTaskSpliter#split org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate#updateIndex org.apache.jackrabbit.oak.plugins.observation.NodeObserver#contentChanged However, I think it would make more sense to add a last child node cache in one of the following classes: SecureNodeBuilder, MemoryNodeBuilder or SegmentNodeState. I will describe my idea in OAK-2758 which contains already a similar suggestions. Closing this issue now because SimpleNodeAggregator won't be used much longer. Improve SimpleNodeAggregator performance with a NodeState cache --- Key: OAK-3154 URL: https://issues.apache.org/jira/browse/OAK-3154 Project: Jackrabbit Oak Issue Type: Improvement Components: query Affects Versions: 1.3.3 Reporter: Joel Richard Labels: performance I have profiled a query where 16% of the query fetching time is spent inside of SimpleNodeAggregator.isNodeType. In my case, a lot of nodes which are read have overlapping paths. Because the nodes seem to be iterated alphabetically, it would be possible to cache the previous NodeState chain in an array and reuse as much as possible if the previous and current path overlap. This would significantly reduce the query fetching time in cases where a lot of paths are similar. Since the NodeState cache array can be reused for the whole query execution, the possible overhead of it should be negligible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2758) Performance: Consider caches for MutableTree#getTree and MemoryNodeBuilder#getChildNode
[
https://issues.apache.org/jira/browse/OAK-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Richard updated OAK-2758:
--
Attachment: OAK-2758-SegmentNodeState-lastChildNode-cache.patch
OAK-2758-SecureNodeBuilder-lastChildNode-cache.patch
I have attached two alternative patches which are less intrusive and only cache
the last accessed child node (per SecureNodeBuilder/SegmentNodeState).
Nevertheless, for my test page it improves the rendering performance by 8.5%
and the time which is spent in getChildNode is reduced by 45%.
After using JCR-3870 in Sling (see SLING-4585), it will likely look quite
different though.
Performance: Consider caches for MutableTree#getTree and
MemoryNodeBuilder#getChildNode
---
Key: OAK-2758
URL: https://issues.apache.org/jira/browse/OAK-2758
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Affects Versions: 1.2
Reporter: Joel Richard
Priority: Critical
Labels: performance
Attachments: OAK-2758-SecureNodeBuilder-lastChildNode-cache.patch,
OAK-2758-SegmentNodeState-lastChildNode-cache.patch, experimental_caches.patch
While I was analysing Sling's rendering performance, I noticed that it would
help a lot to implement a cache for the ResourceResolver. For some pages it
almost doubled the rendering performance. This made me wondering whether
Oak's read performance could still be improved.
I noticed that a lot of time is spent in MutableTree#getTree and
MemoryNodeBuilder#getChildNode and have implemented a specific cache for
them. These two caches improve the read performance up to 4 times and do not
break any oak-core tests.
Here the benchmark results for ReadDeepTreeTest:
{code}
Fixtures: Oak-Tar
Admin User: false
Runtime: 5
Num Items: 1000
Concurrency: 1,2,4
Random User: true
Profiling: false
--
Executing benchmarks as admin: false on Oak-Tar
---
# ReadDeepTreeTest , C,min,10%,50%,90%,max,
N
Oak-Tar , 1, 16, 16, 17, 19, 22,
290
Oak-Tar , 2, 23, 29, 44, 68,115,
216
Oak-Tar , 4, 24, 43, 97,154,232,
207
{code}
The same results with my changes:
{code}
# ReadDeepTreeTest , C,min,10%,50%,90%,max,
N
Oak-Tar , 1, 4, 4, 5, 5, 15,
1038
Oak-Tar , 2, 10, 14, 16, 20, 60,
577
Oak-Tar , 4, 13, 27, 32, 40, 69,
605
{code}
I have also implemented another cache for properties, but it didn't really
help and broke some tests.
The experimental patch is attached. It's not meant to be applied, but just to
point out areas with potential for improvement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-2707) Performance: Consider cache for SessionImpl#getNamespacePrefixes
[ https://issues.apache.org/jira/browse/OAK-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard resolved OAK-2707. --- Resolution: Won't Fix Closing because the problem was solved in Sling for the new JcrValueMap and a cache cannot be justified for the deprecated JcrPropertyMap. Performance: Consider cache for SessionImpl#getNamespacePrefixes Key: OAK-2707 URL: https://issues.apache.org/jira/browse/OAK-2707 Project: Jackrabbit Oak Issue Type: Improvement Components: jcr Affects Versions: 1.1.7 Reporter: Joel Richard Priority: Critical Labels: performance Attachments: OAK-2707.patch, Screen Shot 2015-03-30 at 08.38.58.png Sesison#getNamespacePrefixes is heavily used in Sling (see org.apache.sling.jcr.resource.JcrPropertyMap#escapeKeyName) and can easily be called 5000-8000 times per page load. Therefore, it adds 10-15% page rendering overhead for Sling-based websites (see attachment). Would it be possible to add either a global cache or at least a session cache for this method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2679) Query engine: cache execution plans
[ https://issues.apache.org/jira/browse/OAK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647632#comment-14647632 ] Joel Richard commented on OAK-2679: --- [~mmarth], here are a few numbers about the time which is lost at the moment with finding the best execution plan: Static files: 21% of the request (query time is 22%) Page 1: 9.5% of the request (query time is 10.5%) Page 2: 5% of the request (query time is 5.5%) Login: 65% of the login time Query engine: cache execution plans --- Key: OAK-2679 URL: https://issues.apache.org/jira/browse/OAK-2679 Project: Jackrabbit Oak Issue Type: Improvement Components: core, query Reporter: Thomas Mueller Assignee: Thomas Mueller Labels: performance Fix For: 1.3.5 Attachments: OAK-2679.patch, executionplancache.patch If there are many indexes, preparing a query can take a long time, in relation to executing the query. The query execution plans can be cached. The cache should be invalidated if there are new indexes, or indexes are changed; a simple solution might be to use a timeout, and / or a manual cache clean via JMX or so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3154) Improve SimpleNodeAggregator performance with a NodeState cache
[ https://issues.apache.org/jira/browse/OAK-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647776#comment-14647776 ] Joel Richard commented on OAK-3154: --- I have talked to Alex and this class isn't worth improving anymore. However, there are likely other cases with a similar pattern where such a cache might help. I will check the code for it next week and adjust this issue accordingly. Improve SimpleNodeAggregator performance with a NodeState cache --- Key: OAK-3154 URL: https://issues.apache.org/jira/browse/OAK-3154 Project: Jackrabbit Oak Issue Type: Improvement Components: query Affects Versions: 1.3.3 Reporter: Joel Richard Labels: performance I have profiled a query where 16% of the query fetching time is spent inside of SimpleNodeAggregator.isNodeType. In my case, a lot of nodes which are read have overlapping paths. Because the nodes seem to be iterated alphabetically, it would be possible to cache the previous NodeState chain in an array and reuse as much as possible if the previous and current path overlap. This would significantly reduce the query fetching time in cases where a lot of paths are similar. Since the NodeState cache array can be reused for the whole query execution, the possible overhead of it should be negligible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3166) Apply adjustments for newly exported JackrabbitSession#getItemOrNull
[ https://issues.apache.org/jira/browse/OAK-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-3166: -- Summary: Apply adjustments for newly exported JackrabbitSession#getItemOrNull (was: Implement adjustments for newly exported JackrabbitSession#getItemOrNull) Apply adjustments for newly exported JackrabbitSession#getItemOrNull Key: OAK-3166 URL: https://issues.apache.org/jira/browse/OAK-3166 Project: Jackrabbit Oak Issue Type: Improvement Components: jcr Affects Versions: 1.3.3 Reporter: Joel Richard Once the patch for JCR-3870 has been applied, there are a few cosmetic adjustments necessary and test coverage should be increased. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3166) Implement adjustments for newly exported JackrabbitSession#getItemOrNull
Joel Richard created OAK-3166: - Summary: Implement adjustments for newly exported JackrabbitSession#getItemOrNull Key: OAK-3166 URL: https://issues.apache.org/jira/browse/OAK-3166 Project: Jackrabbit Oak Issue Type: Improvement Components: jcr Affects Versions: 1.3.3 Reporter: Joel Richard Once the patch for JCR-3870 has been applied, there are a few cosmetic adjustments necessary and test coverage should be increased. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3166) Apply adjustments for newly exported JackrabbitSession#getItemOrNull
[ https://issues.apache.org/jira/browse/OAK-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-3166: -- Attachment: 0001-OAK-3166-Apply-adjustments-for-newly-exported-Jackra.patch Apply adjustments for newly exported JackrabbitSession#getItemOrNull Key: OAK-3166 URL: https://issues.apache.org/jira/browse/OAK-3166 Project: Jackrabbit Oak Issue Type: Improvement Components: jcr Affects Versions: 1.3.3 Reporter: Joel Richard Attachments: 0001-OAK-3166-Apply-adjustments-for-newly-exported-Jackra.patch Once the patch for JCR-3870 has been applied, there are a few cosmetic adjustments necessary and test coverage should be increased. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2707) Performance: Consider cache for SessionImpl#getNamespacePrefixes
[ https://issues.apache.org/jira/browse/OAK-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14645948#comment-14645948 ] Joel Richard commented on OAK-2707: --- Sling does now cache the namespaces globally for the new JcrValueMap. Unfortunately, the problem remains for the JcrPropertyMap which is part of the Sling API. Performance: Consider cache for SessionImpl#getNamespacePrefixes Key: OAK-2707 URL: https://issues.apache.org/jira/browse/OAK-2707 Project: Jackrabbit Oak Issue Type: Improvement Components: jcr Affects Versions: 1.1.7 Reporter: Joel Richard Priority: Critical Labels: performance Attachments: OAK-2707.patch, Screen Shot 2015-03-30 at 08.38.58.png Sesison#getNamespacePrefixes is heavily used in Sling (see org.apache.sling.jcr.resource.JcrPropertyMap#escapeKeyName) and can easily be called 5000-8000 times per page load. Therefore, it adds 10-15% page rendering overhead for Sling-based websites (see attachment). Would it be possible to add either a global cache or at least a session cache for this method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3154) Improve SimpleNodeAggregator performance with a NodeState cache
[ https://issues.apache.org/jira/browse/OAK-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14646211#comment-14646211 ] Joel Richard commented on OAK-3154: --- [~alex.parvulescu], I was told that this class might not be used much longer. Is that the case? Improve SimpleNodeAggregator performance with a NodeState cache --- Key: OAK-3154 URL: https://issues.apache.org/jira/browse/OAK-3154 Project: Jackrabbit Oak Issue Type: Improvement Components: query Affects Versions: 1.3.3 Reporter: Joel Richard Labels: performance I have profiled a query where 16% of the query fetching time is spent inside of SimpleNodeAggregator.isNodeType. In my case, a lot of nodes which are read have overlapping paths. Because the nodes seem to be iterated alphabetically, it would be possible to cache the previous NodeState chain in an array and reuse as much as possible if the previous and current path overlap. This would significantly reduce the query fetching time in cases where a lot of paths are similar. Since the NodeState cache array can be reused for the whole query execution, the possible overhead of it should be negligible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3153) Make it possible to disable recording of stack trace in SessionStats
Joel Richard created OAK-3153: - Summary: Make it possible to disable recording of stack trace in SessionStats Key: OAK-3153 URL: https://issues.apache.org/jira/browse/OAK-3153 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.3.3 Reporter: Joel Richard For the rendering of some pages we have to create a lot of sessions. Around 9% of the rendering time is spent inside of RepositoryImpl.login. Half of this time is spent creating the exception in SessionStats. Therefore, it would be useful if the recording of the exception could be disabled to improve the performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3154) Improve SimpleNodeAggregator performance with a NodeState cache
Joel Richard created OAK-3154: - Summary: Improve SimpleNodeAggregator performance with a NodeState cache Key: OAK-3154 URL: https://issues.apache.org/jira/browse/OAK-3154 Project: Jackrabbit Oak Issue Type: Improvement Components: query Affects Versions: 1.3.3 Reporter: Joel Richard I have profiled a query where 16% of the query fetching time is spent inside of SimpleNodeAggregator.isNodeType. In my case, a lot of nodes which are read have overlapping paths. Because the nodes seem to be iterated alphabetically, it would be possible to cache the previous NodeState chain in an array and reuse as much as possible if the previous and current path overlap. This would significantly reduce the query fetching time in cases where a lot of paths are similar. Since the NodeState cache array can be reused for the whole query execution, the possible overhead of it should be negligible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2679) Query engine: cache execution plans
[ https://issues.apache.org/jira/browse/OAK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593091#comment-14593091 ] Joel Richard commented on OAK-2679: --- [~mmarth], I would expect that the cache size stays quite stable. But maybe we could limit the cache entries and once the threshold is reached remove the least used execution plans? Query engine: cache execution plans --- Key: OAK-2679 URL: https://issues.apache.org/jira/browse/OAK-2679 Project: Jackrabbit Oak Issue Type: Improvement Components: core, query Reporter: Thomas Mueller Assignee: Thomas Mueller Labels: performance Fix For: 1.3.1, 1.2.4 Attachments: OAK-2679.patch, executionplancache.patch If there are many indexes, preparing a query can take a long time, in relation to executing the query. The query execution plans can be cached. The cache should be invalidated if there are new indexes, or indexes are changed; a simple solution might be to use a timeout, and / or a manual cache clean via JMX or so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2707) Performance: Consider cache for SessionImpl#getNamespacePrefixes
[ https://issues.apache.org/jira/browse/OAK-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-2707: -- Attachment: OAK-2707.patch [~anchela], in Sling the namespaces are read when you read the first property of a resource which contains a colon. Thanks to some other changes, we could reduce the overhead already to around 5% of the rendering time, but it would be great if we could remove it entirely. I have attached a patch with a namespace cache. Can you please review it? It still contains a few todos with things I wasn't sure about. Performance: Consider cache for SessionImpl#getNamespacePrefixes Key: OAK-2707 URL: https://issues.apache.org/jira/browse/OAK-2707 Project: Jackrabbit Oak Issue Type: Improvement Components: jcr Affects Versions: 1.1.7 Reporter: Joel Richard Priority: Critical Labels: performance Fix For: 1.3.1 Attachments: OAK-2707.patch, Screen Shot 2015-03-30 at 08.38.58.png Sesison#getNamespacePrefixes is heavily used in Sling (see org.apache.sling.jcr.resource.JcrPropertyMap#escapeKeyName) and can easily be called 5000-8000 times per page load. Therefore, it adds 10-15% page rendering overhead for Sling-based websites (see attachment). Would it be possible to add either a global cache or at least a session cache for this method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2758) Performance: Consider caches for MutableTree#getTree and MemoryNodeBuilder#getChildNode
[
https://issues.apache.org/jira/browse/OAK-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Richard updated OAK-2758:
--
Attachment: experimental_caches.patch
Performance: Consider caches for MutableTree#getTree and
MemoryNodeBuilder#getChildNode
---
Key: OAK-2758
URL: https://issues.apache.org/jira/browse/OAK-2758
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Affects Versions: 1.2
Reporter: Joel Richard
Priority: Critical
Labels: performance
Attachments: experimental_caches.patch
While I was analysing Sling's rendering performance, I noticed that it would
help a lot to implement a cache for the ResourceResolver. For some pages it
almost doubled the rendering performance. This made me wondering whether
Oak's read performance could still be improved.
I noticed that a lot of time is spent in MutableTree#getTree and
MemoryNodeBuilder#getChildNode and have implemented a specific cache for
them. These two caches improve the read performance up to 4 times and do not
break any oak-core tests.
Here the benchmark results for ReadDeepTreeTest:
{code}
Fixtures: Oak-Tar
Admin User: false
Runtime: 5
Num Items: 1000
Concurrency: 1,2,4
Random User: true
Profiling: false
--
Executing benchmarks as admin: false on Oak-Tar
---
# ReadDeepTreeTest , C,min,10%,50%,90%,max,
N
Oak-Tar , 1, 16, 16, 17, 19, 22,
290
Oak-Tar , 2, 23, 29, 44, 68,115,
216
Oak-Tar , 4, 24, 43, 97,154,232,
207
{code}
The same results with my changes:
{code}
# ReadDeepTreeTest , C,min,10%,50%,90%,max,
N
Oak-Tar , 1, 4, 4, 5, 5, 15,
1038
Oak-Tar , 2, 10, 14, 16, 20, 60,
577
Oak-Tar , 4, 13, 27, 32, 40, 69,
605
{code}
I have also implemented another cache for properties, but it didn't really
help and broke some tests.
The experimental patch is attached. It's not meant to be applied, but just to
point out areas with potential for improvement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2758) Performance:
Joel Richard created OAK-2758:
-
Summary: Performance:
Key: OAK-2758
URL: https://issues.apache.org/jira/browse/OAK-2758
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Affects Versions: 1.2
Reporter: Joel Richard
Priority: Critical
While I was analysing Sling's rendering performance, I noticed that it would
help a lot to implement a cache for the ResourceResolver. For some pages it
almost doubled the rendering performance. This made me wondering whether Oak's
read performance could still be improved.
I noticed that a lot of time is spent in MutableTree#getTree and
MemoryNodeBuilder#getChildNode and have implemented a specific cache for them.
These two caches improve the read performance up to 4 times and do not break
any oak-core tests.
Here the benchmark results for ReadDeepTreeTest:
{code}
Fixtures: Oak-Tar
Admin User: false
Runtime: 5
Num Items: 1000
Concurrency: 1,2,4
Random User: true
Profiling: false
--
Executing benchmarks as admin: false on Oak-Tar
---
# ReadDeepTreeTest , C,min,10%,50%,90%,max,
N
Oak-Tar , 1, 16, 16, 17, 19, 22,
290
Oak-Tar , 2, 23, 29, 44, 68,115,
216
Oak-Tar , 4, 24, 43, 97,154,232,
207
{code}
The same results with my changes:
{code}
# ReadDeepTreeTest , C,min,10%,50%,90%,max,
N
Oak-Tar , 1, 4, 4, 5, 5, 15,
1038
Oak-Tar , 2, 10, 14, 16, 20, 60,
577
Oak-Tar , 4, 13, 27, 32, 40, 69,
605
{code}
I have also implemented another cache for properties, but it didn't really help
and broke some tests.
The experimental patch is attached. It's not meant to be applied, but just to
point out areas with potential for improvement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2758) Performance: Consider caches for MutableTree#getTree and MemoryNodeBuilder#getChildNode
[
https://issues.apache.org/jira/browse/OAK-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Richard updated OAK-2758:
--
Summary: Performance: Consider caches for MutableTree#getTree and
MemoryNodeBuilder#getChildNode (was: Performance: )
Performance: Consider caches for MutableTree#getTree and
MemoryNodeBuilder#getChildNode
---
Key: OAK-2758
URL: https://issues.apache.org/jira/browse/OAK-2758
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Affects Versions: 1.2
Reporter: Joel Richard
Priority: Critical
Labels: performance
Attachments: experimental_caches.patch
While I was analysing Sling's rendering performance, I noticed that it would
help a lot to implement a cache for the ResourceResolver. For some pages it
almost doubled the rendering performance. This made me wondering whether
Oak's read performance could still be improved.
I noticed that a lot of time is spent in MutableTree#getTree and
MemoryNodeBuilder#getChildNode and have implemented a specific cache for
them. These two caches improve the read performance up to 4 times and do not
break any oak-core tests.
Here the benchmark results for ReadDeepTreeTest:
{code}
Fixtures: Oak-Tar
Admin User: false
Runtime: 5
Num Items: 1000
Concurrency: 1,2,4
Random User: true
Profiling: false
--
Executing benchmarks as admin: false on Oak-Tar
---
# ReadDeepTreeTest , C,min,10%,50%,90%,max,
N
Oak-Tar , 1, 16, 16, 17, 19, 22,
290
Oak-Tar , 2, 23, 29, 44, 68,115,
216
Oak-Tar , 4, 24, 43, 97,154,232,
207
{code}
The same results with my changes:
{code}
# ReadDeepTreeTest , C,min,10%,50%,90%,max,
N
Oak-Tar , 1, 4, 4, 5, 5, 15,
1038
Oak-Tar , 2, 10, 14, 16, 20, 60,
577
Oak-Tar , 4, 13, 27, 32, 40, 69,
605
{code}
I have also implemented another cache for properties, but it didn't really
help and broke some tests.
The experimental patch is attached. It's not meant to be applied, but just to
point out areas with potential for improvement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2724) Export SessionImpl#getItemOrNull in JackrabbitSession
Joel Richard created OAK-2724: - Summary: Export SessionImpl#getItemOrNull in JackrabbitSession Key: OAK-2724 URL: https://issues.apache.org/jira/browse/OAK-2724 Project: Jackrabbit Oak Issue Type: Improvement Components: oak-jcr Affects Versions: 1.1.8 Reporter: Joel Richard Priority: Critical getItemOrNull should be exported in JackrabbitSession. This would allow to combine itemExists and getItem in Sling which would reduce the rendering time by 8%. See the following mail thread for more information: http://mail-archives.apache.org/mod_mbox/jackrabbit-oak-dev/201504.mbox/browser -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2707) Performance: Consider cache for SessionImpl#getNamespacePrefixes
[ https://issues.apache.org/jira/browse/OAK-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390085#comment-14390085 ] Joel Richard commented on OAK-2707: --- [~mduerig], Sling could definitely improve the caching for getNamespacePrefixes (although it would require some bigger changes as well since the cache would have to be moved from the JcrPropertyMap to another level). It's a good question why so many JcrPropertyMap are created per request and I will try to figure that out, but I am pretty sure that this is just because so many resources are read (which is much harder to change than implementing a cache). Performance: Consider cache for SessionImpl#getNamespacePrefixes Key: OAK-2707 URL: https://issues.apache.org/jira/browse/OAK-2707 Project: Jackrabbit Oak Issue Type: Improvement Components: oak-jcr Affects Versions: 1.1.7 Reporter: Joel Richard Priority: Critical Labels: performance Fix For: 1.3.0 Attachments: Screen Shot 2015-03-30 at 08.38.58.png Sesison#getNamespacePrefixes is heavily used in Sling (see org.apache.sling.jcr.resource.JcrPropertyMap#escapeKeyName) and can easily be called 5000-8000 times per page load. Therefore, it adds 10-15% page rendering overhead for Sling-based websites (see attachment). Would it be possible to add either a global cache or at least a session cache for this method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2707) Performance: Consider cache for SessionImpl#getNamespacePrefixes
[ https://issues.apache.org/jira/browse/OAK-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390099#comment-14390099 ] Joel Richard commented on OAK-2707: --- Most of the time in getNamespacePrefixes is spent in SegmentPropertyState.getValue which will likely benefit from fixing OAK-2688. Performance: Consider cache for SessionImpl#getNamespacePrefixes Key: OAK-2707 URL: https://issues.apache.org/jira/browse/OAK-2707 Project: Jackrabbit Oak Issue Type: Improvement Components: oak-jcr Affects Versions: 1.1.7 Reporter: Joel Richard Priority: Critical Labels: performance Fix For: 1.3.0 Attachments: Screen Shot 2015-03-30 at 08.38.58.png Sesison#getNamespacePrefixes is heavily used in Sling (see org.apache.sling.jcr.resource.JcrPropertyMap#escapeKeyName) and can easily be called 5000-8000 times per page load. Therefore, it adds 10-15% page rendering overhead for Sling-based websites (see attachment). Would it be possible to add either a global cache or at least a session cache for this method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2707) Performance: Consider cache for SessionImpl#getNamespacePrefixes
[ https://issues.apache.org/jira/browse/OAK-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390061#comment-14390061 ] Joel Richard commented on OAK-2707: --- [~anchela], no, I don't have a benchmark for it. The method itself is pretty fast and does not need any further optimisation, but since it is called several thousand times, it consumes unexpectedly much time. Caching them for the current session might be the simplest solution and would fix the problem. This would only be a problem for cases where it is expected that namespace changes are reflected immediately. Should that be a real problem, the cache could be evicted in session.save(). Performance: Consider cache for SessionImpl#getNamespacePrefixes Key: OAK-2707 URL: https://issues.apache.org/jira/browse/OAK-2707 Project: Jackrabbit Oak Issue Type: Improvement Components: oak-jcr Affects Versions: 1.1.7 Reporter: Joel Richard Priority: Critical Labels: performance Fix For: 1.3.0 Attachments: Screen Shot 2015-03-30 at 08.38.58.png Sesison#getNamespacePrefixes is heavily used in Sling (see org.apache.sling.jcr.resource.JcrPropertyMap#escapeKeyName) and can easily be called 5000-8000 times per page load. Therefore, it adds 10-15% page rendering overhead for Sling-based websites (see attachment). Would it be possible to add either a global cache or at least a session cache for this method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2707) Performance: SessionImpl#getNamespacePrefixes should be cached
[ https://issues.apache.org/jira/browse/OAK-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-2707: -- Attachment: Screen Shot 2015-03-30 at 08.38.58.png Performance: SessionImpl#getNamespacePrefixes should be cached -- Key: OAK-2707 URL: https://issues.apache.org/jira/browse/OAK-2707 Project: Jackrabbit Oak Issue Type: Improvement Components: oak-jcr Affects Versions: 1.1.7 Reporter: Joel Richard Priority: Critical Labels: performance Attachments: Screen Shot 2015-03-30 at 08.38.58.png Sesison#getNamespacePrefixes is heavily used in Sling (see org.apache.sling.jcr.resource.JcrPropertyMap#escapeKeyName) and can easily be called 5000-8000 times per page load. Therefore, it adds 10-15% page rendering overhead for Sling-based websites. Would it be possible to add either a global cache or at least a session cache for this method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2707) Performance: SessionImpl#getNamespacePrefixes should be cached
[ https://issues.apache.org/jira/browse/OAK-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-2707: -- Description: Sesison#getNamespacePrefixes is heavily used in Sling (see org.apache.sling.jcr.resource.JcrPropertyMap#escapeKeyName) and can easily be called 5000-8000 times per page load. Therefore, it adds 10-15% page rendering overhead for Sling-based websites (see attachment). Would it be possible to add either a global cache or at least a session cache for this method? was: Sesison#getNamespacePrefixes is heavily used in Sling (see org.apache.sling.jcr.resource.JcrPropertyMap#escapeKeyName) and can easily be called 5000-8000 times per page load. Therefore, it adds 10-15% page rendering overhead for Sling-based websites. Would it be possible to add either a global cache or at least a session cache for this method? Performance: SessionImpl#getNamespacePrefixes should be cached -- Key: OAK-2707 URL: https://issues.apache.org/jira/browse/OAK-2707 Project: Jackrabbit Oak Issue Type: Improvement Components: oak-jcr Affects Versions: 1.1.7 Reporter: Joel Richard Priority: Critical Labels: performance Attachments: Screen Shot 2015-03-30 at 08.38.58.png Sesison#getNamespacePrefixes is heavily used in Sling (see org.apache.sling.jcr.resource.JcrPropertyMap#escapeKeyName) and can easily be called 5000-8000 times per page load. Therefore, it adds 10-15% page rendering overhead for Sling-based websites (see attachment). Would it be possible to add either a global cache or at least a session cache for this method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-2707) Performance: SessionImpl#getNamespacePrefixes should be cached
Joel Richard created OAK-2707: - Summary: Performance: SessionImpl#getNamespacePrefixes should be cached Key: OAK-2707 URL: https://issues.apache.org/jira/browse/OAK-2707 Project: Jackrabbit Oak Issue Type: Improvement Components: oak-jcr Affects Versions: 1.1.7 Reporter: Joel Richard Priority: Critical Sesison#getNamespacePrefixes is heavily used in Sling (see org.apache.sling.jcr.resource.JcrPropertyMap#escapeKeyName) and can easily be called 5000-8000 times per page load. Therefore, it adds 10-15% page rendering overhead for Sling-based websites. Would it be possible to add either a global cache or at least a session cache for this method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2688) Segment.readString optimization
[ https://issues.apache.org/jira/browse/OAK-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388405#comment-14388405 ] Joel Richard commented on OAK-2688: --- In my sampling profiling sessions about 5-15% of the time was spent in ConcurrentHashMap.get (for readString and readTemplate together). Segment.readString optimization --- Key: OAK-2688 URL: https://issues.apache.org/jira/browse/OAK-2688 Project: Jackrabbit Oak Issue Type: Improvement Components: oak-core Reporter: Thomas Mueller Assignee: Thomas Mueller Fix For: 1.3.0 Attachments: OAK-2688.patch The method Segment.readString is called a lot, and even a small optimization would improve performance for some use cases. Currently it uses a concurrent hash map to cache strings. It might be possible to speed up this cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2679) Query engine: cache execution plans
[ https://issues.apache.org/jira/browse/OAK-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Richard updated OAK-2679: -- Attachment: executionplancache.patch I have attached a patch with some changes to test the impact of a cache for getBestSelectorExecutionPlan. In my test case it has improved the performance by ~50%. [~tmueller], one problem of having a cache which maps a query string to a execution plan is that the cache would be quickly filled with garbage if there are variables in it. Query engine: cache execution plans --- Key: OAK-2679 URL: https://issues.apache.org/jira/browse/OAK-2679 Project: Jackrabbit Oak Issue Type: Improvement Components: core, query Reporter: Thomas Mueller Assignee: Thomas Mueller Fix For: 1.3.0 Attachments: executionplancache.patch If there are many indexes, preparing a query can take a long time, in relation to executing the query. The query execution plans can be cached. The cache should be invalidated if there are new indexes, or indexes are changed; a simple solution might be to use a timeout, and / or a manual cache clean via JMX or so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
