[jira] [Commented] (OAK-4368) Excerpt extraction from the Lucene index should be more selective
[
https://issues.apache.org/jira/browse/OAK-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925561#comment-15925561
]
Chetan Mehrotra commented on OAK-4368:
--
Note that to make effective use of this enhancement ensure that
* Oak version is >= fix version of this issue
* Indexes are reindexed such that the offset position is also index thus
enabling use for newer approach
> Excerpt extraction from the Lucene index should be more selective
> -
>
> Key: OAK-4368
> URL: https://issues.apache.org/jira/browse/OAK-4368
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Affects Versions: 1.0.30, 1.2.14, 1.4.2, 1.5.2
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.2.17, 1.4.4, 1.5.4, 1.6.0
>
> Attachments: OAK-4368.0.patch
>
>
> Lucene index can be used in order to extract _rep:excerpt_ using
> {{Highlighter}}.
> The current implementation may suffer performance issues when the result set
> of the original query contains a lot of results, each of them possibly
> containing lots of (stored) properties that get passed to the highlighter in
> order to try to extract the excerpt; such a process doesn't stop as soon as
> the first excerpt is found so that excerpt is composed using text from all
> stored properties in all results (if there's a match on the query).
> While we can accept some cost of extracting excerpt at query time (whereas it
> was generated at excerpt retrieval time before OAK-3580, e.g. via
> _row.getValue("rep:excerpt")_) , that should be bounded and mitigated as much
> as possible.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OAK-2556) do intermediate commit during async indexing
[ https://issues.apache.org/jira/browse/OAK-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-2556: - Fix Version/s: 1.8 > do intermediate commit during async indexing > > > Key: OAK-2556 > URL: https://issues.apache.org/jira/browse/OAK-2556 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.0.11 >Reporter: Stefan Egli > Labels: resilience > Fix For: 1.8 > > > A recent issue found at a customer unveils a potential issue with the async > indexer. Reading the AsyncIndexUpdate.updateIndex it looks like it is doing > the entire update of the async indexer *in one go*, ie in one commit. > When there is - for some reason - however, a huge diff that the async indexer > has to process, the 'one big commit' can become gigantic. There is no limit > to the size of the commit in fact. > So the suggestion is to do intermediate commits while the async indexer is > going on. The reason this is acceptable is the fact that by doing async > indexing, that index is anyway not 100% up-to-date - so it would not make > much of a difference if it would commit after every 100 or 1000 changes > either. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OAK-3680) Partial re-index from last known good state
[ https://issues.apache.org/jira/browse/OAK-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-3680: - Fix Version/s: 1.8 > Partial re-index from last known good state > --- > > Key: OAK-3680 > URL: https://issues.apache.org/jira/browse/OAK-3680 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: lucene, query >Affects Versions: 1.0, 1.2 >Reporter: Michael Marth >Assignee: Chetan Mehrotra > Labels: resilience > Fix For: 1.8 > > > ATM indexes break (by whatever circumstances) users need to perform a full > re-index. Depending on the size off the repository this can take a long time. > If the user knows that the indexes were in a good state at a certain revision > in the past then it would be very useful, if the user could trigger a > "partial" re-index where only the content added after a certain revision was > updated in the index. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OAK-2808) Active deletion of 'deleted' Lucene index files from DataStore without relying on full scale Blob GC
[ https://issues.apache.org/jira/browse/OAK-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-2808: - Fix Version/s: 1.8 > Active deletion of 'deleted' Lucene index files from DataStore without > relying on full scale Blob GC > > > Key: OAK-2808 > URL: https://issues.apache.org/jira/browse/OAK-2808 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Reporter: Chetan Mehrotra >Assignee: Thomas Mueller > Labels: datastore, performance > Fix For: 1.8 > > Attachments: copyonread-stats.png, OAK-2808-1.patch > > > With storing of Lucene index files within DataStore our usage pattern > of DataStore has changed between JR2 and Oak. > With JR2 the writes were mostly application based i.e. if application > stores a pdf/image file then that would be stored in DataStore. JR2 by > default would not write stuff to DataStore. Further in deployment > where large number of binary content is present then systems tend to > share the DataStore to avoid duplication of storage. In such cases > running Blob GC is a non trivial task as it involves a manual step and > coordination across multiple deployments. Due to this systems tend to > delay frequency of GC > Now with Oak apart from application the Oak system itself *actively* > uses the DataStore to store the index files for Lucene and there the > churn might be much higher i.e. frequency of creation and deletion of > index file is lot higher. This would accelerate the rate of garbage > generation and thus put lot more pressure on the DataStore storage > requirements. > Discussion thread http://markmail.org/thread/iybd3eq2bh372zrl -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OAK-5250) Avoid temporary files on re-index
[ https://issues.apache.org/jira/browse/OAK-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-5250: - Labels: performance (was: ) > Avoid temporary files on re-index > - > > Key: OAK-5250 > URL: https://issues.apache.org/jira/browse/OAK-5250 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Reporter: Marcel Reutegger > Labels: performance > Fix For: 1.8 > > > A lucene index with copy-on-write writes back files to the repository > asynchronously and the implementation may skip files that are created on the > local filesystem and shortly after deleted again. For the re-index case this > could be further improved. All intermediate lucene segment files that are > later merged together can be skipped if the asynchronous write back is > disabled and only the final files are copied to the repository when the > directory is closed. This would avoid a lot of garbage in the blob store. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OAK-5434) Skip processing of queued changes if async index update is detected in ExternalIndexObserver
[ https://issues.apache.org/jira/browse/OAK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-5434: - Labels: performance (was: ) > Skip processing of queued changes if async index update is detected in > ExternalIndexObserver > > > Key: OAK-5434 > URL: https://issues.apache.org/jira/browse/OAK-5434 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Labels: performance > Fix For: 1.8 > > > ExternalIndexObserver is currently backed by a queue (its wrapped in > BackgroundObserver). Currently it processed the changes one by one as > received from the queue. If this processing takes long time then its possible > that it would lag behind the async indexing cycle. > So ExternalIndexObserver may be busy indexing changes from [r1-r2] but async > indexing is already done indexing changes upto r3 (r3 > r2) and IndexTracker > would move to newer index version. In such case work done by > ExternalIndexObserver is wasted. > This can be optimized by ensuring that ExternalIndexObserver can see the > lastIndexTo of :async as per latest entry in queue. If that is newer than one > its processing then it can skip processing the queue entry and thus free up > space in queue -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OAK-5927) Load excerpt lazily
[
https://issues.apache.org/jira/browse/OAK-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-5927:
-
Labels: performance (was: )
> Load excerpt lazily
> ---
>
> Key: OAK-5927
> URL: https://issues.apache.org/jira/browse/OAK-5927
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
> Labels: performance
> Fix For: 1.8
>
>
> Currently LucenePropertyIndex loads the excerpt eagerly in batch as part of
> loadDocs call. The load docs batch size doubles starting from 50 (max 100k)
> as more data is read.
> We should look into ways to make the excerpt loaded lazily as and when caller
> ask for excerpt.
> Note that currently the excerpt are only loaded when query request for
> excerpt i.e. there is a not null property restriction for {{rep:excerpt}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OAK-4934) Query shapes for JCR Query
[
https://issues.apache.org/jira/browse/OAK-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-4934:
-
Fix Version/s: 1.8
> Query shapes for JCR Query
> --
>
> Key: OAK-4934
> URL: https://issues.apache.org/jira/browse/OAK-4934
> Project: Jackrabbit Oak
> Issue Type: Wish
> Components: query
>Reporter: Chetan Mehrotra
> Fix For: 1.8
>
>
> For certain requirements it would be good to have a notion/support to deduce
> query shape [1]
> {quote}
> A combination of query predicate, sort, and projection specifications.
> For the query predicate, only the structure of the predicate, including the
> field names, are significant; the values in the query predicate are
> insignificant. As such, a query predicate \{ type: 'food' \} is equivalent to
> the query predicate \{ type: 'utensil' \} for a query shape.
> {quote}
> So transforming that to Oak the shape should represent a JCR-SQL2 query
> string (xpath query gets transformed to SQL2) which is a *canonical*
> representation of actual query ignoring the property restriction values.
> Example we have 2 queries
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'published'
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'disabled'
> The query shape would be
> SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
> The plan for query having given shape would remain same irrespective of value
> of property restrictions. Path restriction can cause some difference though
> The shape can then be used for
> * Stats Collection - Currently stats collection gets overflown if same query
> with different value gets invoked
> * Allow configuring hints - See support in Mongo [2] for an example. One
> specify via config that for a query of such and such shape this index should
> be used
> * Less noisy diagnostics - If a query gets invoked with bad plan the QE can
> log the warning once instead of logging it for each query invocation
> involving different values.
> [1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
> [2] https://docs.mongodb.com/manual/reference/command/planCacheSetFilter/
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OAK-5511) Reduce lookup for oak:index node under each changed node
[
https://issues.apache.org/jira/browse/OAK-5511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-5511:
-
Component/s: query
> Reduce lookup for oak:index node under each changed node
>
>
> Key: OAK-5511
> URL: https://issues.apache.org/jira/browse/OAK-5511
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, query
>Reporter: Chetan Mehrotra
> Labels: performance
> Fix For: 1.8
>
>
> Currently {{IndexUpdate}} does a lookup for {{oak:index}} node under each
> changed node. This is done to pickup index definitions and create IndexEditor
> based on those so as to index content under that subtree.
> This lookup results in extra remote calls on DocumentNodeStore based setup as
> for non leaf nodes NodeStore has to check from remote storage to determine if
> {{oak:index}} node is present or not.
> This lookup can be avoided by
> # Having an {{Editor}} which adds a hidden property {{:oak-index-present}} in
> parent node upon addition of {{oak:index}} as a child node
> # IndexUpdate would then does a lookup for {{oak:index}} node only if such a
> hidden property is found
> For upgrade we would have some logic which would make use of Nodetype index
> to identify all such nodes and mark them
> Discussion
> [thread|https://lists.apache.org/thread.html/70d50f950d7fc25bc1bbb41527f5672825f8cf2b238f54df2966@%3Coak-dev.jackrabbit.apache.org%3E]
> on oak-dev
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OAK-5931) Inconsistent behaviour when removing nodes with rep:policy subnodes for users without modify ACL permissions
[
https://issues.apache.org/jira/browse/OAK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925293#comment-15925293
]
Pierre Tager commented on OAK-5931:
---
[~alexxx] [~dhasler] making sure this is on your radar.
> Inconsistent behaviour when removing nodes with rep:policy subnodes for
> users without modify ACL permissions
> -
>
> Key: OAK-5931
> URL: https://issues.apache.org/jira/browse/OAK-5931
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: security
>Affects Versions: 1.4.14, 1.6.1
>Reporter: Tom Blackford
> Attachments: ACLTest.java
>
>
> If a session (without rep:modifyAccessControl) removes a node with a
> rep:policy subnode and then recreates it within the same save (without the
> rep:policy subnode) the commit diff will mistake the action for the removal
> of the ACL, which this session is not authorised to do.
> If the session is saved prior to recreating the node, both saves (after
> remove and after recreate) will succeed.
> From discussion with angela:
> {quote}
> the diff mechanism used within Root.commit cannot distinguish between the
> removal of a policy or the replace of the access controlled node with one
> that doesn't have the policy set. within that diff it looks like the removal
> of the policy node
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OAK-5931) Inconsistent behaviour when removing nodes with rep:policy subnodes for users without modify ACL permissions
[
https://issues.apache.org/jira/browse/OAK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom Blackford updated OAK-5931:
---
Attachment: ACLTest.java
Adding test case showing the different behaviours.
> Inconsistent behaviour when removing nodes with rep:policy subnodes for
> users without modify ACL permissions
> -
>
> Key: OAK-5931
> URL: https://issues.apache.org/jira/browse/OAK-5931
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: security
>Affects Versions: 1.4.14, 1.6.1
>Reporter: Tom Blackford
> Attachments: ACLTest.java
>
>
> If a session (without rep:modifyAccessControl) removes a node with a
> rep:policy subnode and then recreates it within the same save (without the
> rep:policy subnode) the commit diff will mistake the action for the removal
> of the ACL, which this session is not authorised to do.
> If the session is saved prior to recreating the node, both saves (after
> remove and after recreate) will succeed.
> From discussion with angela:
> {quote}
> the diff mechanism used within Root.commit cannot distinguish between the
> removal of a policy or the replace of the access controlled node with one
> that doesn't have the policy set. within that diff it looks like the removal
> of the policy node
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (OAK-5931) Inconsistent behaviour when removing nodes with rep:policy subnodes for users without modify ACL permissions
Tom Blackford created OAK-5931:
--
Summary: Inconsistent behaviour when removing nodes with
rep:policy subnodes for users without modify ACL permissions
Key: OAK-5931
URL: https://issues.apache.org/jira/browse/OAK-5931
Project: Jackrabbit Oak
Issue Type: Bug
Components: security
Affects Versions: 1.6.1, 1.4.14
Reporter: Tom Blackford
If a session (without rep:modifyAccessControl) removes a node with a rep:policy
subnode and then recreates it within the same save (without the rep:policy
subnode) the commit diff will mistake the action for the removal of the ACL,
which this session is not authorised to do.
If the session is saved prior to recreating the node, both saves (after remove
and after recreate) will succeed.
>From discussion with angela:
{quote}
the diff mechanism used within Root.commit cannot distinguish between the
removal of a policy or the replace of the access controlled node with one that
doesn't have the policy set. within that diff it looks like the removal of the
policy node
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OAK-5930) incorrect test assumption in CacheConsistencyTestBase wrt batching
[
https://issues.apache.org/jira/browse/OAK-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-5930:
Description:
When testing
{noformat}
ds.createOrUpdate(Collection.NODES, Lists.newArrayList(up1, up2, up3));
{noformat}
the test case assumes that all three IDs ought to be updated. However, this is
not true if the DocumentStore doesn't support batching here (RDB in OAK <=
1.2), or if batching is disabled
({{org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore.BATCHUPDATES}}
system property).
was:
When testig
{{noformat}}
ds.createOrUpdate(Collection.NODES, Lists.newArrayList(up1, up2, up3));
{{noformat}}
the test case assumes that all three IDs ought to be updated. However, this is
not true if the DocumentStore doesn't support batching here (RDB in OAK <=
1.2), or if batching is disabled
({{org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore.BATCHUPDATES}}
system property).
> incorrect test assumption in CacheConsistencyTestBase wrt batching
> --
>
> Key: OAK-5930
> URL: https://issues.apache.org/jira/browse/OAK-5930
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>
> When testing
> {noformat}
> ds.createOrUpdate(Collection.NODES, Lists.newArrayList(up1, up2, up3));
> {noformat}
> the test case assumes that all three IDs ought to be updated. However, this
> is not true if the DocumentStore doesn't support batching here (RDB in OAK <=
> 1.2), or if batching is disabled
> ({{org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore.BATCHUPDATES}}
> system property).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (OAK-5930) incorrect test assumption in CacheConsistencyTestBase wrt batching
Julian Reschke created OAK-5930:
---
Summary: incorrect test assumption in CacheConsistencyTestBase wrt
batching
Key: OAK-5930
URL: https://issues.apache.org/jira/browse/OAK-5930
Project: Jackrabbit Oak
Issue Type: Bug
Components: documentmk
Reporter: Julian Reschke
Assignee: Julian Reschke
When testig
{{noformat}}
ds.createOrUpdate(Collection.NODES, Lists.newArrayList(up1, up2, up3));
{{noformat}}
the test case assumes that all three IDs ought to be updated. However, this is
not true if the DocumentStore doesn't support batching here (RDB in OAK <=
1.2), or if batching is disabled
({{org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore.BATCHUPDATES}}
system property).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (OAK-5929) Redundant test for null with AuthorizableImpl.checkValidTree implementations
[ https://issues.apache.org/jira/browse/OAK-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angela resolved OAK-5929. - Resolution: Fixed Fix Version/s: 1.8 1.7.0 Committed revision 1786912. > Redundant test for null with AuthorizableImpl.checkValidTree implementations > > > Key: OAK-5929 > URL: https://issues.apache.org/jira/browse/OAK-5929 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Reporter: angela >Assignee: angela >Priority: Minor > Fix For: 1.7.0, 1.8 > > > spotted while working on OAK-5882 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (OAK-5929) Redundant test for null with AuthorizableImpl.checkValidTree implementations
angela created OAK-5929: --- Summary: Redundant test for null with AuthorizableImpl.checkValidTree implementations Key: OAK-5929 URL: https://issues.apache.org/jira/browse/OAK-5929 Project: Jackrabbit Oak Issue Type: Improvement Components: core Reporter: angela Assignee: angela Priority: Minor spotted while working on OAK-5882 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5634) Expose IOMonitor stats via JMX
[ https://issues.apache.org/jira/browse/OAK-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924129#comment-15924129 ] Michael Dürig commented on OAK-5634: I scheduled this to 1.7.0 as I think it is a prerequisite for OAK-5859 > Expose IOMonitor stats via JMX > -- > > Key: OAK-5634 > URL: https://issues.apache.org/jira/browse/OAK-5634 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Alex Parvulescu >Assignee: Alex Parvulescu >Priority: Minor > Labels: monitoring > Fix For: 1.7.0, 1.8 > > > Followup of OAK-5632 and OAK-5631, to expose the collected data via JMX for > external use. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5859) Analyse and reduce IO amplification by OS
[ https://issues.apache.org/jira/browse/OAK-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924126#comment-15924126 ] Michael Dürig commented on OAK-5859: OAK-5634 is a prerequisite for this issue to be able to easily monitor IO within the JVM. > Analyse and reduce IO amplification by OS > - > > Key: OAK-5859 > URL: https://issues.apache.org/jira/browse/OAK-5859 > Project: Jackrabbit Oak > Issue Type: Task > Components: segment-tar >Reporter: Michael Dürig > Fix For: 1.7.2 > > > Certain operation system settings might result in too much data actually > being read from disk causing early setting on of thrashing. E.g. transparent > huge pages or too big read aheads might be contra productive in combination > with the TarMKs memory mapping model. > * Determine the ratio of data being read by the TarMK and actual data being > read from disk. Determine the impact of relevant OS parameters (e.g. > transparent huge pages) on this ratio. > * Compare memory mapped mode with file IO mode with an accordingly increased > segment cache. This would move prediction of what is likely to be read next > from the OS layer into our segment cache eviction strategy. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OAK-5634) Expose IOMonitor stats via JMX
[ https://issues.apache.org/jira/browse/OAK-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-5634: --- Fix Version/s: 1.7.0 > Expose IOMonitor stats via JMX > -- > > Key: OAK-5634 > URL: https://issues.apache.org/jira/browse/OAK-5634 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Alex Parvulescu >Assignee: Alex Parvulescu >Priority: Minor > Labels: monitoring > Fix For: 1.7.0, 1.8 > > > Followup of OAK-5632 and OAK-5631, to expose the collected data via JMX for > external use. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5634) Expose IOMonitor stats via JMX
[
https://issues.apache.org/jira/browse/OAK-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924121#comment-15924121
]
Michael Dürig commented on OAK-5634:
I adjusted the title as I'm not sure whether {{FileStoreStatsMBean}} is the
right place to expose this or whether there is a better place to do so.
> Expose IOMonitor stats via JMX
> --
>
> Key: OAK-5634
> URL: https://issues.apache.org/jira/browse/OAK-5634
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>Priority: Minor
> Labels: monitoring
> Fix For: 1.8
>
>
> Followup of OAK-5632 and OAK-5631, to expose the collected data via JMX for
> external use.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OAK-5634) Expose IOMonitor stats via JMX
[ https://issues.apache.org/jira/browse/OAK-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-5634: --- Summary: Expose IOMonitor stats via JMX (was: Expose IOMonitor stats as part of the FileStoreStats mbean) > Expose IOMonitor stats via JMX > -- > > Key: OAK-5634 > URL: https://issues.apache.org/jira/browse/OAK-5634 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Alex Parvulescu >Assignee: Alex Parvulescu >Priority: Minor > Labels: monitoring > Fix For: 1.8 > > > Followup of OAK-5632 and OAK-5631, to expose the collected data via JMX for > external use. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5860) Compressed segments
[ https://issues.apache.org/jira/browse/OAK-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924041#comment-15924041 ] Michael Dürig commented on OAK-5860: Compressing the segments might not make too much sense when using memory mapped access as in that case it would defeat the purpose of the direct buffer access (i.e. would require copying the uncompressed segments into the heap). So I suggest to look at this issue in combination with OAK-5859, where one suggested approach is to move away from memory mapped access. > Compressed segments > --- > > Key: OAK-5860 > URL: https://issues.apache.org/jira/browse/OAK-5860 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Michael Dürig > Labels: scalability > Fix For: 1.7.2 > > > It would be interesting to see the effect of compressing the segments within > the tar files with a sufficiently effective and performant compression > algorithm: > * Can we increase overall throughput by trading CPU for IO? > * Can we scale to bigger repositories (in number of nodes) by squeezing in > more segments per MB and thus pushing out onset of thrashing? > * What would be a good compression algorithm/library? > * Can/should we make this optional? > * Migration and compatibility issues? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5042) Improve caching of segments
[
https://issues.apache.org/jira/browse/OAK-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924017#comment-15924017
]
Thomas Mueller commented on OAK-5042:
-
According to Michael Duerig, The test is SegmentCompactionIT. After that, the
log files needs to be analyzed in some way. I would need more details on this.
About "cache: Revert to LRU style cache": as far as I understand, it's not
actually LRU but goes as follows: the first level cache increments a counter
for each access (I would call that "frequency"). Then if an entry needs to be
evicted, the last entry in the FIFO queue is read, if frequency > 0 then
frequency >>= 1 and added again to the FIFO queue. If frequency is 0, that
entry is evicted. It looks a bit like the Clock algorithm to me, but using
frequency instead of just an access bit. The other caching algorithms (base,
nobin) don't read the counter.
> Improve caching of segments
> ---
>
> Key: OAK-5042
> URL: https://issues.apache.org/jira/browse/OAK-5042
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: performance, scalability
> Fix For: 1.8, 1.7.1
>
> Attachments: OAK-5042.png
>
>
> Various aspects of how Segment Tar caches segments could possibly improved.
> The current cache is a result of replacing the former ad-hoc cache with a
> proper one in OAK-3055. While the former was prone to contention under
> concurrent load the current cache is too oblivious about reads: read accesses
> are always served through {{SegmentId.segment}} and never actually hit the
> cache. This results in frequently accessed segments not to be seen as such by
> the cache and potentially being prematurely evicted.
> Possibly approaches to address this problem include:
> * Reinstantiating the cache we had pre OAK-3055 but making in fully
> concurrent.
> * Convey the information about read accesses to the current cache.
> * In either of the above cases avoid bulk segments from being placed into the
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (OAK-5926) Readability of membership code
[ https://issues.apache.org/jira/browse/OAK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angela resolved OAK-5926. - Resolution: Fixed Committed revision 1786879. > Readability of membership code > -- > > Key: OAK-5926 > URL: https://issues.apache.org/jira/browse/OAK-5926 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Reporter: angela >Assignee: angela >Priority: Minor > Fix For: 1.8 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OAK-5926) Readability of membership code
[ https://issues.apache.org/jira/browse/OAK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angela updated OAK-5926: Fix Version/s: 1.7.0 > Readability of membership code > -- > > Key: OAK-5926 > URL: https://issues.apache.org/jira/browse/OAK-5926 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Reporter: angela >Assignee: angela >Priority: Minor > Fix For: 1.7.0, 1.8 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (OAK-5928) improve benchmarks for removal of group members
angela created OAK-5928: --- Summary: improve benchmarks for removal of group members Key: OAK-5928 URL: https://issues.apache.org/jira/browse/OAK-5928 Project: Jackrabbit Oak Issue Type: Improvement Reporter: angela Priority: Minor instead of just removing existing members which renders the runTest method a no-op ones there are no members left, the tests should be adjusted to removing both existing a non-existing members -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5927) Load excerpt lazily
[
https://issues.apache.org/jira/browse/OAK-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923960#comment-15923960
]
Tommaso Teofili commented on OAK-5927:
--
+1
> Load excerpt lazily
> ---
>
> Key: OAK-5927
> URL: https://issues.apache.org/jira/browse/OAK-5927
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
> Fix For: 1.8
>
>
> Currently LucenePropertyIndex loads the excerpt eagerly in batch as part of
> loadDocs call. The load docs batch size doubles starting from 50 (max 100k)
> as more data is read.
> We should look into ways to make the excerpt loaded lazily as and when caller
> ask for excerpt.
> Note that currently the excerpt are only loaded when query request for
> excerpt i.e. there is a not null property restriction for {{rep:excerpt}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OAK-5927) Load excerpt lazily
[
https://issues.apache.org/jira/browse/OAK-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-5927:
-
Description:
Currently LucenePropertyIndex loads the excerpt eagerly in batch as part of
loadDocs call. The load docs batch size doubles starting from 50 (max 100k) as
more data is read.
We should look into ways to make the excerpt loaded lazily as and when caller
ask for excerpt.
Note that currently the excerpt are only loaded when query request for excerpt
i.e. there is a not null property restriction for {{rep:excerpt}}.
was:
Currently LucenePropertyIndex loads the excerpt eagerly in batch as part of
loadDocs call. The load docs batch size doubles starting from 50 (max 100k) as
more data is read.
We should look into ways to make the excerpt loaded lazily as and when caller
ask for excerpt.
> Load excerpt lazily
> ---
>
> Key: OAK-5927
> URL: https://issues.apache.org/jira/browse/OAK-5927
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
> Fix For: 1.8
>
>
> Currently LucenePropertyIndex loads the excerpt eagerly in batch as part of
> loadDocs call. The load docs batch size doubles starting from 50 (max 100k)
> as more data is read.
> We should look into ways to make the excerpt loaded lazily as and when caller
> ask for excerpt.
> Note that currently the excerpt are only loaded when query request for
> excerpt i.e. there is a not null property restriction for {{rep:excerpt}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OAK-5927) Load excerpt lazily
[ https://issues.apache.org/jira/browse/OAK-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923945#comment-15923945 ] Chetan Mehrotra commented on OAK-5927: -- [~teofili] [~catholicon] Thoughts? > Load excerpt lazily > --- > > Key: OAK-5927 > URL: https://issues.apache.org/jira/browse/OAK-5927 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Reporter: Chetan Mehrotra > Fix For: 1.8 > > > Currently LucenePropertyIndex loads the excerpt eagerly in batch as part of > loadDocs call. The load docs batch size doubles starting from 50 (max 100k) > as more data is read. > We should look into ways to make the excerpt loaded lazily as and when caller > ask for excerpt. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (OAK-5927) Load excerpt lazily
Chetan Mehrotra created OAK-5927: Summary: Load excerpt lazily Key: OAK-5927 URL: https://issues.apache.org/jira/browse/OAK-5927 Project: Jackrabbit Oak Issue Type: Improvement Components: lucene Reporter: Chetan Mehrotra Fix For: 1.8 Currently LucenePropertyIndex loads the excerpt eagerly in batch as part of loadDocs call. The load docs batch size doubles starting from 50 (max 100k) as more data is read. We should look into ways to make the excerpt loaded lazily as and when caller ask for excerpt. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-5920) Checkpoint migration will fail if the MissingBlobStore is used
[
https://issues.apache.org/jira/browse/OAK-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923883#comment-15923883
]
Tomek Rękawek commented on OAK-5920:
Backported to 1.6 in [r1786864|https://svn.apache.org/r1786864].
> Checkpoint migration will fail if the MissingBlobStore is used
> --
>
> Key: OAK-5920
> URL: https://issues.apache.org/jira/browse/OAK-5920
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: upgrade
>Affects Versions: 1.6.1
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
> Fix For: 1.7.0, 1.6.2
>
> Attachments: OAK-5920.patch
>
>
> It seems that the checkpoint migration doesn't work with the missing blob
> store:
> {noformat}
> 09.03.2017 07:44:22.777 INFO o.a.j.o.u.RepositorySidegrade: Copying node
> #2638:
> /libs/fd/af/components/panel/cq:styleConfig/items/wizardPanel/items/wizardPanelScrollers
> 09.03.2017 07:44:22.813 INFO o.a.j.o.u.RepositorySidegrade: Copying node
> #2639:
> /apps/acs-commons/components/content/sharethis-buttons/dialog/items/items
> 09.03.2017 07:44:31.805 INFO o.a.j.o.s.f.FileStore: TarMK closed:
> /mnt/crx/publish/crx-quickstart/repository-segment-tar-20170309-072655/segmentstore
> 09.03.2017 07:44:32.051 INFO o.a.j.o.p.s.f.FileStore: TarMK closed:
> /mnt/crx/publish/crx-quickstart/repository/segmentstore
> Caused by: java.lang.RuntimeException: javax.jcr.RepositoryException: Failed
> to copy content
> at com.google.common.io.Closer.rethrow(Closer.java:149)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:81)
> at
> com.adobe.granite.crx2oak.engine.MigrationEngine$2.doMigration(MigrationEngine.java:66)
> at
> com.adobe.granite.crx2oak.engine.MigrationEngine.process(MigrationEngine.java:91)
> at com.adobe.granite.crx2oak.pipeline.Pipeline.run(Pipeline.java:103)
> at com.adobe.granite.crx2oak.CRX2Oak.run(CRX2Oak.java:66)
> at com.adobe.granite.crx2oak.CRX2Oak.main(CRX2Oak.java:51)
> ... 6 more
> Caused by: javax.jcr.RepositoryException: Failed to copy content
> at
> org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:286)
> at
> org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:242)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.sidegrade(OakUpgrade.java:92)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:78)
> ... 11 more
> Caused by: java.lang.UnsupportedOperationException
> at
> org.apache.jackrabbit.oak.upgrade.cli.blob.MissingBlobStore.getInputStream(MissingBlobStore.java:62)
> at
> org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:276)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:86)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.openStream(AbstractBlob.java:44)
> at com.google.common.io.ByteSource.contentEquals(ByteSource.java:344)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:227)
> at com.google.common.base.Objects.equal(Objects.java:60)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.equals(SegmentPropertyState.java:242)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:617)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:511)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:551)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:551)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:456)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:604)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
>
[jira] [Updated] (OAK-5920) Checkpoint migration will fail if the MissingBlobStore is used
[
https://issues.apache.org/jira/browse/OAK-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-5920:
---
Fix Version/s: 1.6.2
> Checkpoint migration will fail if the MissingBlobStore is used
> --
>
> Key: OAK-5920
> URL: https://issues.apache.org/jira/browse/OAK-5920
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: upgrade
>Affects Versions: 1.6.1
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
> Fix For: 1.7.0, 1.6.2
>
> Attachments: OAK-5920.patch
>
>
> It seems that the checkpoint migration doesn't work with the missing blob
> store:
> {noformat}
> 09.03.2017 07:44:22.777 INFO o.a.j.o.u.RepositorySidegrade: Copying node
> #2638:
> /libs/fd/af/components/panel/cq:styleConfig/items/wizardPanel/items/wizardPanelScrollers
> 09.03.2017 07:44:22.813 INFO o.a.j.o.u.RepositorySidegrade: Copying node
> #2639:
> /apps/acs-commons/components/content/sharethis-buttons/dialog/items/items
> 09.03.2017 07:44:31.805 INFO o.a.j.o.s.f.FileStore: TarMK closed:
> /mnt/crx/publish/crx-quickstart/repository-segment-tar-20170309-072655/segmentstore
> 09.03.2017 07:44:32.051 INFO o.a.j.o.p.s.f.FileStore: TarMK closed:
> /mnt/crx/publish/crx-quickstart/repository/segmentstore
> Caused by: java.lang.RuntimeException: javax.jcr.RepositoryException: Failed
> to copy content
> at com.google.common.io.Closer.rethrow(Closer.java:149)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:81)
> at
> com.adobe.granite.crx2oak.engine.MigrationEngine$2.doMigration(MigrationEngine.java:66)
> at
> com.adobe.granite.crx2oak.engine.MigrationEngine.process(MigrationEngine.java:91)
> at com.adobe.granite.crx2oak.pipeline.Pipeline.run(Pipeline.java:103)
> at com.adobe.granite.crx2oak.CRX2Oak.run(CRX2Oak.java:66)
> at com.adobe.granite.crx2oak.CRX2Oak.main(CRX2Oak.java:51)
> ... 6 more
> Caused by: javax.jcr.RepositoryException: Failed to copy content
> at
> org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:286)
> at
> org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:242)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.sidegrade(OakUpgrade.java:92)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:78)
> ... 11 more
> Caused by: java.lang.UnsupportedOperationException
> at
> org.apache.jackrabbit.oak.upgrade.cli.blob.MissingBlobStore.getInputStream(MissingBlobStore.java:62)
> at
> org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:276)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:86)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.openStream(AbstractBlob.java:44)
> at com.google.common.io.ByteSource.contentEquals(ByteSource.java:344)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:227)
> at com.google.common.base.Objects.equal(Objects.java:60)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.equals(SegmentPropertyState.java:242)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:617)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:511)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:551)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:551)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:456)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:604)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:551)
> at
>
[jira] [Comment Edited] (OAK-5920) Checkpoint migration will fail if the MissingBlobStore is used
[
https://issues.apache.org/jira/browse/OAK-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15907650#comment-15907650
]
Tomek Rękawek edited comment on OAK-5920 at 3/14/17 9:01 AM:
-
[r1786851|https://svn.apache.org/r1786851] &
[r1786859|https://svn.apache.org/r1786859] breaks the migration if the
checkpoints can't be copied. This behaviour can be overridden with the new
{{\-\-skip\-checkpoints}} option.
was (Author: tomek.rekawek):
[r1786851|https://svn.apache.org/r1786851] breaks the migration if the
checkpoints can't be copied. This behaviour can be overridden with the new
{{\-\-skip\-checkpoints}} option.
> Checkpoint migration will fail if the MissingBlobStore is used
> --
>
> Key: OAK-5920
> URL: https://issues.apache.org/jira/browse/OAK-5920
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: upgrade
>Affects Versions: 1.6.1
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
> Fix For: 1.7.0
>
> Attachments: OAK-5920.patch
>
>
> It seems that the checkpoint migration doesn't work with the missing blob
> store:
> {noformat}
> 09.03.2017 07:44:22.777 INFO o.a.j.o.u.RepositorySidegrade: Copying node
> #2638:
> /libs/fd/af/components/panel/cq:styleConfig/items/wizardPanel/items/wizardPanelScrollers
> 09.03.2017 07:44:22.813 INFO o.a.j.o.u.RepositorySidegrade: Copying node
> #2639:
> /apps/acs-commons/components/content/sharethis-buttons/dialog/items/items
> 09.03.2017 07:44:31.805 INFO o.a.j.o.s.f.FileStore: TarMK closed:
> /mnt/crx/publish/crx-quickstart/repository-segment-tar-20170309-072655/segmentstore
> 09.03.2017 07:44:32.051 INFO o.a.j.o.p.s.f.FileStore: TarMK closed:
> /mnt/crx/publish/crx-quickstart/repository/segmentstore
> Caused by: java.lang.RuntimeException: javax.jcr.RepositoryException: Failed
> to copy content
> at com.google.common.io.Closer.rethrow(Closer.java:149)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:81)
> at
> com.adobe.granite.crx2oak.engine.MigrationEngine$2.doMigration(MigrationEngine.java:66)
> at
> com.adobe.granite.crx2oak.engine.MigrationEngine.process(MigrationEngine.java:91)
> at com.adobe.granite.crx2oak.pipeline.Pipeline.run(Pipeline.java:103)
> at com.adobe.granite.crx2oak.CRX2Oak.run(CRX2Oak.java:66)
> at com.adobe.granite.crx2oak.CRX2Oak.main(CRX2Oak.java:51)
> ... 6 more
> Caused by: javax.jcr.RepositoryException: Failed to copy content
> at
> org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:286)
> at
> org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:242)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.sidegrade(OakUpgrade.java:92)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:78)
> ... 11 more
> Caused by: java.lang.UnsupportedOperationException
> at
> org.apache.jackrabbit.oak.upgrade.cli.blob.MissingBlobStore.getInputStream(MissingBlobStore.java:62)
> at
> org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:276)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:86)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.openStream(AbstractBlob.java:44)
> at com.google.common.io.ByteSource.contentEquals(ByteSource.java:344)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:227)
> at com.google.common.base.Objects.equal(Objects.java:60)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.equals(SegmentPropertyState.java:242)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:617)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:511)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:551)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:551)
> at
>
[jira] [Comment Edited] (OAK-4802) Basic cache consistency test on exception
[ https://issues.apache.org/jira/browse/OAK-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15907087#comment-15907087 ] Julian Reschke edited comment on OAK-4802 at 3/14/17 8:53 AM: -- trunk: [r1760701|http://svn.apache.org/r1760701] [r1760677|http://svn.apache.org/r1760677] 1.4: [r1786646|http://svn.apache.org/r1786646] [r1786640|http://svn.apache.org/r1786640] 1.2: [r1786854|http://svn.apache.org/r1786854] was (Author: reschke): trunk: [r1760701|http://svn.apache.org/r1760701] [r1760677|http://svn.apache.org/r1760677] 1.4: [r1786646|http://svn.apache.org/r1786646] [r1786640|http://svn.apache.org/r1786640] > Basic cache consistency test on exception > - > > Key: OAK-4802 > URL: https://issues.apache.org/jira/browse/OAK-4802 > Project: Jackrabbit Oak > Issue Type: Test > Components: core, documentmk >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Labels: candidate_oak_1_0, resilience > Fix For: 1.4.15, 1.5.11, 1.6.0, 1.2.25 > > > OAK-4774 and OAK-4793 aim to check if the cache behaviour of a DocumentStore > implementation when the underlying backend throws an exception even though > the operation succeeded. E.g. the response cannot be sent back because of a > network issue. > This issue will provide the DocumentStore independent part of those tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OAK-4802) Basic cache consistency test on exception
[ https://issues.apache.org/jira/browse/OAK-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-4802: Fix Version/s: 1.2.25 > Basic cache consistency test on exception > - > > Key: OAK-4802 > URL: https://issues.apache.org/jira/browse/OAK-4802 > Project: Jackrabbit Oak > Issue Type: Test > Components: core, documentmk >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Labels: candidate_oak_1_0, resilience > Fix For: 1.4.15, 1.5.11, 1.6.0, 1.2.25 > > > OAK-4774 and OAK-4793 aim to check if the cache behaviour of a DocumentStore > implementation when the underlying backend throws an exception even though > the operation succeeded. E.g. the response cannot be sent back because of a > network issue. > This issue will provide the DocumentStore independent part of those tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OAK-4802) Basic cache consistency test on exception
[ https://issues.apache.org/jira/browse/OAK-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-4802: Labels: candidate_oak_1_0 resilience (was: candidate_oak_1_0 candidate_oak_1_2 resilience) > Basic cache consistency test on exception > - > > Key: OAK-4802 > URL: https://issues.apache.org/jira/browse/OAK-4802 > Project: Jackrabbit Oak > Issue Type: Test > Components: core, documentmk >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Labels: candidate_oak_1_0, resilience > Fix For: 1.4.15, 1.5.11, 1.6.0, 1.2.25 > > > OAK-4774 and OAK-4793 aim to check if the cache behaviour of a DocumentStore > implementation when the underlying backend throws an exception even though > the operation succeeded. E.g. the response cannot be sent back because of a > network issue. > This issue will provide the DocumentStore independent part of those tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (OAK-5926) Readability of membership code
angela created OAK-5926: --- Summary: Readability of membership code Key: OAK-5926 URL: https://issues.apache.org/jira/browse/OAK-5926 Project: Jackrabbit Oak Issue Type: Improvement Components: core Reporter: angela Assignee: angela Priority: Minor Fix For: 1.8 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (OAK-5920) Checkpoint migration will fail if the MissingBlobStore is used
[
https://issues.apache.org/jira/browse/OAK-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15907650#comment-15907650
]
Tomek Rękawek edited comment on OAK-5920 at 3/14/17 8:14 AM:
-
[r1786851|https://svn.apache.org/r1786851] breaks the migration if the
checkpoints can't be copied. This behaviour can be overridden with the new
{{\-\-skip\-checkpoints}} option.
was (Author: tomek.rekawek):
Attached patch breaks the migration if the checkpoints can't be copied. This
behaviour can be overridden with the new {{\-\-skip\-checkpoints}} option.
> Checkpoint migration will fail if the MissingBlobStore is used
> --
>
> Key: OAK-5920
> URL: https://issues.apache.org/jira/browse/OAK-5920
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: upgrade
>Affects Versions: 1.6.1
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
> Fix For: 1.7.0
>
> Attachments: OAK-5920.patch
>
>
> It seems that the checkpoint migration doesn't work with the missing blob
> store:
> {noformat}
> 09.03.2017 07:44:22.777 INFO o.a.j.o.u.RepositorySidegrade: Copying node
> #2638:
> /libs/fd/af/components/panel/cq:styleConfig/items/wizardPanel/items/wizardPanelScrollers
> 09.03.2017 07:44:22.813 INFO o.a.j.o.u.RepositorySidegrade: Copying node
> #2639:
> /apps/acs-commons/components/content/sharethis-buttons/dialog/items/items
> 09.03.2017 07:44:31.805 INFO o.a.j.o.s.f.FileStore: TarMK closed:
> /mnt/crx/publish/crx-quickstart/repository-segment-tar-20170309-072655/segmentstore
> 09.03.2017 07:44:32.051 INFO o.a.j.o.p.s.f.FileStore: TarMK closed:
> /mnt/crx/publish/crx-quickstart/repository/segmentstore
> Caused by: java.lang.RuntimeException: javax.jcr.RepositoryException: Failed
> to copy content
> at com.google.common.io.Closer.rethrow(Closer.java:149)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:81)
> at
> com.adobe.granite.crx2oak.engine.MigrationEngine$2.doMigration(MigrationEngine.java:66)
> at
> com.adobe.granite.crx2oak.engine.MigrationEngine.process(MigrationEngine.java:91)
> at com.adobe.granite.crx2oak.pipeline.Pipeline.run(Pipeline.java:103)
> at com.adobe.granite.crx2oak.CRX2Oak.run(CRX2Oak.java:66)
> at com.adobe.granite.crx2oak.CRX2Oak.main(CRX2Oak.java:51)
> ... 6 more
> Caused by: javax.jcr.RepositoryException: Failed to copy content
> at
> org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:286)
> at
> org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:242)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.sidegrade(OakUpgrade.java:92)
> at
> org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:78)
> ... 11 more
> Caused by: java.lang.UnsupportedOperationException
> at
> org.apache.jackrabbit.oak.upgrade.cli.blob.MissingBlobStore.getInputStream(MissingBlobStore.java:62)
> at
> org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:276)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:86)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.openStream(AbstractBlob.java:44)
> at com.google.common.io.ByteSource.contentEquals(ByteSource.java:344)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:227)
> at com.google.common.base.Objects.equal(Objects.java:60)
> at
> org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.equals(SegmentPropertyState.java:242)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:617)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:511)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:551)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:551)
> at
> org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeChanged(ApplyDiff.java:87)
> at
>
[jira] [Assigned] (OAK-5923) Document S3 datastore
[ https://issues.apache.org/jira/browse/OAK-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Saar reassigned OAK-5923: --- Assignee: Amit Jain > Document S3 datastore > - > > Key: OAK-5923 > URL: https://issues.apache.org/jira/browse/OAK-5923 > Project: Jackrabbit Oak > Issue Type: Documentation > Components: blob >Reporter: Alexander Klimetschek >Assignee: Amit Jain > Fix For: 1.6.2 > > > The S3 datastore is currently hardly documented. > The [generic blobstore > documentation|http://jackrabbit.apache.org/oak/docs/plugins/blobstore.html] > is very much focused about the internal class structures, but quite confusing > for someone who wants to configure a specific datastore such as file and s3 > (the only ones right now). S3 settings are not documented at all, the [config > page|http://jackrabbit.apache.org/oak/docs/osgi_config.html#config-blobstore] > only mentions the generic maxCachedBinarySize and cacheSizeInMB. > The best bet is the [Adobe AEM product > documentation|https://docs.adobe.com/docs/en/aem/6-2/deploy/platform/data-store-config.html], > but that is for an older version and a few things changed since then. > Specific items below. Some have been confusing people using oak-blob-cloud > 1.5.15: > - "secret" property unclear (new) > - secretKey & accessKey can be omitted to leverage IAM roles (new) > - drop of proactiveCaching property (new) > - aws bucket/region/etc. settings > - config options (timeout, retries, threads) > - understanding caching behavior and performance optimization > - shared vs. non-shared options > - migrating from a previous version, how to update the config > - requirements on the AWS (account) side -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (OAK-5923) Document S3 datastore
[ https://issues.apache.org/jira/browse/OAK-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Saar updated OAK-5923: Fix Version/s: 1.6.2 > Document S3 datastore > - > > Key: OAK-5923 > URL: https://issues.apache.org/jira/browse/OAK-5923 > Project: Jackrabbit Oak > Issue Type: Documentation > Components: blob >Reporter: Alexander Klimetschek >Assignee: Amit Jain > Fix For: 1.6.2 > > > The S3 datastore is currently hardly documented. > The [generic blobstore > documentation|http://jackrabbit.apache.org/oak/docs/plugins/blobstore.html] > is very much focused about the internal class structures, but quite confusing > for someone who wants to configure a specific datastore such as file and s3 > (the only ones right now). S3 settings are not documented at all, the [config > page|http://jackrabbit.apache.org/oak/docs/osgi_config.html#config-blobstore] > only mentions the generic maxCachedBinarySize and cacheSizeInMB. > The best bet is the [Adobe AEM product > documentation|https://docs.adobe.com/docs/en/aem/6-2/deploy/platform/data-store-config.html], > but that is for an older version and a few things changed since then. > Specific items below. Some have been confusing people using oak-blob-cloud > 1.5.15: > - "secret" property unclear (new) > - secretKey & accessKey can be omitted to leverage IAM roles (new) > - drop of proactiveCaching property (new) > - aws bucket/region/etc. settings > - config options (timeout, retries, threads) > - understanding caching behavior and performance optimization > - shared vs. non-shared options > - migrating from a previous version, how to update the config > - requirements on the AWS (account) side -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (OAK-5925) CLONE - Limit the number of times a LuceneResultRow based iterator get reset
[
https://issues.apache.org/jira/browse/OAK-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra reassigned OAK-5925:
Assignee: Chetan Mehrotra (was: Thomas Mueller)
> CLONE - Limit the number of times a LuceneResultRow based iterator get reset
>
>
> Key: OAK-5925
> URL: https://issues.apache.org/jira/browse/OAK-5925
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.5.15, 1.6.0
>
>
> With changes done in OAK-2569 it can happen that the cursor returned by
> {{LucenePropertyIndex}} gets reset multiple times as the index gets updated
> for a long running query
> *Cloned issue to ensure that fix is done for LucenePropertyIndex also*
> {noformat}
> 11697: 26.02.2016 16:12:16.296 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3690 ms
> 47507: 26.02.2016 16:13:26.744 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
> 12800 entries for query :fulltext:foo
> 47513: 26.02.2016 16:13:30.486 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3742 ms
> 55832: 26.02.2016 16:15:43.693 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
> 25600 entries for query :fulltext:foo
> 55835: 26.02.2016 16:15:51.261 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 7568 ms
> 59383: 26.02.2016 16:21:12.355 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex Change in index
> version detected 4186634 => 4186631. Query would be performed without offset
> 59384: 26.02.2016 16:21:12.355 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the first
> 51200 entries for query :fulltext:foo
> {noformat}
> If the index continuously gets updated then such a query would never finish
> and would consume system resources. For such case we should enforce some
> limit
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OAK-5925) CLONE - Limit the number of times a LuceneResultRow based iterator get reset
[
https://issues.apache.org/jira/browse/OAK-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-5925:
-
Description:
With changes done in OAK-2569 it can happen that the cursor returned by
{{LucenePropertyIndex}} gets reset multiple times as the index gets updated for
a long running query
*Cloned issue to ensure that fix is done for LucenePropertyIndex also*
{noformat}
11697: 26.02.2016 16:12:16.296 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3690 ms
47507: 26.02.2016 16:13:26.744 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
12800 entries for query :fulltext:foo
47513: 26.02.2016 16:13:30.486 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3742 ms
55832: 26.02.2016 16:15:43.693 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
25600 entries for query :fulltext:foo
55835: 26.02.2016 16:15:51.261 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 7568 ms
59383: 26.02.2016 16:21:12.355 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex Change in index
version detected 4186634 => 4186631. Query would be performed without offset
59384: 26.02.2016 16:21:12.355 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the first
51200 entries for query :fulltext:foo
{noformat}
If the index continuously gets updated then such a query would never finish and
would consume system resources. For such case we should enforce some limit
was:
With changes done in OAK-2569 it can happen that the cursor returned by
{{LucenePropertyIndex}} gets reset multiple times as the index gets updated for
a long running query
{noformat}
11697: 26.02.2016 16:12:16.296 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3690 ms
47507: 26.02.2016 16:13:26.744 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
12800 entries for query :fulltext:foo
47513: 26.02.2016 16:13:30.486 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3742 ms
55832: 26.02.2016 16:15:43.693 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
25600 entries for query :fulltext:foo
55835: 26.02.2016 16:15:51.261 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 7568 ms
59383: 26.02.2016 16:21:12.355 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex Change in index
version detected 4186634 => 4186631. Query would be performed without offset
59384: 26.02.2016 16:21:12.355 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the first
51200 entries for query :fulltext:foo
{noformat}
If the index continuously gets updated then such a query would never finish and
would consume system resources. For such case we should enforce some limit
> CLONE - Limit the number of times a LuceneResultRow based iterator get reset
>
>
> Key: OAK-5925
> URL: https://issues.apache.org/jira/browse/OAK-5925
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Thomas Mueller
>Priority: Minor
> Fix For: 1.5.15, 1.6.0
>
>
> With changes done in OAK-2569 it can happen that the cursor returned by
> {{LucenePropertyIndex}} gets reset multiple times as the index gets updated
> for a long running query
> *Cloned issue to ensure that fix is done for LucenePropertyIndex also*
> {noformat}
> 11697: 26.02.2016 16:12:16.296 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3690 ms
> 47507: 26.02.2016 16:13:26.744 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
> 12800 entries for query :fulltext:foo
> 47513: 26.02.2016 16:13:30.486 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3742 ms
> 55832: 26.02.2016 16:15:43.693 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
> 25600 entries for query :fulltext:foo
> 55835: 26.02.2016 16:15:51.261 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 7568 ms
> 59383: 26.02.2016 16:21:12.355 *DEBUG* [qtp567069427-14229]
>
[jira] [Created] (OAK-5925) CLONE - Limit the number of times a LuceneResultRow based iterator get reset
Chetan Mehrotra created OAK-5925:
Summary: CLONE - Limit the number of times a LuceneResultRow based
iterator get reset
Key: OAK-5925
URL: https://issues.apache.org/jira/browse/OAK-5925
Project: Jackrabbit Oak
Issue Type: Improvement
Components: lucene
Reporter: Chetan Mehrotra
Assignee: Thomas Mueller
Priority: Minor
Fix For: 1.5.15, 1.6.0
With changes done in OAK-2569 it can happen that the cursor returned by
{{LucenePropertyIndex}} gets reset multiple times as the index gets updated for
a long running query
{noformat}
11697: 26.02.2016 16:12:16.296 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3690 ms
47507: 26.02.2016 16:13:26.744 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
12800 entries for query :fulltext:foo
47513: 26.02.2016 16:13:30.486 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3742 ms
55832: 26.02.2016 16:15:43.693 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
25600 entries for query :fulltext:foo
55835: 26.02.2016 16:15:51.261 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 7568 ms
59383: 26.02.2016 16:21:12.355 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex Change in index
version detected 4186634 => 4186631. Query would be performed without offset
59384: 26.02.2016 16:21:12.355 *DEBUG* [qtp567069427-14229]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the first
51200 entries for query :fulltext:foo
{noformat}
If the index continuously gets updated then such a query would never finish and
would consume system resources. For such case we should enforce some limit
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OAK-4096) Limit the number of times a LuceneResultRow based iterator get reset
[
https://issues.apache.org/jira/browse/OAK-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923646#comment-15923646
]
Chetan Mehrotra commented on OAK-4096:
--
[~tmueller] Looks like this change was not done for LucenePropertyIndex?
> Limit the number of times a LuceneResultRow based iterator get reset
>
>
> Key: OAK-4096
> URL: https://issues.apache.org/jira/browse/OAK-4096
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Thomas Mueller
>Priority: Minor
> Fix For: 1.5.15, 1.6.0
>
>
> With changes done in OAK-2569 it can happen that the cursor returned by
> {{LucenePropertyIndex}} gets reset multiple times as the index gets updated
> for a long running query
> {noformat}
> 11697: 26.02.2016 16:12:16.296 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3690 ms
> 47507: 26.02.2016 16:13:26.744 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
> 12800 entries for query :fulltext:foo
> 47513: 26.02.2016 16:13:30.486 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 3742 ms
> 55832: 26.02.2016 16:15:43.693 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the next
> 25600 entries for query :fulltext:foo
> 55835: 26.02.2016 16:15:51.261 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex ... took 7568 ms
> 59383: 26.02.2016 16:21:12.355 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex Change in index
> version detected 4186634 => 4186631. Query would be performed without offset
> 59384: 26.02.2016 16:21:12.355 *DEBUG* [qtp567069427-14229]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex loading the first
> 51200 entries for query :fulltext:foo
> {noformat}
> If the index continuously gets updated then such a query would never finish
> and would consume system resources. For such case we should enforce some
> limit
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OAK-5924) Prevent long running query from delaying refresh of index
[
https://issues.apache.org/jira/browse/OAK-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-5924:
-
Summary: Prevent long running query from delaying refresh of index (was:
Prevent long running query from delaying close of index)
> Prevent long running query from delaying refresh of index
> -
>
> Key: OAK-5924
> URL: https://issues.apache.org/jira/browse/OAK-5924
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.8
>
>
> Whenever the index gets updated {{IndexTracker}} detects the changes and open
> new {{IndexNode}} and closes old index nodes. This flow would block untill
> all old IndexNode are closed.
> IndexNode close itself relies on a writer lock. It can happen that a long
> running query i.e. a query which is about to read a page of large is
> currently executing on the old IndexNode instance. This query is trying load
> 100k docs and is very slow (due to loading of excerpt) then such a query
> would prevent the IndexNode from getting closed. This in turn would prevent
> the index from seeing latest data and become stale.
> To make query and indexing more resilient we should look if current IndexNode
> being used for query is closing or not. If closing then query should open a
> fresh searcher
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
