[jira] [Created] (OAK-4937) JournalGC failing with RDB DocumentStore
Chetan Mehrotra created OAK-4937:
Summary: JournalGC failing with RDB DocumentStore
Key: OAK-4937
URL: https://issues.apache.org/jira/browse/OAK-4937
Project: Jackrabbit Oak
Issue Type: Bug
Components: rdbmk
Reporter: Chetan Mehrotra
Fix For: 1.6
Journal GC on RDB DocumentStore fails with following exception
{noformat}
org.apache.jackrabbit.oak.plugins.document.DocumentStoreException: Can't insert
the document: null
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore.internalCreateOrUpdate(RDBDocumentStore.java:1216)
at
org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore.createOrUpdate(RDBDocumentStore.java:295)
at
org.apache.jackrabbit.oak.plugins.document.util.LeaseCheckDocumentStoreWrapper.createOrUpdate(LeaseCheckDocumentStoreWrapper.java:128)
at
org.apache.jackrabbit.oak.plugins.document.JournalGarbageCollector.updateTailTimestamp(JournalGarbageCollector.java:167)
at
org.apache.jackrabbit.oak.plugins.document.JournalGarbageCollector.gc(JournalGarbageCollector.java:112)
at
org.apache.jackrabbit.oak.plugins.document.JournalGCIT.journalGCSimpleInvocation(JournalGCIT.java:42)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4403) Diff traversal in persisted branch commit traversing to unrelated paths
[ https://issues.apache.org/jira/browse/OAK-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574250#comment-15574250 ] Chetan Mehrotra commented on OAK-4403: -- Setting a fix version to ensure that we look into this issue sooner for 1.6 > Diff traversal in persisted branch commit traversing to unrelated paths > --- > > Key: OAK-4403 > URL: https://issues.apache.org/jira/browse/OAK-4403 > Project: Jackrabbit Oak > Issue Type: Bug > Components: documentmk >Reporter: Chetan Mehrotra > Fix For: 1.6, 1.5.13 > > > With DocumentNodeStore when a commit involves a persisted branch the diff > logic traverses to those paths which are not affected by the current commit. > For e.g. in following flow > # Commit C1 (base rev R1) - Starts and modifies paths under /content. The > number of changes done exceed the update.limit and hence the in memory branch > transitions to Persisted state > # Before C1 commits another commit C2 starts and modified paths under /etc > and /var > # C2 commits (before C1). Head revision R2 > # C1 commits and perform the merge. Revision moves to R3 > Now at #4 the merge would trigger a diff which should normally traverse and > perform diff for those paths which are modified by C1. However currently it > seems to also perform diff on all those paths which have been modified after > R1 and R3. In doing so it would traverse the complete sub trees of such > paths. Note that functionally the end result is same just that too many extra > nodes are read which would put load on system > If the number of changes are reduced such that persisted branch is not > created then diff only reads those parts which are modified by the current > commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4403) Diff traversal in persisted branch commit traversing to unrelated paths
[ https://issues.apache.org/jira/browse/OAK-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-4403: - Fix Version/s: 1.5.13 > Diff traversal in persisted branch commit traversing to unrelated paths > --- > > Key: OAK-4403 > URL: https://issues.apache.org/jira/browse/OAK-4403 > Project: Jackrabbit Oak > Issue Type: Bug > Components: documentmk >Reporter: Chetan Mehrotra > Fix For: 1.6, 1.5.13 > > > With DocumentNodeStore when a commit involves a persisted branch the diff > logic traverses to those paths which are not affected by the current commit. > For e.g. in following flow > # Commit C1 (base rev R1) - Starts and modifies paths under /content. The > number of changes done exceed the update.limit and hence the in memory branch > transitions to Persisted state > # Before C1 commits another commit C2 starts and modified paths under /etc > and /var > # C2 commits (before C1). Head revision R2 > # C1 commits and perform the merge. Revision moves to R3 > Now at #4 the merge would trigger a diff which should normally traverse and > perform diff for those paths which are modified by C1. However currently it > seems to also perform diff on all those paths which have been modified after > R1 and R3. In doing so it would traverse the complete sub trees of such > paths. Note that functionally the end result is same just that too many extra > nodes are read which would put load on system > If the number of changes are reduced such that persisted branch is not > created then diff only reads those parts which are modified by the current > commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3036) DocumentRootBuilder: revisit update.limit default
[ https://issues.apache.org/jira/browse/OAK-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-3036: - Labels: candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4 performance (was: performance) > DocumentRootBuilder: revisit update.limit default > - > > Key: OAK-3036 > URL: https://issues.apache.org/jira/browse/OAK-3036 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk, rdbmk >Reporter: Julian Reschke >Priority: Blocker > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4, > performance > Fix For: 1.6, 1.5.13 > > > update.limit decides whether a commit is persisted using a branch or not. The > default is 1 (and can be overridden using the system property). > A typical call pattern in JCR is to persist batches of ~1024 nodes. These > translate to more than 1 changes (see PackageImportIT), due to JCR > properties, and also indexing commit hooks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3036) DocumentRootBuilder: revisit update.limit default
[ https://issues.apache.org/jira/browse/OAK-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574239#comment-15574239 ] Chetan Mehrotra commented on OAK-3036: -- Marking this as blocker for next release to ensure that it gets resolved or we come to some conclusion on what needs to be done here > DocumentRootBuilder: revisit update.limit default > - > > Key: OAK-3036 > URL: https://issues.apache.org/jira/browse/OAK-3036 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk, rdbmk >Reporter: Julian Reschke >Priority: Blocker > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4, > performance > Fix For: 1.6, 1.5.13 > > > update.limit decides whether a commit is persisted using a branch or not. The > default is 1 (and can be overridden using the system property). > A typical call pattern in JCR is to persist batches of ~1024 nodes. These > translate to more than 1 changes (see PackageImportIT), due to JCR > properties, and also indexing commit hooks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3036) DocumentRootBuilder: revisit update.limit default
[ https://issues.apache.org/jira/browse/OAK-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-3036: - Fix Version/s: 1.5.13 1.6 > DocumentRootBuilder: revisit update.limit default > - > > Key: OAK-3036 > URL: https://issues.apache.org/jira/browse/OAK-3036 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk, rdbmk >Reporter: Julian Reschke >Priority: Minor > Labels: performance > Fix For: 1.6, 1.5.13 > > > update.limit decides whether a commit is persisted using a branch or not. The > default is 1 (and can be overridden using the system property). > A typical call pattern in JCR is to persist batches of ~1024 nodes. These > translate to more than 1 changes (see PackageImportIT), due to JCR > properties, and also indexing commit hooks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3036) DocumentRootBuilder: revisit update.limit default
[ https://issues.apache.org/jira/browse/OAK-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-3036: - Priority: Blocker (was: Minor) > DocumentRootBuilder: revisit update.limit default > - > > Key: OAK-3036 > URL: https://issues.apache.org/jira/browse/OAK-3036 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk, rdbmk >Reporter: Julian Reschke >Priority: Blocker > Labels: performance > Fix For: 1.6, 1.5.13 > > > update.limit decides whether a commit is persisted using a branch or not. The > default is 1 (and can be overridden using the system property). > A typical call pattern in JCR is to persist batches of ~1024 nodes. These > translate to more than 1 changes (see PackageImportIT), due to JCR > properties, and also indexing commit hooks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4936) Too many segment cache misses
[
https://issues.apache.org/jira/browse/OAK-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572413#comment-15572413
]
Michael Dürig commented on OAK-4936:
Switching from LIRS to a Guava caches greatly improves the cache's hit rate:
https://github.com/mduerig/jackrabbit-oak/commit/cf24625c05ebd20acd3b34e702f9ef5b9af7ea5a
Will run a couple of benchmarks with and without the cache and share the
numbers once I have them.
> Too many segment cache misses
> -
>
> Key: OAK-4936
> URL: https://issues.apache.org/jira/browse/OAK-4936
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: performance
> Fix For: Segment Tar 0.0.16
>
> Attachments: OAK-4936-monitoring.patch
>
>
> Running the {{ConcurrentWriteTest}} benchmark and monitoring the hits and
> misses of the segment cache (LIRS), I noticed that some segments are loaded
> over and over again (up to 3000 times).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4936) Too many segment cache misses
[
https://issues.apache.org/jira/browse/OAK-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572393#comment-15572393
]
Michael Dürig commented on OAK-4936:
[~tmueller], could you have a look into this from the LIRS cache POV? Please
open an dedicated issue if required linking this one.
> Too many segment cache misses
> -
>
> Key: OAK-4936
> URL: https://issues.apache.org/jira/browse/OAK-4936
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: performance
> Fix For: Segment Tar 0.0.16
>
> Attachments: OAK-4936-monitoring.patch
>
>
> Running the {{ConcurrentWriteTest}} benchmark and monitoring the hits and
> misses of the segment cache (LIRS), I noticed that some segments are loaded
> over and over again (up to 3000 times).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4936) Too many segment cache misses
[
https://issues.apache.org/jira/browse/OAK-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4936:
---
Attachment: OAK-4936-monitoring.patch
[^OAK-4936-monitoring.patch] adds monitoring for the cache misses and hits to
reproduce described behaviour.
> Too many segment cache misses
> -
>
> Key: OAK-4936
> URL: https://issues.apache.org/jira/browse/OAK-4936
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: performance
> Fix For: Segment Tar 0.0.16
>
> Attachments: OAK-4936-monitoring.patch
>
>
> Running the {{ConcurrentWriteTest}} benchmark and monitoring the hits and
> misses of the segment cache (LIRS), I noticed that some segments are loaded
> over and over again (up to 3000 times).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-4936) Too many segment cache misses
Michael Dürig created OAK-4936:
--
Summary: Too many segment cache misses
Key: OAK-4936
URL: https://issues.apache.org/jira/browse/OAK-4936
Project: Jackrabbit Oak
Issue Type: Bug
Components: segment-tar
Reporter: Michael Dürig
Assignee: Michael Dürig
Fix For: Segment Tar 0.0.16
Running the {{ConcurrentWriteTest}} benchmark and monitoring the hits and
misses of the segment cache (LIRS), I noticed that some segments are loaded
over and over again (up to 3000 times).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4923) Improve segment deserialization performance
[
https://issues.apache.org/jira/browse/OAK-4923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572353#comment-15572353
]
Michael Dürig commented on OAK-4923:
A couple of ideas for improvements:
https://github.com/mduerig/jackrabbit-oak/commits/OAK-4923
So far this is just a POC to evaluate the performance impacts. The various maps
in the current implementation quite often appear in stack traces, so I would
assume there is room for optimisation here. However, we need to carefully
evaluate this first before we jump to conclusion. Specifically I have
discovered an issue with the segment (LIRS) cache I'm researching (no issue
yet). Once that one is sorted out we need to see where we stand and whether we
need to further improve things here.
> Improve segment deserialization performance
> ---
>
> Key: OAK-4923
> URL: https://issues.apache.org/jira/browse/OAK-4923
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Francesco Mari
>Assignee: Francesco Mari
> Fix For: Segment Tar 0.0.16
>
> Attachments: OAK-4923-01.patch, OAK-4923-02.patch
>
>
> The methods {{readReferencedSegments}} and {{readRecordNumberOffsets}} in
> {{Segment}} compute the returned data every time they are called. While this
> is a very clean implementation, this might force unexpected I/O operations,
> since the "buffer" the data is read from usually is a memory-mapped file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4923) Improve segment deserialization performance
[
https://issues.apache.org/jira/browse/OAK-4923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4923:
---
Fix Version/s: Segment Tar 0.0.16
> Improve segment deserialization performance
> ---
>
> Key: OAK-4923
> URL: https://issues.apache.org/jira/browse/OAK-4923
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Francesco Mari
>Assignee: Francesco Mari
> Fix For: Segment Tar 0.0.16
>
> Attachments: OAK-4923-01.patch, OAK-4923-02.patch
>
>
> The methods {{readReferencedSegments}} and {{readRecordNumberOffsets}} in
> {{Segment}} compute the returned data every time they are called. While this
> is a very clean implementation, this might force unexpected I/O operations,
> since the "buffer" the data is read from usually is a memory-mapped file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-4916) Add support for excluding commits to BackgroundObserver
[
https://issues.apache.org/jira/browse/OAK-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572294#comment-15572294
]
Stefan Egli edited comment on OAK-4916 at 10/13/16 4:00 PM:
Note that the issue with switching a filter [mentioned
here|https://issues.apache.org/jira/browse/OAK-4796?focusedCommentId=15548765=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15548765]
still applies. It would require OAK-4898 to pass the FilterProvider with each
CommitInfo into the BackgroundObserver's queue for this to work correctly.
Assuming OAK-4898 and passing the FilterProvider doesn't make it into 1.6 I'm
wondering if this should be added as a known-issue to
https://jackrabbit.apache.org/oak/docs/known_issues.html
was (Author: egli):
Note that the issue with switching a filter [mentioned
here|https://issues.apache.org/jira/browse/OAK-4796?focusedCommentId=15548765=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15548765]
still applies. It would require OAK-4898 to pass the FilterProvider with each
CommitInfo into the BackgroundObserver's queue for this to work correctly.
> Add support for excluding commits to BackgroundObserver
> ---
>
> Key: OAK-4916
> URL: https://issues.apache.org/jira/browse/OAK-4916
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: core
>Affects Versions: 1.5.11
>Reporter: Stefan Egli
>Assignee: Stefan Egli
> Fix For: 1.6
>
> Attachments: OAK-4916.patch, OAK-4916.v2.patch
>
>
> As part of pre-filtering commits it would be useful to have support in the
> BackgroundObserver (in general) that would allow to exclude certain commits
> from being added to the (BackgroundObserver's) queue, thus keeping the queue
> smaller. The actual filtering is up to subclasses.
> The suggested implementation is as follows:
> * a new method {{isExcluded}} is introduced which represents a subclass hook
> for filtering
> * excluded commits are not added to the queue
> * when multiple commits are excluded subsequently, this is collapsed
> * the first non-excluded commit (ContentChange) added to the queue is marked
> with the last non-excluded root state as the 'previous root'
> * downstream Observers are notified of the exclusion of a commit via a
> special CommitInfo {{NOOP_CHANGE}}: this instructs it to exclude this change
> while at the same time 'fast-forwarding' the root state to the new one.
> ** this extra token is one way of solving the problem that
> {{Observer.contentChanged}} represents a diff between two states but does not
> transport the 'from' state explicitly - that is implicitly taken from the
> previous call to {{contentChanged}}. Thus using such a gap token
> ({{NOOP_CHANGE}}) seems to be the only way to instruct Observers to skip a
> change.
> To repeat: whoever extends BackgroundObserver with filtering must be aware of
> the new {{NOOP_CHANGE}} token. Anyone not doing filtering will not get any
> {{NOOP_CHANGE}} tokens though.
> NOTE: See [comment further
> below|https://issues.apache.org/jira/browse/OAK-4916?focusedCommentId=15572165=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15572165]
> with a new suggested approach, which doesn't use NOOP_CHANGED but introduces
> a new FilteringAwareObserver instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4935) support prefiltering of async index updates
[ https://issues.apache.org/jira/browse/OAK-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Egli updated OAK-4935: - Fix Version/s: 1.6 > support prefiltering of async index updates > --- > > Key: OAK-4935 > URL: https://issues.apache.org/jira/browse/OAK-4935 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Affects Versions: 1.5.12 >Reporter: Stefan Egli > Fix For: 1.6 > > > As pointed out > [here|https://issues.apache.org/jira/browse/OAK-4924?focusedCommentId=15568308=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15568308] > at the moment the AsyncIndexUpdate, via SegmentNodeStore.refreshHead passes > null in the contentChanged call. This prevents prefiltering from being > applied. > [~chetanm] suggested to explicitly run the ChangeCollector ValidationProvider > in the AsyncIndexUpdate.mergeWithConcurrencyCheck (see [comment > here|https://issues.apache.org/jira/browse/OAK-4924?focusedCommentId=15568339=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15568339]). > Alternatively the AsyncIndexUpdate.mergeWithConcurrencyCheck could provide an > explicit ChangeSet representing empty sets for all (paths, names, types, > properties), as really an index update shouldn't generate anything of > interest for any jcr listener. Not sure if this is always 100% the case but > it sounds like a bit of a waste of CPU to collect hidden paths (of the > indices) in a ChangeSet which then anyway shouldn't be applicable to any > listener. But yes, it would be somewhat of a violation of the general > contract to have the ChangeSet represent all changes. Then again, we could > argue that hidden paths aren't included. > [~chetanm], wdyt? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4924) avoid CommitInfo==null in contentChanged call in SegmentNodeStore.refresh
[
https://issues.apache.org/jira/browse/OAK-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572315#comment-15572315
]
Stefan Egli commented on OAK-4924:
--
Created OAK-4935 to track this further
> avoid CommitInfo==null in contentChanged call in SegmentNodeStore.refresh
> -
>
> Key: OAK-4924
> URL: https://issues.apache.org/jira/browse/OAK-4924
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segmentmk
>Affects Versions: 1.5.12
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>
> Currently
> [SegmentNodeStore.refreshHead|https://github.com/apache/jackrabbit-oak/blob/d82f9b21d02a7eaf856f546e778118161d71b760/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStore.java#L231]
> calls {{contentChanged}} with null as the CommitInfo. This is problematic
> for prefiltering (OAK-4796, OAK-4907 et al) as for that a ChangeSet of
> changed items in a commit is stored in the CommitContext in the CommitInfo.
> Thus it would be useful to not send null but a CommitInfo that has the
> equivalent meaning of null but is a new object for each call (so that the
> ChangeSet can be set).
> [~mduerig], [~frm], [~alex.parvulescu] wdyt?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-4935) support prefiltering of async index updates
Stefan Egli created OAK-4935: Summary: support prefiltering of async index updates Key: OAK-4935 URL: https://issues.apache.org/jira/browse/OAK-4935 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.5.12 Reporter: Stefan Egli As pointed out [here|https://issues.apache.org/jira/browse/OAK-4924?focusedCommentId=15568308=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15568308] at the moment the AsyncIndexUpdate, via SegmentNodeStore.refreshHead passes null in the contentChanged call. This prevents prefiltering from being applied. [~chetanm] suggested to explicitly run the ChangeCollector ValidationProvider in the AsyncIndexUpdate.mergeWithConcurrencyCheck (see [comment here|https://issues.apache.org/jira/browse/OAK-4924?focusedCommentId=15568339=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15568339]). Alternatively the AsyncIndexUpdate.mergeWithConcurrencyCheck could provide an explicit ChangeSet representing empty sets for all (paths, names, types, properties), as really an index update shouldn't generate anything of interest for any jcr listener. Not sure if this is always 100% the case but it sounds like a bit of a waste of CPU to collect hidden paths (of the indices) in a ChangeSet which then anyway shouldn't be applicable to any listener. But yes, it would be somewhat of a violation of the general contract to have the ChangeSet represent all changes. Then again, we could argue that hidden paths aren't included. [~chetanm], wdyt? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4916) Add support for excluding commits to BackgroundObserver
[
https://issues.apache.org/jira/browse/OAK-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572294#comment-15572294
]
Stefan Egli commented on OAK-4916:
--
Note that the issue with switching a filter [mentioned
here|https://issues.apache.org/jira/browse/OAK-4796?focusedCommentId=15548765=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15548765]
still applies. It would require OAK-4898 to pass the FilterProvider with each
CommitInfo into the BackgroundObserver's queue for this to work correctly.
> Add support for excluding commits to BackgroundObserver
> ---
>
> Key: OAK-4916
> URL: https://issues.apache.org/jira/browse/OAK-4916
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: core
>Affects Versions: 1.5.11
>Reporter: Stefan Egli
>Assignee: Stefan Egli
> Fix For: 1.6
>
> Attachments: OAK-4916.patch, OAK-4916.v2.patch
>
>
> As part of pre-filtering commits it would be useful to have support in the
> BackgroundObserver (in general) that would allow to exclude certain commits
> from being added to the (BackgroundObserver's) queue, thus keeping the queue
> smaller. The actual filtering is up to subclasses.
> The suggested implementation is as follows:
> * a new method {{isExcluded}} is introduced which represents a subclass hook
> for filtering
> * excluded commits are not added to the queue
> * when multiple commits are excluded subsequently, this is collapsed
> * the first non-excluded commit (ContentChange) added to the queue is marked
> with the last non-excluded root state as the 'previous root'
> * downstream Observers are notified of the exclusion of a commit via a
> special CommitInfo {{NOOP_CHANGE}}: this instructs it to exclude this change
> while at the same time 'fast-forwarding' the root state to the new one.
> ** this extra token is one way of solving the problem that
> {{Observer.contentChanged}} represents a diff between two states but does not
> transport the 'from' state explicitly - that is implicitly taken from the
> previous call to {{contentChanged}}. Thus using such a gap token
> ({{NOOP_CHANGE}}) seems to be the only way to instruct Observers to skip a
> change.
> To repeat: whoever extends BackgroundObserver with filtering must be aware of
> the new {{NOOP_CHANGE}} token. Anyone not doing filtering will not get any
> {{NOOP_CHANGE}} tokens though.
> NOTE: See [comment further
> below|https://issues.apache.org/jira/browse/OAK-4916?focusedCommentId=15572165=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15572165]
> with a new suggested approach, which doesn't use NOOP_CHANGED but introduces
> a new FilteringAwareObserver instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4931) LdapIdentityProvider doesn't use configured custom attributes for all searches
[ https://issues.apache.org/jira/browse/OAK-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke updated OAK-4931: Fix Version/s: 1.4.9 1.6 > LdapIdentityProvider doesn't use configured custom attributes for all searches > -- > > Key: OAK-4931 > URL: https://issues.apache.org/jira/browse/OAK-4931 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-ldap >Affects Versions: 1.4.8, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.6, 1.4.9, 1.5.13 > > > The list of custom attibutes retrieved when looking up LDAP entries is not > used for every search. Most notably LdapIdentityProvider.getIdentity() still > retrieves all user attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4908) Best-effort prefiltering in ChangeProcessor based on ChangeSet
[
https://issues.apache.org/jira/browse/OAK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Egli updated OAK-4908:
-
Attachment: OAK-4908.v2.patch
Attaching updated patch [^OAK-4908.v2.patch] which is based on v2 of OAK-4916
patch (which switches from NOOP_CHANGE to {{FilteringAwareObserver}}). (Can
only be committed after OAK-4916)
> Best-effort prefiltering in ChangeProcessor based on ChangeSet
> --
>
> Key: OAK-4908
> URL: https://issues.apache.org/jira/browse/OAK-4908
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: core, jcr
>Affects Versions: 1.5.11
>Reporter: Stefan Egli
>Assignee: Stefan Egli
> Fix For: 1.6
>
> Attachments: OAK-4908.patch, OAK-4908.v2.patch
>
>
> This is a subtask as a result of
> [discussions|https://issues.apache.org/jira/browse/OAK-4796?focusedCommentId=15550962=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15550962]
> around design in OAK-4796:
> Based on the ChangeSet provided with OAK-4907 in the CommitContext, the
> ChangeProcessor should do a best-effort prefiltering of commits before they
> get added to the (BackgroundObserver's) queue.
> This consists of the following parts:
> * -the support for optionally excluding commits from being added to the queue
> in the BackgroundObserver- EDIT: factored that out into OAK-4916
> * -the BackgroundObserver signaling downstream Observers that a change should
> be excluded via a {{NOOP_CHANGE}} CommitInfo- EDIT: factored that out into
> OAK-4916
> * the ChangeProcessor using OAK-4907's ChangeSet of the CommitContext for
> best-effort prefiltering - and handling the {{NOOP_CHANGED}} CommitInfo
> introduced in OAK-4916
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4916) Add support for excluding commits to BackgroundObserver
[
https://issues.apache.org/jira/browse/OAK-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Egli updated OAK-4916:
-
Description:
As part of pre-filtering commits it would be useful to have support in the
BackgroundObserver (in general) that would allow to exclude certain commits
from being added to the (BackgroundObserver's) queue, thus keeping the queue
smaller. The actual filtering is up to subclasses.
The suggested implementation is as follows:
* a new method {{isExcluded}} is introduced which represents a subclass hook
for filtering
* excluded commits are not added to the queue
* when multiple commits are excluded subsequently, this is collapsed
* the first non-excluded commit (ContentChange) added to the queue is marked
with the last non-excluded root state as the 'previous root'
* downstream Observers are notified of the exclusion of a commit via a special
CommitInfo {{NOOP_CHANGE}}: this instructs it to exclude this change while at
the same time 'fast-forwarding' the root state to the new one.
** this extra token is one way of solving the problem that
{{Observer.contentChanged}} represents a diff between two states but does not
transport the 'from' state explicitly - that is implicitly taken from the
previous call to {{contentChanged}}. Thus using such a gap token
({{NOOP_CHANGE}}) seems to be the only way to instruct Observers to skip a
change.
To repeat: whoever extends BackgroundObserver with filtering must be aware of
the new {{NOOP_CHANGE}} token. Anyone not doing filtering will not get any
{{NOOP_CHANGE}} tokens though.
NOTE: See [comment further
below|https://issues.apache.org/jira/browse/OAK-4916?focusedCommentId=15572165=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15572165]
with a new suggested approach, which doesn't use NOOP_CHANGED but introduces a
new FilteringAwareObserver instead.
was:
As part of pre-filtering commits it would be useful to have support in the
BackgroundObserver (in general) that would allow to exclude certain commits
from being added to the (BackgroundObserver's) queue, thus keeping the queue
smaller. The actual filtering is up to subclasses.
The suggested implementation is as follows:
* a new method {{isExcluded}} is introduced which represents a subclass hook
for filtering
* excluded commits are not added to the queue
* when multiple commits are excluded subsequently, this is collapsed
* the first non-excluded commit (ContentChange) added to the queue is marked
with the last non-excluded root state as the 'previous root'
* downstream Observers are notified of the exclusion of a commit via a special
CommitInfo {{NOOP_CHANGE}}: this instructs it to exclude this change while at
the same time 'fast-forwarding' the root state to the new one.
** this extra token is one way of solving the problem that
{{Observer.contentChanged}} represents a diff between two states but does not
transport the 'from' state explicitly - that is implicitly taken from the
previous call to {{contentChanged}}. Thus using such a gap token
({{NOOP_CHANGE}}) seems to be the only way to instruct Observers to skip a
change.
To repeat: whoever extends BackgroundObserver with filtering must be aware of
the new {{NOOP_CHANGE}} token. Anyone not doing filtering will not get any
{{NOOP_CHANGE}} tokens though.
> Add support for excluding commits to BackgroundObserver
> ---
>
> Key: OAK-4916
> URL: https://issues.apache.org/jira/browse/OAK-4916
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: core
>Affects Versions: 1.5.11
>Reporter: Stefan Egli
>Assignee: Stefan Egli
> Fix For: 1.6
>
> Attachments: OAK-4916.patch, OAK-4916.v2.patch
>
>
> As part of pre-filtering commits it would be useful to have support in the
> BackgroundObserver (in general) that would allow to exclude certain commits
> from being added to the (BackgroundObserver's) queue, thus keeping the queue
> smaller. The actual filtering is up to subclasses.
> The suggested implementation is as follows:
> * a new method {{isExcluded}} is introduced which represents a subclass hook
> for filtering
> * excluded commits are not added to the queue
> * when multiple commits are excluded subsequently, this is collapsed
> * the first non-excluded commit (ContentChange) added to the queue is marked
> with the last non-excluded root state as the 'previous root'
> * downstream Observers are notified of the exclusion of a commit via a
> special CommitInfo {{NOOP_CHANGE}}: this instructs it to exclude this change
> while at the same time 'fast-forwarding' the root state to the new one.
> ** this extra token is one way of solving the problem that
> {{Observer.contentChanged}} represents a diff between two states but does not
>
[jira] [Comment Edited] (OAK-4931) LdapIdentityProvider doesn't use configured custom attributes for all searches
[ https://issues.apache.org/jira/browse/OAK-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572215#comment-15572215 ] Manfred Baedke edited comment on OAK-4931 at 10/13/16 3:21 PM: --- Fixed in trunk: c1764705. Fixed in 1.4: c1764706. was (Author: baedke): Fixed in trunk: c1764705. Fixed in 1.4: c1764706 > LdapIdentityProvider doesn't use configured custom attributes for all searches > -- > > Key: OAK-4931 > URL: https://issues.apache.org/jira/browse/OAK-4931 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-ldap >Affects Versions: 1.4.8, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.6, 1.4.9, 1.5.13 > > > The list of custom attibutes retrieved when looking up LDAP entries is not > used for every search. Most notably LdapIdentityProvider.getIdentity() still > retrieves all user attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-4931) LdapIdentityProvider doesn't use configured custom attributes for all searches
[ https://issues.apache.org/jira/browse/OAK-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke resolved OAK-4931. - Resolution: Fixed > LdapIdentityProvider doesn't use configured custom attributes for all searches > -- > > Key: OAK-4931 > URL: https://issues.apache.org/jira/browse/OAK-4931 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-ldap >Affects Versions: 1.4.8, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.6, 1.4.9, 1.5.13 > > > The list of custom attibutes retrieved when looking up LDAP entries is not > used for every search. Most notably LdapIdentityProvider.getIdentity() still > retrieves all user attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-4931) LdapIdentityProvider doesn't use configured custom attributes for all searches
[ https://issues.apache.org/jira/browse/OAK-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572215#comment-15572215 ] Manfred Baedke edited comment on OAK-4931 at 10/13/16 3:21 PM: --- Fixed in trunk: c1764705. Fixed in 1.4: c1764706 was (Author: baedke): Fixed in trunk: c1764705. > LdapIdentityProvider doesn't use configured custom attributes for all searches > -- > > Key: OAK-4931 > URL: https://issues.apache.org/jira/browse/OAK-4931 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-ldap >Affects Versions: 1.4.8, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.6, 1.4.9, 1.5.13 > > > The list of custom attibutes retrieved when looking up LDAP entries is not > used for every search. Most notably LdapIdentityProvider.getIdentity() still > retrieves all user attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-4916) Add support for excluding commits to BackgroundObserver
[
https://issues.apache.org/jira/browse/OAK-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572165#comment-15572165
]
Stefan Egli edited comment on OAK-4916 at 10/13/16 3:19 PM:
Attaching [^OAK-4916.v2.patch] which is an alternative variant of filtering in
the BackgroundObserver:
* as suggested by Chetan introduced a {{FilteringAwareObserver}} which is
implemented by BackgroundObserver and thus moves the filtering _decision_ to an
upstream Observer. Thus an element doesn't even get passed to the
BackgroundObserver if it is filtered. However, what the BackgroundObserver must
do is to correctly pass this exclusion to its downstream Observer - and it must
take special care when the queue is full: as basically then exclusion doesn't
work.
* added a base class for doing the upstream filtering in {{FilteringObserver}}
* added several tests to the existing BackgroundObserverTest as well as added a
new PrefilteringBackgroundObserverTest.
* NOTE: downstream observers of a BackgroundObserver should now implement
FilteringAwareObserver in order to correctly get the filtering signal. If they
don't they will get a normal contentChanged however they then loose the
CommitInfo (and the commit-boundary might be wrong). Ie a BackgroundObserver
shouldn't be used in a chain where filtering is applied (ie where there is an
upstream FilteringObserver) to avoid lossing the CommitInfo.
[~chetanm], [~mduerig], wdyt?
was (Author: egli):
Attaching [^OAK-4916.v2.patch] which is an alternative variant of filtering in
the BackgroundObserver:
* as suggested by Chetan introduced a {{FilteringAwareObserver}} which is
implemented by BackgroundObserver and thus moves the filtering _decision_ to an
upstream Observer. Thus an element doesn't even get passed to the
BackgroundObserver if it is filtered. However, what the BackgroundObserver must
do is to correctly pass this exclusion to its downstream Observer - and it must
take special care when the queue is full: as basically then exclusion doesn't
work.
* added a base class for doing the upstream filtering in {{FilteringObserver}}
* added several tests to the existing BackgroundObserverTest as well as added a
new PrefilteringBackgroundObserverTest.
* NOTE: downstream observers of a BackgroundObserver should now implement
FilteringAwareObserver in order to correctly get the filtering signal. If they
don't they will get a normal contentChanged however they then loose the
CommitInfo. Ie a BackgroundObserver shouldn't be used in a chain where
filtering is applied (ie where there is an upstream FilteringObserver) to avoid
lossing the CommitInfo.
[~chetanm], [~mduerig], wdyt?
> Add support for excluding commits to BackgroundObserver
> ---
>
> Key: OAK-4916
> URL: https://issues.apache.org/jira/browse/OAK-4916
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: core
>Affects Versions: 1.5.11
>Reporter: Stefan Egli
>Assignee: Stefan Egli
> Fix For: 1.6
>
> Attachments: OAK-4916.patch, OAK-4916.v2.patch
>
>
> As part of pre-filtering commits it would be useful to have support in the
> BackgroundObserver (in general) that would allow to exclude certain commits
> from being added to the (BackgroundObserver's) queue, thus keeping the queue
> smaller. The actual filtering is up to subclasses.
> The suggested implementation is as follows:
> * a new method {{isExcluded}} is introduced which represents a subclass hook
> for filtering
> * excluded commits are not added to the queue
> * when multiple commits are excluded subsequently, this is collapsed
> * the first non-excluded commit (ContentChange) added to the queue is marked
> with the last non-excluded root state as the 'previous root'
> * downstream Observers are notified of the exclusion of a commit via a
> special CommitInfo {{NOOP_CHANGE}}: this instructs it to exclude this change
> while at the same time 'fast-forwarding' the root state to the new one.
> ** this extra token is one way of solving the problem that
> {{Observer.contentChanged}} represents a diff between two states but does not
> transport the 'from' state explicitly - that is implicitly taken from the
> previous call to {{contentChanged}}. Thus using such a gap token
> ({{NOOP_CHANGE}}) seems to be the only way to instruct Observers to skip a
> change.
> To repeat: whoever extends BackgroundObserver with filtering must be aware of
> the new {{NOOP_CHANGE}} token. Anyone not doing filtering will not get any
> {{NOOP_CHANGE}} tokens though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4931) LdapIdentityProvider doesn't use configured custom attributes for all searches
[ https://issues.apache.org/jira/browse/OAK-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke updated OAK-4931: Fix Version/s: 1.5.12 > LdapIdentityProvider doesn't use configured custom attributes for all searches > -- > > Key: OAK-4931 > URL: https://issues.apache.org/jira/browse/OAK-4931 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-ldap >Affects Versions: 1.4.8, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.6, 1.4.9, 1.5.13 > > > The list of custom attibutes retrieved when looking up LDAP entries is not > used for every search. Most notably LdapIdentityProvider.getIdentity() still > retrieves all user attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4931) LdapIdentityProvider doesn't use configured custom attributes for all searches
[ https://issues.apache.org/jira/browse/OAK-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke updated OAK-4931: Fix Version/s: (was: 1.5.12) 1.5.13 > LdapIdentityProvider doesn't use configured custom attributes for all searches > -- > > Key: OAK-4931 > URL: https://issues.apache.org/jira/browse/OAK-4931 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-ldap >Affects Versions: 1.4.8, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.6, 1.4.9, 1.5.13 > > > The list of custom attibutes retrieved when looking up LDAP entries is not > used for every search. Most notably LdapIdentityProvider.getIdentity() still > retrieves all user attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4931) LdapIdentityProvider doesn't use configured custom attributes for all searches
[ https://issues.apache.org/jira/browse/OAK-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572215#comment-15572215 ] Manfred Baedke commented on OAK-4931: - Fixed in trunk: c1764705. > LdapIdentityProvider doesn't use configured custom attributes for all searches > -- > > Key: OAK-4931 > URL: https://issues.apache.org/jira/browse/OAK-4931 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-ldap >Affects Versions: 1.4.8, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > > The list of custom attibutes retrieved when looking up LDAP entries is not > used for every search. Most notably LdapIdentityProvider.getIdentity() still > retrieves all user attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-4916) Add support for excluding commits to BackgroundObserver
[
https://issues.apache.org/jira/browse/OAK-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572165#comment-15572165
]
Stefan Egli edited comment on OAK-4916 at 10/13/16 3:03 PM:
Attaching [^OAK-4916.v2.patch] which is an alternative variant of filtering in
the BackgroundObserver:
* as suggested by Chetan introduced a {{FilteringAwareObserver}} which is
implemented by BackgroundObserver and thus moves the filtering _decision_ to an
upstream Observer. Thus an element doesn't even get passed to the
BackgroundObserver if it is filtered. However, what the BackgroundObserver must
do is to correctly pass this exclusion to its downstream Observer - and it must
take special care when the queue is full: as basically then exclusion doesn't
work.
* added a base class for doing the upstream filtering in {{FilteringObserver}}
* added several tests to the existing BackgroundObserverTest as well as added a
new PrefilteringBackgroundObserverTest.
* NOTE: downstream observers of a BackgroundObserver should now implement
FilteringAwareObserver in order to correctly get the filtering signal. If they
don't they will get a normal contentChanged however they then loose the
CommitInfo. Ie a BackgroundObserver shouldn't be used in a chain where
filtering is applied (ie where there is an upstream FilteringObserver) to avoid
lossing the CommitInfo.
[~chetanm], [~mduerig], wdyt?
was (Author: egli):
Attaching [^OAK-4916.v2.patch] which is an alternative variant of filtering in
the BackgroundObserver:
* as suggested by Chetan introduced a {{FilteringAwareObserver}} which is
implemented by BackgroundObserver and thus moves the filtering _decision_ to an
upstream Observer. Thus an element doesn't even get passed to the
BackgroundObserver if it is filtered. However, what the BackgroundObserver must
do is to correctly pass this exclusion to its downstream Observer - and it must
take special care when the queue is full: as basically then exclusion doesn't
work.
* added a base class for doing the upstream filtering in {{FilteringObserver}}
* added several tests to the existing BackgroundObserverTest as well as added a
new PrefilteringBackgroundObserverTest.
[~chetanm], [~mduerig], wdyt?
> Add support for excluding commits to BackgroundObserver
> ---
>
> Key: OAK-4916
> URL: https://issues.apache.org/jira/browse/OAK-4916
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: core
>Affects Versions: 1.5.11
>Reporter: Stefan Egli
>Assignee: Stefan Egli
> Fix For: 1.6
>
> Attachments: OAK-4916.patch, OAK-4916.v2.patch
>
>
> As part of pre-filtering commits it would be useful to have support in the
> BackgroundObserver (in general) that would allow to exclude certain commits
> from being added to the (BackgroundObserver's) queue, thus keeping the queue
> smaller. The actual filtering is up to subclasses.
> The suggested implementation is as follows:
> * a new method {{isExcluded}} is introduced which represents a subclass hook
> for filtering
> * excluded commits are not added to the queue
> * when multiple commits are excluded subsequently, this is collapsed
> * the first non-excluded commit (ContentChange) added to the queue is marked
> with the last non-excluded root state as the 'previous root'
> * downstream Observers are notified of the exclusion of a commit via a
> special CommitInfo {{NOOP_CHANGE}}: this instructs it to exclude this change
> while at the same time 'fast-forwarding' the root state to the new one.
> ** this extra token is one way of solving the problem that
> {{Observer.contentChanged}} represents a diff between two states but does not
> transport the 'from' state explicitly - that is implicitly taken from the
> previous call to {{contentChanged}}. Thus using such a gap token
> ({{NOOP_CHANGE}}) seems to be the only way to instruct Observers to skip a
> change.
> To repeat: whoever extends BackgroundObserver with filtering must be aware of
> the new {{NOOP_CHANGE}} token. Anyone not doing filtering will not get any
> {{NOOP_CHANGE}} tokens though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4916) Add support for excluding commits to BackgroundObserver
[
https://issues.apache.org/jira/browse/OAK-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Egli updated OAK-4916:
-
Attachment: OAK-4916.v2.patch
Attaching [^OAK-4916.v2.patch] which is an alternative variant of filtering in
the BackgroundObserver:
* as suggested by Chetan introduced a {{FilteringAwareObserver}} which is
implemented by BackgroundObserver and thus moves the filtering _decision_ to an
upstream Observer. Thus an element doesn't even get passed to the
BackgroundObserver if it is filtered. However, what the BackgroundObserver must
do is to correctly pass this exclusion to its downstream Observer - and it must
take special care when the queue is full: as basically then exclusion doesn't
work.
* added a base class for doing the upstream filtering in {{FilteringObserver}}
* added several tests to the existing BackgroundObserverTest as well as added a
new PrefilteringBackgroundObserverTest.
[~chetanm], [~mduerig], wdyt?
> Add support for excluding commits to BackgroundObserver
> ---
>
> Key: OAK-4916
> URL: https://issues.apache.org/jira/browse/OAK-4916
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: core
>Affects Versions: 1.5.11
>Reporter: Stefan Egli
>Assignee: Stefan Egli
> Fix For: 1.6
>
> Attachments: OAK-4916.patch, OAK-4916.v2.patch
>
>
> As part of pre-filtering commits it would be useful to have support in the
> BackgroundObserver (in general) that would allow to exclude certain commits
> from being added to the (BackgroundObserver's) queue, thus keeping the queue
> smaller. The actual filtering is up to subclasses.
> The suggested implementation is as follows:
> * a new method {{isExcluded}} is introduced which represents a subclass hook
> for filtering
> * excluded commits are not added to the queue
> * when multiple commits are excluded subsequently, this is collapsed
> * the first non-excluded commit (ContentChange) added to the queue is marked
> with the last non-excluded root state as the 'previous root'
> * downstream Observers are notified of the exclusion of a commit via a
> special CommitInfo {{NOOP_CHANGE}}: this instructs it to exclude this change
> while at the same time 'fast-forwarding' the root state to the new one.
> ** this extra token is one way of solving the problem that
> {{Observer.contentChanged}} represents a diff between two states but does not
> transport the 'from' state explicitly - that is implicitly taken from the
> previous call to {{contentChanged}}. Thus using such a gap token
> ({{NOOP_CHANGE}}) seems to be the only way to instruct Observers to skip a
> change.
> To repeat: whoever extends BackgroundObserver with filtering must be aware of
> the new {{NOOP_CHANGE}} token. Anyone not doing filtering will not get any
> {{NOOP_CHANGE}} tokens though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-4930) External Principal Management: DynamicSyncContext makes redundant calls to IdentityProvider.getIdentity()
[ https://issues.apache.org/jira/browse/OAK-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571993#comment-15571993 ] Manfred Baedke edited comment on OAK-4930 at 10/13/16 2:59 PM: --- Fixed in trunk: c1764678. Fixed in 1.4: c1764700, c1764702 was (Author: baedke): Fixed in trunk: r1764678. Fixed in 1.4: r1764700 > External Principal Management: DynamicSyncContext makes redundant calls to > IdentityProvider.getIdentity() > -- > > Key: OAK-4930 > URL: https://issues.apache.org/jira/browse/OAK-4930 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-external >Affects Versions: 1.4.8, 1.2.20, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.6, 1.4.9, 1.5.13 > > > When recursively collecting principal names associated with the declared > group memberships of a given principal, the method > DynamicSyncContext.collectPrincipalNames() unnecessarily calls > IdentityProvider.getIdentity() for every declared group reference, though it > is only necessary when the actual depth of the remaining recursion is >1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4344) LdapIdentityProvider always retrieves all attributes when looking up an LDAP entity.
[ https://issues.apache.org/jira/browse/OAK-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke updated OAK-4344: Fix Version/s: 1.4.7 1.2.19 > LdapIdentityProvider always retrieves all attributes when looking up an LDAP > entity. > > > Key: OAK-4344 > URL: https://issues.apache.org/jira/browse/OAK-4344 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Manfred Baedke >Assignee: Manfred Baedke >Priority: Minor > Fix For: 1.6, 1.5.7, 1.4.7, 1.2.19 > > > Retrieving all attributes is usually unnecessary and may be costly. The > current behavior should be the default, but it should also be possible to > configure an explicit list of attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-4907) Collect changes (paths, nts, props..) of a commit in a validator
[ https://issues.apache.org/jira/browse/OAK-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571645#comment-15571645 ] Stefan Egli edited comment on OAK-4907 at 10/13/16 2:49 PM: Committed in http://svn.apache.org/viewvc?rev=1764657=rev and http://svn.apache.org/viewvc?rev=1764701=rev /cc [~chetanm], applied your suggestions, thx was (Author: egli): Committed in http://svn.apache.org/viewvc?rev=1764657=rev /cc [~chetanm], applied your suggestions, thx > Collect changes (paths, nts, props..) of a commit in a validator > > > Key: OAK-4907 > URL: https://issues.apache.org/jira/browse/OAK-4907 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: core >Affects Versions: 1.5.11 >Reporter: Stefan Egli >Assignee: Stefan Egli > Fix For: 1.5.13 > > Attachments: OAK-4907.patch, OAK-4907.v2.patch > > > It would be useful to collect a set of changes of a commit (eg in a > validator) that could later be used in an Observer for eg prefiltering. > Such a change collector should collect paths, nodetypes, properties, > node-names (and perhaps more at a later stage) of all changes and store the > result in the CommitInfo's CommitContext. > Note that this is a result of > [discussions|https://issues.apache.org/jira/browse/OAK-4796?focusedCommentId=15550962=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15550962] > around design in OAK-4796 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-4930) External Principal Management: DynamicSyncContext makes redundant calls to IdentityProvider.getIdentity()
[ https://issues.apache.org/jira/browse/OAK-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke resolved OAK-4930. - Resolution: Fixed > External Principal Management: DynamicSyncContext makes redundant calls to > IdentityProvider.getIdentity() > -- > > Key: OAK-4930 > URL: https://issues.apache.org/jira/browse/OAK-4930 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-external >Affects Versions: 1.4.8, 1.2.20, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.6, 1.4.9, 1.5.13 > > > When recursively collecting principal names associated with the declared > group memberships of a given principal, the method > DynamicSyncContext.collectPrincipalNames() unnecessarily calls > IdentityProvider.getIdentity() for every declared group reference, though it > is only necessary when the actual depth of the remaining recursion is >1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4930) External Principal Management: DynamicSyncContext makes redundant calls to IdentityProvider.getIdentity()
[ https://issues.apache.org/jira/browse/OAK-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke updated OAK-4930: Fix Version/s: 1.6 > External Principal Management: DynamicSyncContext makes redundant calls to > IdentityProvider.getIdentity() > -- > > Key: OAK-4930 > URL: https://issues.apache.org/jira/browse/OAK-4930 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-external >Affects Versions: 1.4.8, 1.2.20, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.6, 1.4.9, 1.5.13 > > > When recursively collecting principal names associated with the declared > group memberships of a given principal, the method > DynamicSyncContext.collectPrincipalNames() unnecessarily calls > IdentityProvider.getIdentity() for every declared group reference, though it > is only necessary when the actual depth of the remaining recursion is >1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4930) External Principal Management: DynamicSyncContext makes redundant calls to IdentityProvider.getIdentity()
[ https://issues.apache.org/jira/browse/OAK-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke updated OAK-4930: Fix Version/s: 1.5.13 1.4.9 > External Principal Management: DynamicSyncContext makes redundant calls to > IdentityProvider.getIdentity() > -- > > Key: OAK-4930 > URL: https://issues.apache.org/jira/browse/OAK-4930 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-external >Affects Versions: 1.4.8, 1.2.20, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.4.9, 1.5.13 > > > When recursively collecting principal names associated with the declared > group memberships of a given principal, the method > DynamicSyncContext.collectPrincipalNames() unnecessarily calls > IdentityProvider.getIdentity() for every declared group reference, though it > is only necessary when the actual depth of the remaining recursion is >1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-4930) External Principal Management: DynamicSyncContext makes redundant calls to IdentityProvider.getIdentity()
[ https://issues.apache.org/jira/browse/OAK-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571993#comment-15571993 ] Manfred Baedke edited comment on OAK-4930 at 10/13/16 2:45 PM: --- Fixed in trunk: r1764678. Fixed in 1.4: r1764700 was (Author: baedke): Fixed in trunk: r1764678. > External Principal Management: DynamicSyncContext makes redundant calls to > IdentityProvider.getIdentity() > -- > > Key: OAK-4930 > URL: https://issues.apache.org/jira/browse/OAK-4930 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-external >Affects Versions: 1.4.8, 1.2.20, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.4.9, 1.5.13 > > > When recursively collecting principal names associated with the declared > group memberships of a given principal, the method > DynamicSyncContext.collectPrincipalNames() unnecessarily calls > IdentityProvider.getIdentity() for every declared group reference, though it > is only necessary when the actual depth of the remaining recursion is >1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4930) External Principal Management: DynamicSyncContext makes redundant calls to IdentityProvider.getIdentity()
[ https://issues.apache.org/jira/browse/OAK-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571993#comment-15571993 ] Manfred Baedke commented on OAK-4930: - Fixed in trunk: r1764678. > External Principal Management: DynamicSyncContext makes redundant calls to > IdentityProvider.getIdentity() > -- > > Key: OAK-4930 > URL: https://issues.apache.org/jira/browse/OAK-4930 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-external >Affects Versions: 1.4.8, 1.2.20, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > > When recursively collecting principal names associated with the declared > group memberships of a given principal, the method > DynamicSyncContext.collectPrincipalNames() unnecessarily calls > IdentityProvider.getIdentity() for every declared group reference, though it > is only necessary when the actual depth of the remaining recursion is >1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4930) External Principal Management: DynamicSyncContext makes redundant calls to IdentityProvider.getIdentity()
[ https://issues.apache.org/jira/browse/OAK-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke updated OAK-4930: Issue Type: Bug (was: Improvement) > External Principal Management: DynamicSyncContext makes redundant calls to > IdentityProvider.getIdentity() > -- > > Key: OAK-4930 > URL: https://issues.apache.org/jira/browse/OAK-4930 > Project: Jackrabbit Oak > Issue Type: Bug > Components: auth-external >Affects Versions: 1.4.8, 1.2.20, 1.5.12 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > > When recursively collecting principal names associated with the declared > group memberships of a given principal, the method > DynamicSyncContext.collectPrincipalNames() unnecessarily calls > IdentityProvider.getIdentity() for every declared group reference, though it > is only necessary when the actual depth of the remaining recursion is >1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4934) Query shapes for JCR Query
[
https://issues.apache.org/jira/browse/OAK-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571737#comment-15571737
]
Thomas Mueller commented on OAK-4934:
-
Sure, it can be done.
In some cases, knowing the values is important, and the values might have a
(more or less) big impact on query time and cost. For example, for a property
called "status", there might be "status='active'" with 1000 entries, and
"status='inactive'" with 1 million entries. The property index does distinguish
between the two (cost estimates are for each value). Special are nodetype
restrictions, and path restrictions.
What about using a utility (that internally uses regular expression logic or
similar) to reduce the details? The query could be transformed depending on the
need:
* Detail level 3 = as of now
* Detail level 2 = without literals, but with path and nodetypes
* Detail level 1 = without literals, but with path
* Detail level 0 = without any literals
Where literals are strings, numbers, dates, and so on.
> Query shapes for JCR Query
> --
>
> Key: OAK-4934
> URL: https://issues.apache.org/jira/browse/OAK-4934
> Project: Jackrabbit Oak
> Issue Type: Wish
> Components: query
>Reporter: Chetan Mehrotra
>
> For certain requirements it would be good to have a notion/support to deduce
> query shape [1]
> {quote}
> A combination of query predicate, sort, and projection specifications.
> For the query predicate, only the structure of the predicate, including the
> field names, are significant; the values in the query predicate are
> insignificant. As such, a query predicate \{ type: 'food' \} is equivalent to
> the query predicate \{ type: 'utensil' \} for a query shape.
> {quote}
> So transforming that to Oak the shape should represent a JCR-SQL2 query
> string (xpath query gets transformed to SQL2) which is a *canonical*
> representation of actual query ignoring the property restriction values.
> Example we have 2 queries
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'published'
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'disabled'
> The query shape would be
> SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
> The plan for query having given shape would remain same irrespective of value
> of property restrictions. Path restriction can cause some difference though
> The shape can then be used for
> * Stats Collection - Currently stats collection gets overflown if same query
> with different value gets invoked
> * Allow configuring hints - See support in Mongo [2] for an example. One
> specify via config that for a query of such and such shape this index should
> be used
> * Less noisy diagnostics - If a query gets invoked with bad plan the QE can
> log the warning once instead of logging it for each query invocation
> involving different values.
> [1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
> [2] https://docs.mongodb.com/manual/reference/command/planCacheSetFilter/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4934) Query shapes for JCR Query
[
https://issues.apache.org/jira/browse/OAK-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-4934:
-
Description:
For certain requirements it would be good to have a notion/support to deduce
query shape [1]
{quote}
A combination of query predicate, sort, and projection specifications.
For the query predicate, only the structure of the predicate, including the
field names, are significant; the values in the query predicate are
insignificant. As such, a query predicate \{ type: 'food' \} is equivalent to
the query predicate \{ type: 'utensil' \} for a query shape.
{quote}
So transforming that to Oak the shape should represent a JCR-SQL2 query string
(xpath query gets transformed to SQL2) which is a *canonical* representation of
actual query ignoring the property restriction values.
Example we have 2 queries
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'published'
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'disabled'
The query shape would be
SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
The plan for query having given shape would remain same irrespective of value
of property restrictions. Path restriction can cause some difference though
The shape can then be used for
* Stats Collection - Currently stats collection gets overflown if same query
with different value gets invoked
* Allow configuring hints - See support in Mongo [2] for an example. One
specify via config that for a query of such and such shape this index should be
used
* Less noisy diagnostics - If a query gets invoked with bad plan the QE can log
the warning once instead of logging it for each query invocation involving
different values.
[1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
[2] https://docs.mongodb.com/manual/reference/command/planCacheSetFilter/
was:
For certain requirements it would be good to have a notion/support to deduce
query shape [1]
{quote}
A combination of query predicate, sort, and projection specifications.
For the query predicate, only the structure of the predicate, including the
field names, are significant; the values in the query predicate are
insignificant. As such, a query predicate \{ type: 'food' \} is equivalent to
the query predicate \{ type: 'utensil' \} for a query shape.
{quote}
So transforming that to Oak the shape should represent a JCR-SQL2 query string
(xpath query gets transformed to SQL2) which is a *canonical* representation of
actual query ignoring the property restriction values.
Example we have 2 queries
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'published'
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'disabled'
The query shape would be
SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
The plan for query having given shape would remain same irrespective of value
of property restrictions
The shape can then be used for
* Stats Collection - Currently stats collection gets overflown if same query
with different value gets invoked
* Allow configuring hints - See support in Mongo [2] for an example. One
specify via config that for a query of such and such shape this index should be
used
* Less noisy diagnostics - If a query gets invoked with bad plan the QE can log
the warning once instead of logging it for each query invocation involving
different values.
[1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
[2] https://docs.mongodb.com/manual/reference/command/planCacheSetFilter/
> Query shapes for JCR Query
> --
>
> Key: OAK-4934
> URL: https://issues.apache.org/jira/browse/OAK-4934
> Project: Jackrabbit Oak
> Issue Type: Wish
> Components: query
>Reporter: Chetan Mehrotra
>
> For certain requirements it would be good to have a notion/support to deduce
> query shape [1]
> {quote}
> A combination of query predicate, sort, and projection specifications.
> For the query predicate, only the structure of the predicate, including the
> field names, are significant; the values in the query predicate are
> insignificant. As such, a query predicate \{ type: 'food' \} is equivalent to
> the query predicate \{ type: 'utensil' \} for a query shape.
> {quote}
> So transforming that to Oak the shape should represent a JCR-SQL2 query
> string (xpath query gets transformed to SQL2) which is a *canonical*
> representation of actual query ignoring the property restriction values.
> Example we have 2 queries
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'published'
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'disabled'
> The query shape would be
> SELECT * FROM
[jira] [Updated] (OAK-4934) Query shapes for JCR Query
[
https://issues.apache.org/jira/browse/OAK-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-4934:
-
Description:
For certain requirements it would be good to have a notion/support to deduce
query shape [1]
{quote}
A combination of query predicate, sort, and projection specifications.
For the query predicate, only the structure of the predicate, including the
field names, are significant; the values in the query predicate are
insignificant. As such, a query predicate \{ type: 'food' \} is equivalent to
the query predicate \{ type: 'utensil' \} for a query shape.
{quote}
So transforming that to Oak the shape should represent a JCR-SQL2 query string
(xpath query gets transformed to SQL2) which is a *canonical* representation of
actual query ignoring the property restriction values.
Example we have 2 queries
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'published'
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'disabled'
The query shape would be
SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
The plan for query having given shape would remain same irrespective of value
of property restrictions
The shape can then be used for
* Stats Collection - Currently stats collection gets overflown if same query
with different value gets invoked
* Allow configuring hints - See support in Mongo [2] for an example. One
specify via config that for a query of such and such shape this index should be
used
* Less noisy diagnostics - If a query gets invoked with bad plan the QE can log
the warning once instead of logging it for each query invocation involving
different values.
[1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
[2] https://docs.mongodb.com/manual/reference/command/planCacheSetFilter/
was:
For certain requirements it would be good to have a notion/support to deduce
query shape [1]
{quote}
A combination of query predicate, sort, and projection specifications.
For the query predicate, only the structure of the predicate, including the
field names, are significant; the values in the query predicate are
insignificant. As such, a query predicate { type: 'food' } is equivalent to the
query predicate { type: 'utensil' } for a query shape.
{quote}
So transforming that to Oak the shape should represent a JCR-SQL2 query string
(xpath query gets transformed to SQL2) which is a *canonical* representation of
actual query ignoring the property restriction values.
Example we have 2 queries
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'published'
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'disabled'
The query shape would be
SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
The plan for query having given shape would remain same irrespective of value
of property restrictions
The shape can then be used for
* Stats Collection - Currently stats collection gets overflown if same query
with different value gets invoked
* Allow configuring hints - See support in Mongo [2] for an example. One
specify via config that for a query of such and such shape this index should be
used
* Less noisy diagnostics - If a query gets invoked with bad plan the QE can log
the warning once instead of logging it for each query invocation involving
different values.
[1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
[2] https://docs.mongodb.com/manual/reference/command/planCacheSetFilter/
> Query shapes for JCR Query
> --
>
> Key: OAK-4934
> URL: https://issues.apache.org/jira/browse/OAK-4934
> Project: Jackrabbit Oak
> Issue Type: Wish
> Components: query
>Reporter: Chetan Mehrotra
>
> For certain requirements it would be good to have a notion/support to deduce
> query shape [1]
> {quote}
> A combination of query predicate, sort, and projection specifications.
> For the query predicate, only the structure of the predicate, including the
> field names, are significant; the values in the query predicate are
> insignificant. As such, a query predicate \{ type: 'food' \} is equivalent to
> the query predicate \{ type: 'utensil' \} for a query shape.
> {quote}
> So transforming that to Oak the shape should represent a JCR-SQL2 query
> string (xpath query gets transformed to SQL2) which is a *canonical*
> representation of actual query ignoring the property restriction values.
> Example we have 2 queries
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'published'
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'disabled'
> The query shape would be
> SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
[jira] [Updated] (OAK-4934) Query shapes for JCR Query
[
https://issues.apache.org/jira/browse/OAK-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-4934:
-
Description:
For certain requirements it would be good to have a notion/support to deduce
query shape [1]
{quote}
A combination of query predicate, sort, and projection specifications.
For the query predicate, only the structure of the predicate, including the
field names, are significant; the values in the query predicate are
insignificant. As such, a query predicate { type: 'food' } is equivalent to the
query predicate { type: 'utensil' } for a query shape.
{quote}
So transforming that to Oak the shape should represent a JCR-SQL2 query string
(xpath query gets transformed to SQL2) which is a *canonical* representation of
actual query ignoring the property restriction values.
Example we have 2 queries
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'published'
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'disabled'
The query shape would be
SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
The plan for query having given shape would remain same irrespective of value
of property restrictions
The shape can then be used for
* Stats Collection - Currently stats collection gets overflown if same query
with different value gets invoked
* Allow configuring hints - See support in Mongo [2] for an example. One
specify via config that for a query of such and such shape this index should be
used
* Less noisy diagnostics - If a query gets invoked with bad plan the QE can log
the warning once instead of logging it for each query invocation involving
different values.
[1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
[2] https://docs.mongodb.com/manual/reference/command/planCacheSetFilter/
was:
For certain requirements it would be good to have a notion/support to deduce
query shape [1]
{quote}
A combination of query predicate, sort, and projection specifications.
For the query predicate, only the structure of the predicate, including the
field names, are significant; the values in the query predicate are
insignificant. As such, a query predicate { type: 'food' } is equivalent to the
query predicate { type: 'utensil' } for a query shape.
{quote}
So transforming that to Oak the shape should represent a JCR-SQL2 query string
which is a canonical representation of actual query ignoring the property
restriction values.
Example we have 2 queries
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'published'
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'disabled'
The query shape would be
SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
The plan for query having given shape would remain same irrespective of value
of property restrictions
The shape can then be used for
* Stats Collection - Currently stats collection gets overflown if same query
with different value gets invoked
* Allow configuring hints - See support in Mongo [2] for an example. One
specify via config that for a query of such and such shape this index should be
used
* Less noisy diagnostics - If a query gets invoked with bad plan the QE can log
the warning once instead of logging it for each query invocation involving
different values.
[1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
[2] https://docs.mongodb.com/manual/reference/command/planCacheSetFilter/
> Query shapes for JCR Query
> --
>
> Key: OAK-4934
> URL: https://issues.apache.org/jira/browse/OAK-4934
> Project: Jackrabbit Oak
> Issue Type: Wish
> Components: query
>Reporter: Chetan Mehrotra
>
> For certain requirements it would be good to have a notion/support to deduce
> query shape [1]
> {quote}
> A combination of query predicate, sort, and projection specifications.
> For the query predicate, only the structure of the predicate, including the
> field names, are significant; the values in the query predicate are
> insignificant. As such, a query predicate { type: 'food' } is equivalent to
> the query predicate { type: 'utensil' } for a query shape.
> {quote}
> So transforming that to Oak the shape should represent a JCR-SQL2 query
> string (xpath query gets transformed to SQL2) which is a *canonical*
> representation of actual query ignoring the property restriction values.
> Example we have 2 queries
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'published'
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'disabled'
> The query shape would be
> SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
> The plan for query having given shape
[jira] [Commented] (OAK-4934) Query shapes for JCR Query
[
https://issues.apache.org/jira/browse/OAK-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571668#comment-15571668
]
Chetan Mehrotra commented on OAK-4934:
--
[~tmueller] Thoughts?
> Query shapes for JCR Query
> --
>
> Key: OAK-4934
> URL: https://issues.apache.org/jira/browse/OAK-4934
> Project: Jackrabbit Oak
> Issue Type: Wish
> Components: query
>Reporter: Chetan Mehrotra
>
> For certain requirements it would be good to have a notion/support to deduce
> query shape [1]
> {quote}
> A combination of query predicate, sort, and projection specifications.
> For the query predicate, only the structure of the predicate, including the
> field names, are significant; the values in the query predicate are
> insignificant. As such, a query predicate { type: 'food' } is equivalent to
> the query predicate { type: 'utensil' } for a query shape.
> {quote}
> So transforming that to Oak the shape should represent a JCR-SQL2 query
> string which is a canonical representation of actual query ignoring the
> property restriction values.
> Example we have 2 queries
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'published'
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'disabled'
> The query shape would be
> SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
> The plan for query having given shape would remain same irrespective of value
> of property restrictions
> The shape can then be used for
> * Stats Collection - Currently stats collection gets overflown if same query
> with different value gets invoked
> * Allow configuring hints - See support in Mongo [2] for an example. One
> specify via config that for a query of such and such shape this index should
> be used
> * Less noisy diagnostics - If a query gets invoked with bad plan the QE can
> log the warning once instead of logging it for each query invocation
> involving different values.
> [1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
> [2] https://docs.mongodb.com/manual/reference/command/planCacheSetFilter/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4934) Query shapes for JCR Query
[
https://issues.apache.org/jira/browse/OAK-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-4934:
-
Description:
For certain requirements it would be good to have a notion/support to deduce
query shape [1]
{quote}
A combination of query predicate, sort, and projection specifications.
For the query predicate, only the structure of the predicate, including the
field names, are significant; the values in the query predicate are
insignificant. As such, a query predicate { type: 'food' } is equivalent to the
query predicate { type: 'utensil' } for a query shape.
{quote}
So transforming that to Oak the shape should represent a JCR-SQL2 query string
which is a canonical representation of actual query ignoring the property
restriction values.
Example we have 2 queries
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'published'
* SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
'disabled'
The query shape would be
SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
The plan for query having given shape would remain same irrespective of value
of property restrictions
The shape can then be used for
* Stats Collection - Currently stats collection gets overflown if same query
with different value gets invoked
* Allow configuring hints - See support in Mongo [2] for an example. One
specify via config that for a query of such and such shape this index should be
used
* Less noisy diagnostics - If a query gets invoked with bad plan the QE can log
the warning once instead of logging it for each query invocation involving
different values.
[1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
[2] https://docs.mongodb.com/manual/reference/command/planCacheSetFilter/
was:
For certain requirements it would be good to have a notion/support to deduce
query shape [1]
{quote}
[1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
> Query shapes for JCR Query
> --
>
> Key: OAK-4934
> URL: https://issues.apache.org/jira/browse/OAK-4934
> Project: Jackrabbit Oak
> Issue Type: Wish
> Components: query
>Reporter: Chetan Mehrotra
>
> For certain requirements it would be good to have a notion/support to deduce
> query shape [1]
> {quote}
> A combination of query predicate, sort, and projection specifications.
> For the query predicate, only the structure of the predicate, including the
> field names, are significant; the values in the query predicate are
> insignificant. As such, a query predicate { type: 'food' } is equivalent to
> the query predicate { type: 'utensil' } for a query shape.
> {quote}
> So transforming that to Oak the shape should represent a JCR-SQL2 query
> string which is a canonical representation of actual query ignoring the
> property restriction values.
> Example we have 2 queries
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'published'
> * SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] =
> 'disabled'
> The query shape would be
> SELECT * FROM [app:Asset] AS a WHERE a.[jcr:content/metadata/status] = 'A'.
> The plan for query having given shape would remain same irrespective of value
> of property restrictions
> The shape can then be used for
> * Stats Collection - Currently stats collection gets overflown if same query
> with different value gets invoked
> * Allow configuring hints - See support in Mongo [2] for an example. One
> specify via config that for a query of such and such shape this index should
> be used
> * Less noisy diagnostics - If a query gets invoked with bad plan the QE can
> log the warning once instead of logging it for each query invocation
> involving different values.
> [1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
> [2] https://docs.mongodb.com/manual/reference/command/planCacheSetFilter/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-4934) Query shapes for JCR Query
Chetan Mehrotra created OAK-4934:
Summary: Query shapes for JCR Query
Key: OAK-4934
URL: https://issues.apache.org/jira/browse/OAK-4934
Project: Jackrabbit Oak
Issue Type: Wish
Components: query
Reporter: Chetan Mehrotra
For certain requirements it would be good to have a notion/support to deduce
query shape [1]
{quote}
[1] https://docs.mongodb.com/manual/reference/glossary/#term-query-shape
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4586) Collect affected node types on commit
[ https://issues.apache.org/jira/browse/OAK-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Egli updated OAK-4586: - Fix Version/s: 1.5.13 > Collect affected node types on commit > - > > Key: OAK-4586 > URL: https://issues.apache.org/jira/browse/OAK-4586 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: jcr >Reporter: Marcel Reutegger > Labels: observation > Fix For: 1.5.13 > > > One of the major optimizations for generating events in oak-jcr is to avoid > traversal into sub trees when there are path filters on an observation > listener. Another optimization could be done based on the node type filters > provided by a listener. If oak-jcr knows the affected node types of a cache > set it could shortcut event processing in case the listeners node type filter > is disjoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-4907) Collect changes (paths, nts, props..) of a commit in a validator
[ https://issues.apache.org/jira/browse/OAK-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Egli resolved OAK-4907. -- Resolution: Fixed Fix Version/s: (was: 1.6) 1.5.13 Committed in http://svn.apache.org/viewvc?rev=1764657=rev /cc [~chetanm], applied your suggestions, thx > Collect changes (paths, nts, props..) of a commit in a validator > > > Key: OAK-4907 > URL: https://issues.apache.org/jira/browse/OAK-4907 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: core >Affects Versions: 1.5.11 >Reporter: Stefan Egli >Assignee: Stefan Egli > Fix For: 1.5.13 > > Attachments: OAK-4907.patch, OAK-4907.v2.patch > > > It would be useful to collect a set of changes of a commit (eg in a > validator) that could later be used in an Observer for eg prefiltering. > Such a change collector should collect paths, nodetypes, properties, > node-names (and perhaps more at a later stage) of all changes and store the > result in the CommitInfo's CommitContext. > Note that this is a result of > [discussions|https://issues.apache.org/jira/browse/OAK-4796?focusedCommentId=15550962=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15550962] > around design in OAK-4796 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4817) QueryEngineSettings without MBean
[
https://issues.apache.org/jira/browse/OAK-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571368#comment-15571368
]
Chetan Mehrotra commented on OAK-4817:
--
Can you remove the wrap method from {{QueryEngineSettings}} as it adds
reference to {{QueryEngineSettingsMBeanImpl}} which has dependency on JMX api
and this is not allowed in Google App Engine env [1] and due to this I have to
override this in
[oakutils|https://github.com/chetanmeh/oakutils/blob/master/src/main/java/org/apache/jackrabbit/oak/query/QueryEngineSettings.java]
{noformat}
*/
+public QueryEngineSettingsMBeanImpl wrap() {
+return new QueryEngineSettingsMBeanImpl(this);
+}
{noformat}
[1] https://cloud.google.com/appengine/docs/java/jrewhitelist?csw=1
> QueryEngineSettings without MBean
> -
>
> Key: OAK-4817
> URL: https://issues.apache.org/jira/browse/OAK-4817
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: query
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Fix For: 1.6
>
> Attachments: OAK-4817.patch
>
>
> QueryEngineSettings is currently an MBean, and constructing a new instance is
> expensive. This is seem in some oak-core security components. The MBean
> functionality should be decoupled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-4586) Collect affected node types on commit
[ https://issues.apache.org/jira/browse/OAK-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Egli resolved OAK-4586. -- Resolution: Duplicate Duplicate of OAK-4907 (which is newer but already has a patch + review, so closing this one instead) > Collect affected node types on commit > - > > Key: OAK-4586 > URL: https://issues.apache.org/jira/browse/OAK-4586 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: jcr >Reporter: Marcel Reutegger > Labels: observation > > One of the major optimizations for generating events in oak-jcr is to avoid > traversal into sub trees when there are path filters on an observation > listener. Another optimization could be done based on the node type filters > provided by a listener. If oak-jcr knows the affected node types of a cache > set it could shortcut event processing in case the listeners node type filter > is disjoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4522) Improve CommitRateLimiter to optionally block some commits
[ https://issues.apache.org/jira/browse/OAK-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571155#comment-15571155 ] Thomas Mueller commented on OAK-4522: - http://svn.apache.org/r1764611 (fix test case) Actually the test case was "broken" before the above changes: it was relying on thread timing. Now the test uses an implementation detail of the CommitRateLimiter, which is also slightly bad, but I don't see an alternative. Also, the test is now faster. > Improve CommitRateLimiter to optionally block some commits > -- > > Key: OAK-4522 > URL: https://issues.apache.org/jira/browse/OAK-4522 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: jcr >Reporter: Thomas Mueller >Assignee: Thomas Mueller > Labels: observation > Fix For: 1.5.12 > > Attachments: OAK-4522-b.patch, OAK-4522-c.patch, OAK-4522.patch > > > The CommitRateLimiter of OAK-1659 can delay commits, but doesn't currently > block them, and delays even those commits that are part of handling events. > Because of that, the queue can still get full, and possibly delaying commits > while handling events can make the situation even worse. > In Jackrabbit 2.x, we had a similar feature: JCR-2402. Also related is > JCR-2746. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3696) Improve SegmentMK resilience
[ https://issues.apache.org/jira/browse/OAK-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3696: --- Fix Version/s: Segment Tar 0.0.16 > Improve SegmentMK resilience > > > Key: OAK-3696 > URL: https://issues.apache.org/jira/browse/OAK-3696 > Project: Jackrabbit Oak > Issue Type: Epic > Components: segment-tar >Reporter: Michael Marth >Assignee: Michael Dürig > Labels: resilience > Fix For: Segment Tar 0.0.16 > > > Epic for collection SegmentMK resilience improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-3696) Improve SegmentMK resilience
[ https://issues.apache.org/jira/browse/OAK-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig resolved OAK-3696. Resolution: Fixed > Improve SegmentMK resilience > > > Key: OAK-3696 > URL: https://issues.apache.org/jira/browse/OAK-3696 > Project: Jackrabbit Oak > Issue Type: Epic > Components: segment-tar >Reporter: Michael Marth >Assignee: Michael Dürig > Labels: resilience > Fix For: Segment Tar 0.0.16 > > > Epic for collection SegmentMK resilience improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
