[jira] [Commented] (OAK-8629) Node bundling exposes hidden properties
[
https://issues.apache.org/jira/browse/OAK-8629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933105#comment-16933105
]
Chetan Mehrotra commented on OAK-8629:
--
Would have a closer look. Note that currently nodeType change with bundling is
not properly handled. There is a TODO left in
[CommitDiff|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/CommitDiff.java#L100]
{noformat}
@Override
public boolean childNodeChanged(String name,
NodeState before,
NodeState after) {
//TODO [bundling] Handle change of primaryType. Current approach would
work
//but if bundling was enabled for previous nodetype its "side effect"
//would still impact even though new nodetype does not have bundling
enabled
BundlingHandler child = bundlingHandler.childChanged(name, before,
after);
return after.compareAgainstBaseState(before,
new CommitDiff(store, commit, child, builder, blobs));
}
{noformat}
Problem here is that nodeType change would not result in deletion and
recreation. Hence Bundling logic would still need to be activated as underlying
structure is still bundled.
> Node bundling exposes hidden properties
> ---
>
> Key: OAK-8629
> URL: https://issues.apache.org/jira/browse/OAK-8629
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk
>Affects Versions: 1.16.0
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.18.0
>
> Attachments: OAK-8629.patch
>
>
> The DocumentNodeStore node bundling feature may expose a hidden internal
> property when a bundled node structure is deleted and re-created with a
> non-bundling nodetype.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OAK-6619) Async indexer thread may get stuck in CopyOnWriteDirectory close method
[
https://issues.apache.org/jira/browse/OAK-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-6619:
-
Fix Version/s: (was: 1.16.0)
> Async indexer thread may get stuck in CopyOnWriteDirectory close method
> ---
>
> Key: OAK-6619
> URL: https://issues.apache.org/jira/browse/OAK-6619
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Critical
> Attachments: status-threaddump-Sep-5.txt
>
>
> With copy-on-write mode enabled at times its seen that async index thread
> remain stuck in CopyOnWriteDirectory#close method
> {noformat}
> "async-index-update-async" prio=5 tid=0xb9e63 nid=0x timed_waiting
>java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> - waiting to lock <0x2504cd51> (a
> java.util.concurrent.CountDownLatch$Sync) owned by "null" tid=0x-1
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory.close(CopyOnWriteDirectory.java:221)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.updateSuggester(DefaultIndexWriter.java:177)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.close(DefaultIndexWriter.java:121)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext.closeWriter(LuceneIndexEditorContext.java:136)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.leave(LuceneIndexEditor.java:154)
> at
> org.apache.jackrabbit.oak.plugins.index.IndexUpdate.leave(IndexUpdate.java:357)
> at
> org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:60)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:56)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:727)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:572)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:431)
> - locked <0x3d542de5> (a
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate)
> at
> org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:245)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The thread is waiting on a latch and no other thread is going to release the
> latch.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (OAK-6619) Async indexer thread may get stuck in CopyOnWriteDirectory close method
[
https://issues.apache.org/jira/browse/OAK-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra reassigned OAK-6619:
Assignee: (was: Chetan Mehrotra)
> Async indexer thread may get stuck in CopyOnWriteDirectory close method
> ---
>
> Key: OAK-6619
> URL: https://issues.apache.org/jira/browse/OAK-6619
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Priority: Critical
> Attachments: status-threaddump-Sep-5.txt
>
>
> With copy-on-write mode enabled at times its seen that async index thread
> remain stuck in CopyOnWriteDirectory#close method
> {noformat}
> "async-index-update-async" prio=5 tid=0xb9e63 nid=0x timed_waiting
>java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> - waiting to lock <0x2504cd51> (a
> java.util.concurrent.CountDownLatch$Sync) owned by "null" tid=0x-1
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory.close(CopyOnWriteDirectory.java:221)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.updateSuggester(DefaultIndexWriter.java:177)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.close(DefaultIndexWriter.java:121)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext.closeWriter(LuceneIndexEditorContext.java:136)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.leave(LuceneIndexEditor.java:154)
> at
> org.apache.jackrabbit.oak.plugins.index.IndexUpdate.leave(IndexUpdate.java:357)
> at
> org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:60)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:56)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:727)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:572)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:431)
> - locked <0x3d542de5> (a
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate)
> at
> org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:245)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The thread is waiting on a latch and no other thread is going to release the
> latch.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OAK-7947) Lazy loading of Lucene index files startup
[ https://issues.apache.org/jira/browse/OAK-7947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712595#comment-16712595 ] Chetan Mehrotra commented on OAK-7947: -- [~tmueller] One reason for doing eager loading was to avoid contention in queries hitting at very start. To make is lazy what we can do is store the data points required for index planning in index data node itself in repository. So stuff like numDocs and field count etc can recorded in repo upon index close. Then at least for index planning phase we need not open the IndexWriter at all > Lazy loading of Lucene index files startup > -- > > Key: OAK-7947 > URL: https://issues.apache.org/jira/browse/OAK-7947 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene, query >Reporter: Thomas Mueller >Assignee: Thomas Mueller >Priority: Major > Attachments: OAK-7947.patch > > > Right now, all Lucene index binaries are loaded on startup (I think when the > first query is run, to do cost calculation). This is a performance problem if > the index files are large, and need to be downloaded from the data store. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7832) oak-run console export should handle exceptions such as missing segments
[
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654656#comment-16654656
]
Chetan Mehrotra commented on OAK-7832:
--
Given there are changes in {{JsonSerializer}} which is used in
DocumentNodeStore it would be better to have another review before pushing it
to 1.8
[~mreutegg] [~catholicon] Would you be able to review this change?
> oak-run console export should handle exceptions such as missing segments
>
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: run, store-spi
>Reporter: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run
> console currently will choke and abort the traversal on exceptions such as
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed
> in the exported json, and just continue. This also ensures all missing
> segments for a subtree are listed with one command (if it fails right now you
> just see the first missing segment).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7832) oak-run console export should handle exceptions such as missing segments
[
https://issues.apache.org/jira/browse/OAK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653318#comment-16653318
]
Chetan Mehrotra commented on OAK-7832:
--
Had a quick look and it looks fine (barring use of non Apache license header!).
I would not be able to commit it i till next week. So if someone else can merge
it then it would be better
{quote}
A related improvement for the export would be to create one json file per
page/asset/folder /jcr:content etc., especially if one is exporting larger
trees. A single nodestate.json is hard to "browse" as a human. That's probably
something for another issue I guess.
{quote}
Yup. Aim of this tool was more to aid in debugging and analyzing bits of repo
specially when higher level tools like Content Package cannot be used. For
structural export it may be better to have a cli mode for using Content Package
feature set.
> oak-run console export should handle exceptions such as missing segments
>
>
> Key: OAK-7832
> URL: https://issues.apache.org/jira/browse/OAK-7832
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: run, store-spi
>Reporter: Alexander Klimetschek
>Priority: Major
> Attachments: OAK-7832.patch, oak-run-1.8.9-SNAPSHOT.jar
>
>
> Problem: When trying to look at data using "pn" or running "export" oak-run
> console currently will choke and abort the traversal on exceptions such as
> SegmentNotFoundException.
> Expected: It would be nice if it would be best-effort, log the issue or embed
> in the exported json, and just continue. This also ensures all missing
> segments for a subtree are listed with one command (if it fails right now you
> just see the first missing segment).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7679) oak-core: update Mockito version and fix test case to be compatible with it
[ https://issues.apache.org/jira/browse/OAK-7679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570287#comment-16570287 ] Chetan Mehrotra commented on OAK-7679: -- +1 > oak-core: update Mockito version and fix test case to be compatible with it > --- > > Key: OAK-7679 > URL: https://issues.apache.org/jira/browse/OAK-7679 > Project: Jackrabbit Oak > Issue Type: Task > Components: core >Reporter: Julian Reschke >Priority: Minor > Attachments: OAK-7679.diff > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7495) async,sync index not synchronous
[
https://issues.apache.org/jira/browse/OAK-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541683#comment-16541683
]
Chetan Mehrotra commented on OAK-7495:
--
Would need to revisit the flow to confirm the hypothesis.
{quote}
All this said, I don't feel comfortable with trying to maintain quite a complex
sync implementation where hybridV2 does a repository level sync view for
specific properties (so force sync for only some properties and not all
properties being handled by the index). I'd rather prefer we deprecate
"async='sync' " type (of course, that's my personal view)
{quote}
+1. The async='sync' would not work properly in cluster setups and that was the
reason for introducing HybridV2. So investing further here does not make sense.
> async,sync index not synchronous
>
>
> Key: OAK-7495
> URL: https://issues.apache.org/jira/browse/OAK-7495
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: indexing
>Affects Versions: 1.6.1
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
>Priority: Major
> Attachments: GetJobVerifier.java, OAK-7495-add-logs.patch,
> OAK-7495-potential-fix.patch, OAK-7495-test.patch, OAK-7495.demo.patch,
> slingeventJob.-1.tidy.json, unit-tests.log
>
>
> On an oak 1.6.1 (AEM 6.3) a suspicious behaviour was detected, where in Sling
> an
> [addJob|https://github.com/apache/sling-old-svn-mirror/blob/org.apache.sling.event-4.2.0/src/main/java/org/apache/sling/event/impl/jobs/JobManagerImpl.java#L286]
> followed by a
> [getJobById|https://github.com/apache/sling-old-svn-mirror/blob/org.apache.sling.event-4.2.0/src/main/java/org/apache/sling/event/impl/jobs/JobManagerImpl.java#L294]
> (in a different thread though, but perhaps would also fail in same thread)
> was not seeing the job that was just created.
> To give a bit more background, in Sling getJobById results in a query. That
> query uses an index which is built using {{"async, sync"}}. So the assumption
> is that the index is actually synchronous. But a test reproducing initially
> mentioned scenario showed the opposite.
> Attached:
> * [^GetJobVerifier.java] a Sling job test case that has 2 threads: a thread
> that does addJob, adds the resulting jobId to a list (synchronized). and a
> second thread that reads the jobId off that list and does a getJobById. That
> getJobById should find the job, as it was just created (how else could you
> figure out the jobId) - but sometimes it FAILs (see system out FAIL)
> * [^slingeventJob.-1.tidy.json] the index definition showing it is indeed
> "async, sync"
> PS: Example query that is executed:
> {{/jcr:root/var/eventing/jobs//element(*,slingevent:Job)[@slingevent:eventId
> = '2018/5/11/2/12/bca505d9-3044-4de9-9732-056ab1b6c513_5569']}}
> /cc [~catholicon]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7629) Lucene index: hide the settings COR and COW
[
https://issues.apache.org/jira/browse/OAK-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541669#comment-16541669
]
Chetan Mehrotra commented on OAK-7629:
--
+1. {{propertyPrivate}} should make it hidden
> Lucene index: hide the settings COR and COW
>
>
> Key: OAK-7629
> URL: https://issues.apache.org/jira/browse/OAK-7629
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
>Priority: Major
> Fix For: 1.10
>
>
> I have seen people disabling copy-on-read or copy-on-write hoping to improve
> performance or save space. We know about problems if those are disabled, for
> example it can result in many open file handles, OAK-7460.
> So best if we hide these features, so that people don't change them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7410) Define SPIs for Oak Search module
[ https://issues.apache.org/jira/browse/OAK-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504551#comment-16504551 ] Chetan Mehrotra commented on OAK-7410: -- [~tmueller] +1 for the changes. > Define SPIs for Oak Search module > - > > Key: OAK-7410 > URL: https://issues.apache.org/jira/browse/OAK-7410 > Project: Jackrabbit Oak > Issue Type: Technical task >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Major > Fix For: 1.9.3 > > Attachments: AggregateGenerics.patch > > > Abstract IndexEditor and QueryIndex implementations and SPIs should be > created for Oak Search module. > Those classes should be then reused / extended by implementors in and outside > Oak . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7490) oak-run console lc rmdata command uses second parameter for index path (and defaults to /oak:index/lucene)
[
https://issues.apache.org/jira/browse/OAK-7490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473878#comment-16473878
]
Chetan Mehrotra commented on OAK-7490:
--
bq. Destructive functionality, imo, shouldn't have default value for things to
be deleted
+1. Thanks [~catholicon] for fixing this!
> oak-run console lc rmdata command uses second parameter for index path (and
> defaults to /oak:index/lucene)
> --
>
> Key: OAK-7490
> URL: https://issues.apache.org/jira/browse/OAK-7490
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: oak-run
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Major
> Fix For: 1.10, 1.8.3, 1.2.30, 1.4.22, 1.6.12
>
>
> {{rmdata}} command currently takes 2 arguments where 1st one isn't even used
> - the second one is represents the index to be deleted.
> Worse yet, the command assumes a default index to be deleted to be
> {{/oak:index/lucene}} - that can lead to unintentional deletion.
> Destructive functionality, imo, shouldn't have default value for things to be
> deleted. Also, of course, we should fix that only 1 parameter is expected and
> parsed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-4638) Mostly async unique index (for UUIDs for example)
[ https://issues.apache.org/jira/browse/OAK-4638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460576#comment-16460576 ] Chetan Mehrotra commented on OAK-4638: -- [~tmueller] [~catholicon] Should we resolve this issue as OAK-6535 implemented the approach discussed here. Whats left is work on converting existing indexes to switch to Lucene based indexes which can be done in a separate issue dedicated for the conversion. Currently this issue being open conveys that indexing support needs to be implemented which is misleading > Mostly async unique index (for UUIDs for example) > - > > Key: OAK-4638 > URL: https://issues.apache.org/jira/browse/OAK-4638 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: property-index, query >Reporter: Thomas Mueller >Priority: Major > > The UUID index takes a lot of space. For the UUID index, we should consider > using mainly an async index. This is possible because there are two types of > UUIDs: those generated in Oak, which are sure to be unique (no need to > check), and those set in the application code, for example by importing > packages. For older nodes, an async index is sufficient, and a synchronous > index is only (temporarily) needed for imported nodes. For UUIDs, we could > also change the generation algorithm if needed. > It might be possible to use a similar pattern for regular unique indexes as > well: only keep the added entries of the last 24 hours (for example) in a > property index, and then move entries to an async index which needs less > space. That would slow down adding entries, as two indexes need to be checked. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6087) Avoid reads from MongoDB primary
[ https://issues.apache.org/jira/browse/OAK-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449258#comment-16449258 ] Chetan Mehrotra commented on OAK-6087: -- +1 to use client sessions and remove time based custom heuristic to route the calls to secondary. > Avoid reads from MongoDB primary > > > Key: OAK-6087 > URL: https://issues.apache.org/jira/browse/OAK-6087 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Major > Labels: scalability > > With OAK-2106 Oak now attempts to read from a MongoDB secondary when it > detects the requested data is available on the secondary. > When multiple Oak cluster nodes are deployed on a MongoDB replica set, many > reads are still directed to the primary. One of the reasons why this is seen > in practice, are observers and JCR event listeners that are triggered rather > soon after a change happens and therefore read recently modified documents. > This makes it difficult for Oak to direct calls to a nearby secondary, > because changes may not yet be available there. > A rather simple solution for the observers may be to delay processing of > changes until they are available on the near secondary. > A more sophisticated solution discussed offline could hide the replica set > entirely and always read from the nearest secondary. Writes would obviously > still go to the primary, but only return when the write is available also on > the nearest secondary. This guarantees that any subsequent read is able to > see the preceding write. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7406) relax guava version range in Import-Package declarations
[ https://issues.apache.org/jira/browse/OAK-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434978#comment-16434978 ] Chetan Mehrotra commented on OAK-7406: -- Looks fine. Probably it would need to be applied on all subprojects where Guava library is being used. Or we can add it to oak-parent (in [bundle plugin config in pluginManagement|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-parent/pom.xml#L165]) itself given its required at minimum for oak-core > relax guava version range in Import-Package declarations > > > Key: OAK-7406 > URL: https://issues.apache.org/jira/browse/OAK-7406 > Project: Jackrabbit Oak > Issue Type: Technical task >Reporter: Julian Reschke >Priority: Major > Attachments: OAK-7406.diff > > > We should relax the Guava version to the range that we test with, that is > 15.0 to 21 excl. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7304) Deploy oak-pojosr as part of standard deployment
[
https://issues.apache.org/jira/browse/OAK-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7304:
-
Labels: (was: candidate_oak_1_8)
Fix Version/s: 1.8.3
Merged to 1.8 with 1827468
> Deploy oak-pojosr as part of standard deployment
>
>
> Key: OAK-7304
> URL: https://issues.apache.org/jira/browse/OAK-7304
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: pojosr
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.9.0, 1.10, 1.8.3
>
>
> oak-pojosr currently has {{skip.deployment}} set to true and hence is being
> skipped from deployment. This setting should be removed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-7304) Deploy oak-pojosr as part of standard deployment
[
https://issues.apache.org/jira/browse/OAK-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7304:
-
Fix Version/s: 1.9.0
> Deploy oak-pojosr as part of standard deployment
>
>
> Key: OAK-7304
> URL: https://issues.apache.org/jira/browse/OAK-7304
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: pojosr
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Labels: candidate_oak_1_8
> Fix For: 1.9.0, 1.10
>
>
> oak-pojosr currently has {{skip.deployment}} set to true and hence is being
> skipped from deployment. This setting should be removed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-7266) Standalone example system console fails to render
[
https://issues.apache.org/jira/browse/OAK-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7266:
-
Labels: (was: candidate_oak_1_8)
Fix Version/s: 1.8.3
Merged to 1.8 with 1826523
> Standalone example system console fails to render
> -
>
> Key: OAK-7266
> URL: https://issues.apache.org/jira/browse/OAK-7266
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: examples
>Affects Versions: 1.8.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Major
> Fix For: 1.9.0, 1.10, 1.8.3
>
>
> Accessing http://localhost:8080/osgi/system/console/bundles with
> oak-standalone fails to render with following exception
> {noformat}
> 2018-02-14 10:36:40.415 WARN 23249 --- [qtp349420578-14]
> org.eclipse.jetty.server.HttpChannel : /osgi/system/console/bundles
> java.lang.NoSuchMethodError:
> org.json.JSONObject.valueToString(Ljava/lang/Object;)Ljava/lang/String;
> at org.json.JSONWriter.value(JSONWriter.java:316)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.writeJSON(BundlesServlet.java:612)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.writeJSON(BundlesServlet.java:558)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.renderContent(BundlesServlet.java:532)
> at
> org.apache.felix.webconsole.AbstractWebConsolePlugin.doGet(AbstractWebConsolePlugin.java:194)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.doGet(BundlesServlet.java:317)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-7265) Standalone example application fails to start
[
https://issues.apache.org/jira/browse/OAK-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7265:
-
Labels: (was: candidate_oak_1_8)
Fix Version/s: 1.8.3
Merged to 1.8 with 1826522
> Standalone example application fails to start
> -
>
> Key: OAK-7265
> URL: https://issues.apache.org/jira/browse/OAK-7265
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: examples, pojosr
>Affects Versions: 1.8.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Major
> Fix For: 1.9.0, 1.10, 1.8.3
>
>
> On users dl it was reported that standalone example application fails to
> start with following exception
> {noformat}
> java.lang.NoSuchMethodException:
> org.springframework.boot.loader.jar.JarEntry.getUrl()
> at java.lang.Class.getMethod(Class.java:1786)
> at
> org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getUrlMethod(SpringBootSupport.java:135)
> at
> org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getEntry(SpringBootSupport.java:109)
> at org.apache.felix.connect.PojoSRBundle.getEntry(PojoSRBundle.java:471)
> at
> org.apache.felix.webconsole.plugins.scriptconsole.internal.ScriptEngineManager.bundleChanged(ScriptEngineManager.java:117)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:821)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:771)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.run(EventDispatcher.java:993)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:52)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:94)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OAK-7304) Deploy oak-pojosr as part of standard deployment
[
https://issues.apache.org/jira/browse/OAK-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-7304.
--
Resolution: Fixed
Done with 1826516
> Deploy oak-pojosr as part of standard deployment
>
>
> Key: OAK-7304
> URL: https://issues.apache.org/jira/browse/OAK-7304
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: pojosr
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Labels: candidate_oak_1_8
> Fix For: 1.10
>
>
> oak-pojosr currently has {{skip.deployment}} set to true and hence is being
> skipped from deployment. This setting should be removed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-7304) Deploy oak-pojosr as part of standard deployment
[
https://issues.apache.org/jira/browse/OAK-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7304:
-
Labels: candidate_oak_1_8 (was: )
> Deploy oak-pojosr as part of standard deployment
>
>
> Key: OAK-7304
> URL: https://issues.apache.org/jira/browse/OAK-7304
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: pojosr
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Labels: candidate_oak_1_8
> Fix For: 1.10
>
>
> oak-pojosr currently has {{skip.deployment}} set to true and hence is being
> skipped from deployment. This setting should be removed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-4943) Keep Lucene Commits so that if the Segments file gets corrupted recovery can be attempted.
[ https://issues.apache.org/jira/browse/OAK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392544#comment-16392544 ] Chetan Mehrotra commented on OAK-4943: -- For now we can close it as none of the corruption issues analyzed so far would have benefited from this. > Keep Lucene Commits so that if the Segments file gets corrupted recovery can > be attempted. > -- > > Key: OAK-4943 > URL: https://issues.apache.org/jira/browse/OAK-4943 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.6.0 >Reporter: Ian Boston >Assignee: Chetan Mehrotra >Priority: Major > > Currently, Lucene deletes all previous generations of the segments files as > it has it uses the default IndexDeletionPolicy. Changing this to an > IndexDeletionPolicy that keeps a number of previous segments files will allow > an operator to find the most recent valid commit when the current segments > file reports corruption. The patch found at > https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:KeepLuceneCommits > keeps 10 previous commits. > A more sophisticated policy could be used to save commits non-linearly over > time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OAK-7304) Deploy oak-pojosr as part of standard deployment
Chetan Mehrotra created OAK-7304:
Summary: Deploy oak-pojosr as part of standard deployment
Key: OAK-7304
URL: https://issues.apache.org/jira/browse/OAK-7304
Project: Jackrabbit Oak
Issue Type: Task
Components: pojosr
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Fix For: 1.10
oak-pojosr currently has {{skip.deployment}} set to true and hence is being
skipped from deployment. This setting should be removed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7285) Reindexing using --doc-traversal-mode can OOM while aggregation in some cases
[
https://issues.apache.org/jira/browse/OAK-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378929#comment-16378929
]
Chetan Mehrotra commented on OAK-7285:
--
First thing we can do is make "preferredPathElements" a Set instead of iterable
which would simply the code later and make it more clear
{code}
if (pitr.hasNext() && isAncestor(path, pitr.peek().getPath())) {
NodeStateEntry nextEntry = pitr.next();
String nextEntryName =
PathUtils.getName(nextEntry.getPath());
if (preferred && !Iterables.contains(preferredPathElements,
nextEntryName)) {
return endOfData();
}
return nextEntry;
}
{code}
Here {{nextEntryName}} would child node name of immediate or descendent child.
It may break flow if we have 2 preferred path elements and first has a child
Say preferred path is "a", "b"
{noformat}
a
a/x
a/x/y
b
{noformat}
Here iterator would break early and would not return 'b'. So we should only
check for preferred condition only if the path is immediate child path.
> Reindexing using --doc-traversal-mode can OOM while aggregation in some cases
> -
>
> Key: OAK-7285
> URL: https://issues.apache.org/jira/browse/OAK-7285
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene, mongomk
>Affects Versions: 1.8.0
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Major
> Labels: candidate_oak_1_8
> Fix For: 1.9.0, 1.10
>
>
> {{--doc-traversal-mode}} works on the notion of {{preferred}} children which
> is computed using path fragments that form aggregate rules.
> The idea is reading through aggregated paths should avoid keeping non useful
> nodes (for path being currently indexed) in memory.
> But, currently, in case, say when there multiple preferred children -
> {{jcr:content}}, {{metadata}}, then an index defn indexing parent of a very
> deep tree root would try to read in the whole tree before concluding that it
> doesn't have preferred children
> e.g. with preferred list - {{jcr:content}} and {{metadata}} and index looking
> for {{jcr:content}} indexing following structure
> {noformat}
> + /path/being/indexed
>+ very
> + very
> + very
>+ deep
>+ tree
> + /some-sibling
> {noformat}
> Currently, while looking for {{jcr:content}}, the code concludes that it
> doesn't exist only after reaching {{/some-sibling}} (or if number of children
> read of {{/path/being/indexed}} is >= num_preferred_children).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-7266) Standalone example system console fails to render
[
https://issues.apache.org/jira/browse/OAK-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7266:
-
Labels: candidate_oak_1_8 (was: )
> Standalone example system console fails to render
> -
>
> Key: OAK-7266
> URL: https://issues.apache.org/jira/browse/OAK-7266
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: examples
>Affects Versions: 1.8.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Major
> Labels: candidate_oak_1_8
> Fix For: 1.9.0, 1.10
>
>
> Accessing http://localhost:8080/osgi/system/console/bundles with
> oak-standalone fails to render with following exception
> {noformat}
> 2018-02-14 10:36:40.415 WARN 23249 --- [qtp349420578-14]
> org.eclipse.jetty.server.HttpChannel : /osgi/system/console/bundles
> java.lang.NoSuchMethodError:
> org.json.JSONObject.valueToString(Ljava/lang/Object;)Ljava/lang/String;
> at org.json.JSONWriter.value(JSONWriter.java:316)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.writeJSON(BundlesServlet.java:612)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.writeJSON(BundlesServlet.java:558)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.renderContent(BundlesServlet.java:532)
> at
> org.apache.felix.webconsole.AbstractWebConsolePlugin.doGet(AbstractWebConsolePlugin.java:194)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.doGet(BundlesServlet.java:317)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OAK-7266) Standalone example system console fails to render
[
https://issues.apache.org/jira/browse/OAK-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-7266.
--
Resolution: Fixed
Done with 1824198
> Standalone example system console fails to render
> -
>
> Key: OAK-7266
> URL: https://issues.apache.org/jira/browse/OAK-7266
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: examples
>Affects Versions: 1.8.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Major
> Labels: candidate_oak_1_8
> Fix For: 1.9.0, 1.10
>
>
> Accessing http://localhost:8080/osgi/system/console/bundles with
> oak-standalone fails to render with following exception
> {noformat}
> 2018-02-14 10:36:40.415 WARN 23249 --- [qtp349420578-14]
> org.eclipse.jetty.server.HttpChannel : /osgi/system/console/bundles
> java.lang.NoSuchMethodError:
> org.json.JSONObject.valueToString(Ljava/lang/Object;)Ljava/lang/String;
> at org.json.JSONWriter.value(JSONWriter.java:316)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.writeJSON(BundlesServlet.java:612)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.writeJSON(BundlesServlet.java:558)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.renderContent(BundlesServlet.java:532)
> at
> org.apache.felix.webconsole.AbstractWebConsolePlugin.doGet(AbstractWebConsolePlugin.java:194)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.doGet(BundlesServlet.java:317)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7266) Standalone example system console fails to render
[
https://issues.apache.org/jira/browse/OAK-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363518#comment-16363518
]
Chetan Mehrotra commented on OAK-7266:
--
This happens due an old oak-json dependency being pulled by tika parsers
> Standalone example system console fails to render
> -
>
> Key: OAK-7266
> URL: https://issues.apache.org/jira/browse/OAK-7266
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: examples
>Affects Versions: 1.8.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Major
> Fix For: 1.9.0, 1.10
>
>
> Accessing http://localhost:8080/osgi/system/console/bundles with
> oak-standalone fails to render with following exception
> {noformat}
> 2018-02-14 10:36:40.415 WARN 23249 --- [qtp349420578-14]
> org.eclipse.jetty.server.HttpChannel : /osgi/system/console/bundles
> java.lang.NoSuchMethodError:
> org.json.JSONObject.valueToString(Ljava/lang/Object;)Ljava/lang/String;
> at org.json.JSONWriter.value(JSONWriter.java:316)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.writeJSON(BundlesServlet.java:612)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.writeJSON(BundlesServlet.java:558)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.renderContent(BundlesServlet.java:532)
> at
> org.apache.felix.webconsole.AbstractWebConsolePlugin.doGet(AbstractWebConsolePlugin.java:194)
> at
> org.apache.felix.webconsole.internal.core.BundlesServlet.doGet(BundlesServlet.java:317)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OAK-7266) Standalone example system console fails to render
Chetan Mehrotra created OAK-7266:
Summary: Standalone example system console fails to render
Key: OAK-7266
URL: https://issues.apache.org/jira/browse/OAK-7266
Project: Jackrabbit Oak
Issue Type: Bug
Components: examples
Affects Versions: 1.8.0
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Fix For: 1.9.0, 1.10
Accessing http://localhost:8080/osgi/system/console/bundles with oak-standalone
fails to render with following exception
{noformat}
2018-02-14 10:36:40.415 WARN 23249 --- [qtp349420578-14]
org.eclipse.jetty.server.HttpChannel : /osgi/system/console/bundles
java.lang.NoSuchMethodError:
org.json.JSONObject.valueToString(Ljava/lang/Object;)Ljava/lang/String;
at org.json.JSONWriter.value(JSONWriter.java:316)
at
org.apache.felix.webconsole.internal.core.BundlesServlet.writeJSON(BundlesServlet.java:612)
at
org.apache.felix.webconsole.internal.core.BundlesServlet.writeJSON(BundlesServlet.java:558)
at
org.apache.felix.webconsole.internal.core.BundlesServlet.renderContent(BundlesServlet.java:532)
at
org.apache.felix.webconsole.AbstractWebConsolePlugin.doGet(AbstractWebConsolePlugin.java:194)
at
org.apache.felix.webconsole.internal.core.BundlesServlet.doGet(BundlesServlet.java:317)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-7265) Standalone example application fails to start
[
https://issues.apache.org/jira/browse/OAK-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7265:
-
Labels: candidate_oak_1_8 (was: )
> Standalone example application fails to start
> -
>
> Key: OAK-7265
> URL: https://issues.apache.org/jira/browse/OAK-7265
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: examples, pojosr
>Affects Versions: 1.8.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Major
> Labels: candidate_oak_1_8
> Fix For: 1.9.0, 1.10
>
>
> On users dl it was reported that standalone example application fails to
> start with following exception
> {noformat}
> java.lang.NoSuchMethodException:
> org.springframework.boot.loader.jar.JarEntry.getUrl()
> at java.lang.Class.getMethod(Class.java:1786)
> at
> org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getUrlMethod(SpringBootSupport.java:135)
> at
> org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getEntry(SpringBootSupport.java:109)
> at org.apache.felix.connect.PojoSRBundle.getEntry(PojoSRBundle.java:471)
> at
> org.apache.felix.webconsole.plugins.scriptconsole.internal.ScriptEngineManager.bundleChanged(ScriptEngineManager.java:117)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:821)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:771)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.run(EventDispatcher.java:993)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:52)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:94)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OAK-7265) Standalone example application fails to start
[
https://issues.apache.org/jira/browse/OAK-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-7265.
--
Resolution: Fixed
Fixed with 1824196
> Standalone example application fails to start
> -
>
> Key: OAK-7265
> URL: https://issues.apache.org/jira/browse/OAK-7265
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: examples, pojosr
>Affects Versions: 1.8.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Major
> Fix For: 1.10
>
>
> On users dl it was reported that standalone example application fails to
> start with following exception
> {noformat}
> java.lang.NoSuchMethodException:
> org.springframework.boot.loader.jar.JarEntry.getUrl()
> at java.lang.Class.getMethod(Class.java:1786)
> at
> org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getUrlMethod(SpringBootSupport.java:135)
> at
> org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getEntry(SpringBootSupport.java:109)
> at org.apache.felix.connect.PojoSRBundle.getEntry(PojoSRBundle.java:471)
> at
> org.apache.felix.webconsole.plugins.scriptconsole.internal.ScriptEngineManager.bundleChanged(ScriptEngineManager.java:117)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:821)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:771)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.run(EventDispatcher.java:993)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:52)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:94)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-7265) Standalone example application fails to start
[
https://issues.apache.org/jira/browse/OAK-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7265:
-
Fix Version/s: 1.9.0
> Standalone example application fails to start
> -
>
> Key: OAK-7265
> URL: https://issues.apache.org/jira/browse/OAK-7265
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: examples, pojosr
>Affects Versions: 1.8.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Major
> Fix For: 1.9.0, 1.10
>
>
> On users dl it was reported that standalone example application fails to
> start with following exception
> {noformat}
> java.lang.NoSuchMethodException:
> org.springframework.boot.loader.jar.JarEntry.getUrl()
> at java.lang.Class.getMethod(Class.java:1786)
> at
> org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getUrlMethod(SpringBootSupport.java:135)
> at
> org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getEntry(SpringBootSupport.java:109)
> at org.apache.felix.connect.PojoSRBundle.getEntry(PojoSRBundle.java:471)
> at
> org.apache.felix.webconsole.plugins.scriptconsole.internal.ScriptEngineManager.bundleChanged(ScriptEngineManager.java:117)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:821)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:771)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.run(EventDispatcher.java:993)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:52)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:94)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OAK-7265) Standalone example application fails to start
Chetan Mehrotra created OAK-7265:
Summary: Standalone example application fails to start
Key: OAK-7265
URL: https://issues.apache.org/jira/browse/OAK-7265
Project: Jackrabbit Oak
Issue Type: Bug
Components: examples, pojosr
Affects Versions: 1.8.0
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Fix For: 1.10
On users dl it was reported that standalone example application fails to start
with following exception
{noformat}
java.lang.NoSuchMethodException:
org.springframework.boot.loader.jar.JarEntry.getUrl()
at java.lang.Class.getMethod(Class.java:1786)
at
org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getUrlMethod(SpringBootSupport.java:135)
at
org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getEntry(SpringBootSupport.java:109)
at org.apache.felix.connect.PojoSRBundle.getEntry(PojoSRBundle.java:471)
at
org.apache.felix.webconsole.plugins.scriptconsole.internal.ScriptEngineManager.bundleChanged(ScriptEngineManager.java:117)
at
org.apache.felix.connect.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:821)
at
org.apache.felix.connect.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:771)
at
org.apache.felix.connect.felix.framework.util.EventDispatcher.run(EventDispatcher.java:993)
at
org.apache.felix.connect.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:52)
at
org.apache.felix.connect.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:94)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7265) Standalone example application fails to start
[
https://issues.apache.org/jira/browse/OAK-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363495#comment-16363495
]
Chetan Mehrotra commented on OAK-7265:
--
This happens because the
[getUrl|https://github.com/spring-projects/spring-boot/blob/master/spring-boot-project/spring-boot-tools/spring-boot-loader/src/main/java/org/springframework/boot/loader/jar/JarEntry.java#L73]
method is made package scoped
> Standalone example application fails to start
> -
>
> Key: OAK-7265
> URL: https://issues.apache.org/jira/browse/OAK-7265
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: examples, pojosr
>Affects Versions: 1.8.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Major
> Fix For: 1.10
>
>
> On users dl it was reported that standalone example application fails to
> start with following exception
> {noformat}
> java.lang.NoSuchMethodException:
> org.springframework.boot.loader.jar.JarEntry.getUrl()
> at java.lang.Class.getMethod(Class.java:1786)
> at
> org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getUrlMethod(SpringBootSupport.java:135)
> at
> org.apache.jackrabbit.oak.run.osgi.SpringBootSupport$SpringBootJarRevision.getEntry(SpringBootSupport.java:109)
> at org.apache.felix.connect.PojoSRBundle.getEntry(PojoSRBundle.java:471)
> at
> org.apache.felix.webconsole.plugins.scriptconsole.internal.ScriptEngineManager.bundleChanged(ScriptEngineManager.java:117)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:821)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:771)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.run(EventDispatcher.java:993)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:52)
> at
> org.apache.felix.connect.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:94)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7264) Missing OSGi Provide-Capability for classed
[ https://issues.apache.org/jira/browse/OAK-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362508#comment-16362508 ] Chetan Mehrotra commented on OAK-7264: -- Some of these service are registered programatically via bundleContext upon component activation hence Bnd is not able to detect them. I have not been able to find a way where I can hint Bnd to include such capabilities. Similar problem was seen with SLING-6474 > Missing OSGi Provide-Capability for classed > --- > > Key: OAK-7264 > URL: https://issues.apache.org/jira/browse/OAK-7264 > Project: Jackrabbit Oak > Issue Type: Bug > Components: composite, core >Affects Versions: 1.8.2 >Reporter: Oleg Cohen >Priority: Minor > > Deploying Oak 1.8.2 into Karaf OSGi container v4.1.4 in addition to needing > oak-store-composite bundle I had to add a custom bundle with dummy > Provide-Capability for the following services: > Â Â Â Â Â Â Â Â Â Â Â Â Â Â > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > osgi.service;objectClass="com.codahale.metrics.MetricRegistry", > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > osgi.service;objectClass="org.apache.jackrabbit.oak.api.jmx.CacheStatsMBean", > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > osgi.service;objectClass="org.apache.jackrabbit.oak.api.jmx.QueryEngineSettingsMBean", > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > osgi.service;objectClass="org.apache.jackrabbit.oak.spi.state.NodeStore", > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > osgi.service;objectClass="org.apache.jackrabbit.oak.spi.state.NodeStoreProvider", > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > osgi.service;objectClass="org.apache.jackrabbit.oak.stats.StatisticsProvider", > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > osgi.service;objectClass="org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore", > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > osgi.service;objectClass="java.util.concurrent.Executor", > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > osgi.service;objectClass="org.apache.jackrabbit.oak.spi.blob.BlobStore", > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > osgi.service;objectClass="org.apache.jackrabbit.oak.spi.mount.MountInfoProvider" > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > Â > The suspicion is that the OSGi bundles providing these services don't have > the respective Provide-Capability generated in the MANIFEST. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OAK-7263) oak-lucene should not depend on oak-store-document
[
https://issues.apache.org/jira/browse/OAK-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362499#comment-16362499
]
Chetan Mehrotra edited comment on OAK-7263 at 2/13/18 3:34 PM:
---
This dependency is required untill we tackle OAK-6513. Probably we can work on
to make it optional
was (Author: chetanm):
This dependency is required untill we tackle OAK-6513
> oak-lucene should not depend on oak-store-document
> --
>
> Key: OAK-7263
> URL: https://issues.apache.org/jira/browse/OAK-7263
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Robert Munteanu
>Priority: Major
> Fix For: 1.9.0, 1.10
>
>
> {{oak-lucene}} has a hard dependency on {{oak-store-document}} and that looks
> wrong to me.
> {noformat}[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile
> (default-compile) on project oak-lucene: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneDocumentHolder.java:[31,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneDocumentHolder.java:[37,46]
> cannot find symbol
> [ERROR] symbol: class JournalProperty
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyBuilder.java:[33,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyBuilder.java:[34,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyBuilder.java:[38,47]
> cannot find symbol
> [ERROR] symbol: class JournalPropertyBuilder
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyBuilder.java:[106,12]
> cannot find symbol
> [ERROR] symbol: class JournalProperty
> [ERROR] location: class
> org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.LuceneJournalPropertyBuilder
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneIndexProviderService.java:[55,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/IndexedPaths.java:[29,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/IndexedPaths.java:[33,31]
> cannot find symbol
> [ERROR] symbol: class JournalProperty
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyService.java:[22,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyService.java:[23,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyService.java:[25,54]
> cannot find symbol
> [ERROR] symbol: class JournalPropertyService
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyService.java:[33,12]
> cannot find symbol
> [ERROR] symbol: class JournalPropertyBuilder
> [ERROR] location: class
> org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.LuceneJournalPropertyService
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyBuilder.java:[50,5]
> method does not override or implement a method from a supertype
> [ERROR]
>
[jira] [Commented] (OAK-7263) oak-lucene should not depend on oak-store-document
[
https://issues.apache.org/jira/browse/OAK-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362499#comment-16362499
]
Chetan Mehrotra commented on OAK-7263:
--
This dependency is required untill we tackle OAK-6513
> oak-lucene should not depend on oak-store-document
> --
>
> Key: OAK-7263
> URL: https://issues.apache.org/jira/browse/OAK-7263
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Robert Munteanu
>Priority: Major
> Fix For: 1.9.0, 1.10
>
>
> {{oak-lucene}} has a hard dependency on {{oak-store-document}} and that looks
> wrong to me.
> {noformat}[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile
> (default-compile) on project oak-lucene: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneDocumentHolder.java:[31,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneDocumentHolder.java:[37,46]
> cannot find symbol
> [ERROR] symbol: class JournalProperty
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyBuilder.java:[33,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyBuilder.java:[34,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyBuilder.java:[38,47]
> cannot find symbol
> [ERROR] symbol: class JournalPropertyBuilder
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyBuilder.java:[106,12]
> cannot find symbol
> [ERROR] symbol: class JournalProperty
> [ERROR] location: class
> org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.LuceneJournalPropertyBuilder
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneIndexProviderService.java:[55,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/IndexedPaths.java:[29,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/IndexedPaths.java:[33,31]
> cannot find symbol
> [ERROR] symbol: class JournalProperty
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyService.java:[22,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyService.java:[23,54]
> package org.apache.jackrabbit.oak.plugins.document.spi does not exist
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyService.java:[25,54]
> cannot find symbol
> [ERROR] symbol: class JournalPropertyService
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyService.java:[33,12]
> cannot find symbol
> [ERROR] symbol: class JournalPropertyBuilder
> [ERROR] location: class
> org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.LuceneJournalPropertyService
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyBuilder.java:[50,5]
> method does not override or implement a method from a supertype
> [ERROR]
> /home/robert/Documents/sources/apache/jackrabbit-oak/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/hybrid/LuceneJournalPropertyBuilder.java:[61,5]
> method does not override or implement
[jira] [Commented] (OAK-7256) Query: option to wait for indexes to be updated
[
https://issues.apache.org/jira/browse/OAK-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361788#comment-16361788
]
Chetan Mehrotra commented on OAK-7256:
--
[~edivad] The issue you are referring to is OAK-3606
> Query: option to wait for indexes to be updated
> ---
>
> Key: OAK-7256
> URL: https://issues.apache.org/jira/browse/OAK-7256
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: query
>Reporter: Thomas Mueller
>Priority: Minor
> Fix For: 1.10
>
>
> Sometimes (rarely, but still) queries should include the very latest changes,
> even if the index is updated asynchronously. For example when running unit
> test: data is added, and then a query is run to check if the data is there.
> The problem with asynchronous indexes is, you don't know exactly how long you
> have to wait. Often the index is updated quickly, and sometimes it takes a
> few seconds.
> What about extending the query syntax as follows:
> Wait for all indexes (no matter which index is used for this query) - this
> would be used rarely, just for testing:
> {noformat}
> /jcr:root/*
> option(wait for all indexes timeout 60)
> {noformat}
> Wait for just those indexes (well, usually it's just one, but sometimes it's
> multiple) that are needed for the given query. This query could also be used
> in an application that strictly needs the very latest results, even for
> fulltext queries. The "timeout" would mean "wait at most 10 seconds, and if
> not up-to-date then throw an exeption", while "max 10" would mean "wait at
> most 10 seconds, but still run the query in any case".
> {noformat}
> /jcr:root/content//*[jcr:contains(., 'hello')]
> option(wait for indexes max 10)
> {noformat}
> The query would wait, and once the indexes are up-to-date, return the
> requested result.
> So the syntax is (both SQL-2 and XPath):
> {noformat}
> option(wait for [all] indexes
> { max | timeout }
> [, ] )
> {noformat}
> So other options can also be used (option traversal fail,...).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7251) BinaryTextExtractor should not ignore parse exception - they should at least be logged at DEBUG in all cases
[
https://issues.apache.org/jira/browse/OAK-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356632#comment-16356632
]
Chetan Mehrotra commented on OAK-7251:
--
This convention was a carry over of [Jackrabbit
|https://github.com/apache/jackrabbit/blob/407bd5c7803e8518531559d3bf6fa480197e6341/jackrabbit-core/src/main/java/org/apache/jackrabbit/core/query/lucene/LazyTextExtractorField.java#L185]
way of doing things!.
+1 to change that now
> BinaryTextExtractor should not ignore parse exception - they should at least
> be logged at DEBUG in all cases
>
>
> Key: OAK-7251
> URL: https://issues.apache.org/jira/browse/OAK-7251
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Major
> Fix For: 1.10
>
>
> BinaryTextExtractor ignores missing library error like:
> {noformat}
> } catch (LinkageError e) {
> // Capture and ignore errors caused by extraction libraries
> // not being present. This is equivalent to disabling
> // selected media types in configuration, so we can simply
> // ignore these errors.
> {noformat}
> or
> {noformat}
> // Capture and report any other full text extraction problems.
> // The special STOP exception is used for normal termination.
> if (!handler.isWriteLimitReached(t)) {
> {noformat}
> We should at not skip these errors - some information should at least be
> available at DEBUG.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-5051) Document XPath (and SQL-2) syntax as supported by Oak
[ https://issues.apache.org/jira/browse/OAK-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349807#comment-16349807 ] Chetan Mehrotra commented on OAK-5051: -- bq. I've updated grammar-xpath.md [1] to utilize a doxia macro (part of the same branch) that renders railroads Interesting stuff! > Document XPath (and SQL-2) syntax as supported by Oak > - > > Key: OAK-5051 > URL: https://issues.apache.org/jira/browse/OAK-5051 > Project: Jackrabbit Oak > Issue Type: Documentation > Components: query >Reporter: Thomas Mueller >Assignee: Thomas Mueller >Priority: Major > > The JCR 2.0 SQL-2 language is documented at > http://h2database.com/jcr/grammar.html > But extensions to that are not documented. And XPath (as supported by Oak) is > not documented at all. > It would be good to have such a documentation, with examples. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OAK-7226) Warn messages ignoring exception parameter
[
https://issues.apache.org/jira/browse/OAK-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346825#comment-16346825
]
Chetan Mehrotra edited comment on OAK-7226 at 1/31/18 1:33 PM:
---
Logback also supports it [1]. Also this is stated in
[Logger|https://www.slf4j.org/api/org/slf4j/Logger.html] docs
bq. Note that logging statements can be parameterized in presence of an
exception/throwable.
[1]
https://github.com/qos-ch/logback/blob/master/logback-classic/src/main/java/ch/qos/logback/classic/spi/LoggingEvent.java#L114
was (Author: chetanm):
Logback also supports it [1]
[1]
https://github.com/qos-ch/logback/blob/master/logback-classic/src/main/java/ch/qos/logback/classic/spi/LoggingEvent.java#L114
> Warn messages ignoring exception parameter
> --
>
> Key: OAK-7226
> URL: https://issues.apache.org/jira/browse/OAK-7226
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob-plugins
>Affects Versions: 1.8.0
>Reporter: Marcel Reutegger
>Priority: Minor
> Fix For: 1.10
>
>
> There are a few usages of Logger.warn() with a pattern similar to this
> example:
> {noformat}
> log.warn("Error occurred while fetching DataRecord for identifier {}", input,
> exeption);
> {noformat}
> The intention probably is that the third parameter is treated as an exception
> and e.g. logged with the stack trace. However, this method signature
> interprets the exception as a second argument for the message format. This
> means the exception is effectively ignored.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7226) Warn messages ignoring exception parameter
[
https://issues.apache.org/jira/browse/OAK-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346825#comment-16346825
]
Chetan Mehrotra commented on OAK-7226:
--
Logback also supports it [1]
[1]
https://github.com/qos-ch/logback/blob/master/logback-classic/src/main/java/ch/qos/logback/classic/spi/LoggingEvent.java#L114
> Warn messages ignoring exception parameter
> --
>
> Key: OAK-7226
> URL: https://issues.apache.org/jira/browse/OAK-7226
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob-plugins
>Affects Versions: 1.8.0
>Reporter: Marcel Reutegger
>Priority: Minor
> Fix For: 1.10
>
>
> There are a few usages of Logger.warn() with a pattern similar to this
> example:
> {noformat}
> log.warn("Error occurred while fetching DataRecord for identifier {}", input,
> exeption);
> {noformat}
> The intention probably is that the third parameter is treated as an exception
> and e.g. logged with the stack trace. However, this method signature
> interprets the exception as a second argument for the message format. This
> means the exception is effectively ignored.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7226) Warn messages ignoring exception parameter
[
https://issues.apache.org/jira/browse/OAK-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346776#comment-16346776
]
Chetan Mehrotra commented on OAK-7226:
--
[~mreutegg] This is a "feature" of slf4j
https://www.slf4j.org/faq.html#paramException
> Warn messages ignoring exception parameter
> --
>
> Key: OAK-7226
> URL: https://issues.apache.org/jira/browse/OAK-7226
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob-plugins
>Affects Versions: 1.8.0
>Reporter: Marcel Reutegger
>Priority: Minor
> Fix For: 1.10
>
>
> There are a few usages of Logger.warn() with a pattern similar to this
> example:
> {noformat}
> log.warn("Error occurred while fetching DataRecord for identifier {}", input,
> exeption);
> {noformat}
> The intention probably is that the third parameter is treated as an exception
> and e.g. logged with the stack trace. However, this method signature
> interprets the exception as a second argument for the message format. This
> means the exception is effectively ignored.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7198) Index rule with REGEX_ALL_PROPS includes relative node
[
https://issues.apache.org/jira/browse/OAK-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346319#comment-16346319
]
Chetan Mehrotra commented on OAK-7198:
--
Thanks [~mreutegg] for digging into this. See "^[^\" nodes being looked up in
quite a few logs but did not got time to dig it deeper. Current support for
regex is limited to immediate property so this looks fine.
Later when we plan to support regex for relative property OAK-5187 we can have
a more generic approach
> Index rule with REGEX_ALL_PROPS includes relative node
> --
>
> Key: OAK-7198
> URL: https://issues.apache.org/jira/browse/OAK-7198
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.8.0
>Reporter: Marcel Reutegger
>Priority: Minor
> Fix For: 1.10
>
> Attachments: OAK-7198.patch
>
>
> A lucene index with an index rule that includes properties with
> {{LuceneIndexConstants.REGEX_ALL_PROPS}} also includes some child node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7213) Avoid call for child node when bundle contains all children
[
https://issues.apache.org/jira/browse/OAK-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346310#comment-16346310
]
Chetan Mehrotra commented on OAK-7213:
--
Changes look good. Some more info - for bundling root
{{bundlingContext.hasOnlyBundledChildren()}} relies on "_children" flag while
for any child node it relies on the {{:doc-has-child-non-bundled}}
> Avoid call for child node when bundle contains all children
> ---
>
> Key: OAK-7213
> URL: https://issues.apache.org/jira/browse/OAK-7213
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Labels: bundling
> Fix For: 1.9.0, 1.10
>
> Attachments: OAK-7213.patch
>
>
> When nodes are bundled in a document, the DocumentNodeStore keeps track of
> whether all children are included in a document. The presence of the hidden
> {{:doc-has-child-non-bundled}} property indicates there are non bundled child
> nodes. For the case when a document contains all children in the bundle, the
> DocumentNodeStore still does a find call on the DocumentStore when asked for
> an unknown child node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OAK-7065) Remove orphan file from local directory in case indexing fails
[
https://issues.apache.org/jira/browse/OAK-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346271#comment-16346271
]
Chetan Mehrotra edited comment on OAK-7065 at 1/31/18 5:24 AM:
---
IndexUpdate should support a close operation for IndexEditor which should be
invoked at the end of indexing cycle both in case of success and failure.
Currently the index editor like LuceneIndexEditor implement this close logic by
relying on {{leave}} call and checking if parent is null. However if there is
some exception in any other editor then the editors do not get a chance to
close.
So in case of failure LuceneIndexEditor should
# Close the IndexWriter
# -Also ensure that any new files which have been added by the IndexWriter get
removed-
Note that cleaning here may not cover all cases. For e.g. if the IndexUpdates
completes fine but AsyncIndexUpdate merge fails then also new files should be
considered orphan and be removed. This cleaning would need to be performed for
both index open for read and write phase
* Index open for read - Here it may happen that previous indexing failed (say
merge failed) and it did not got chance to cleanup stuff. So when new reader is
closed it may pull down files with same name which would cause conflict with
existing files. So it would need to clean local index of any such file first
before pulling down new files from remote directory
* Index open for write - Here it can also happen that previous indexing failed
and left some files. And further suppose this indexing happened on a different
cluster node i.e. different from the one where last indexing happened. So here
it would try to download files with same name and thus lead to corrupt index
scenario.
*Current Index File Garbage Collection Logic*
Currently the files are removed by COR directory upon close. It removes those
files which are present in local directory at time when COR is initialized but
not present in remote directory.
In doing so it also account for any new file opened by COW via shared working
set managed in IndexCopier
was (Author: chetanm):
IndexUpdate should support a close operation for IndexEditor which should be
invoked at the end of indexing cycle both in case of success and failure.
Currently the index editor like LuceneIndexEditor implement this close logic by
relying on {{leave}} call and checking if parent is null. However if there is
some exception in any other editor then the editors do not get a chance to
close.
So in case of failure LuceneIndexEditor should
# Close the IndexWriter
# Also ensure that any new files which have been added by the IndexWriter get
removed
> Remove orphan file from local directory in case indexing fails
> --
>
> Key: OAK-7065
> URL: https://issues.apache.org/jira/browse/OAK-7065
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
>Priority: Major
> Fix For: 1.10
>
>
> If an indexing cycle fails for some reason it may leave orphan files in local
> directory. Later on in next indexing cycle Lucene would try to create files
> with same name on local disk and this may fail on Windows where such files
> may have been memory mapped and hence cannot be deleted.
> We should analyze such a scenario and see if system can handle the failure
> case properly
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-7065) Remove orphan file from local directory in case indexing fails
[ https://issues.apache.org/jira/browse/OAK-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-7065: - Summary: Remove orphan file from local directory in case indexing fails (was: Analyze case around failure of indexing cycle leading to orphan files) > Remove orphan file from local directory in case indexing fails > -- > > Key: OAK-7065 > URL: https://issues.apache.org/jira/browse/OAK-7065 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Reporter: Chetan Mehrotra >Priority: Major > Fix For: 1.10 > > > If an indexing cycle fails for some reason it may leave orphan files in local > directory. Later on in next indexing cycle Lucene would try to create files > with same name on local disk and this may fail on Windows where such files > may have been memory mapped and hence cannot be deleted. > We should analyze such a scenario and see if system can handle the failure > case properly -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OAK-7065) Analyze case around failure of indexing cycle leading to orphan files
[ https://issues.apache.org/jira/browse/OAK-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra reassigned OAK-7065: Assignee: (was: Chetan Mehrotra) > Analyze case around failure of indexing cycle leading to orphan files > - > > Key: OAK-7065 > URL: https://issues.apache.org/jira/browse/OAK-7065 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Reporter: Chetan Mehrotra >Priority: Major > Fix For: 1.10 > > > If an indexing cycle fails for some reason it may leave orphan files in local > directory. Later on in next indexing cycle Lucene would try to create files > with same name on local disk and this may fail on Windows where such files > may have been memory mapped and hence cannot be deleted. > We should analyze such a scenario and see if system can handle the failure > case properly -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7065) Analyze case around failure of indexing cycle leading to orphan files
[
https://issues.apache.org/jira/browse/OAK-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346271#comment-16346271
]
Chetan Mehrotra commented on OAK-7065:
--
IndexUpdate should support a close operation for IndexEditor which should be
invoked at the end of indexing cycle both in case of success and failure.
Currently the index editor like LuceneIndexEditor implement this close logic by
relying on {{leave}} call and checking if parent is null. However if there is
some exception in any other editor then the editors do not get a chance to
close.
So in case of failure LuceneIndexEditor should
# Close the IndexWriter
# Also ensure that any new files which have been added by the IndexWriter get
removed
> Analyze case around failure of indexing cycle leading to orphan files
> -
>
> Key: OAK-7065
> URL: https://issues.apache.org/jira/browse/OAK-7065
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Major
> Fix For: 1.10
>
>
> If an indexing cycle fails for some reason it may leave orphan files in local
> directory. Later on in next indexing cycle Lucene would try to create files
> with same name on local disk and this may fail on Windows where such files
> may have been memory mapped and hence cannot be deleted.
> We should analyze such a scenario and see if system can handle the failure
> case properly
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-7213) Avoid call for child node when bundle contains all children
[
https://issues.apache.org/jira/browse/OAK-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7213:
-
Labels: bundling (was: )
> Avoid call for child node when bundle contains all children
> ---
>
> Key: OAK-7213
> URL: https://issues.apache.org/jira/browse/OAK-7213
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Labels: bundling
> Fix For: 1.10
>
>
> When nodes are bundled in a document, the DocumentNodeStore keeps track of
> whether all children are included in a document. The presence of the hidden
> {{:doc-has-child-non-bundled}} property indicates there are non bundled child
> nodes. For the case when a document contains all children in the bundle, the
> DocumentNodeStore still does a find call on the DocumentStore when asked for
> an unknown child node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OAK-7212) Document the document order traversal option
Chetan Mehrotra created OAK-7212: Summary: Document the document order traversal option Key: OAK-7212 URL: https://issues.apache.org/jira/browse/OAK-7212 Project: Jackrabbit Oak Issue Type: Documentation Components: doc, run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Fix For: 1.8.2 Document the doc-order-traversal option introduced with OAK-6353 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6254) DataStore: API to retrieve approximate storage size
[
https://issues.apache.org/jira/browse/OAK-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335543#comment-16335543
]
Chetan Mehrotra commented on OAK-6254:
--
bq. The size could be stored in a file, and updated whenever datastore GC is
run.
It may be better to store the unstructured data in NodeStore itself under
specific node.
> DataStore: API to retrieve approximate storage size
> ---
>
> Key: OAK-6254
> URL: https://issues.apache.org/jira/browse/OAK-6254
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob
>Reporter: Thomas Mueller
>Priority: Major
> Fix For: 1.10
>
>
> The estimated size of the datastore (on disk) is needed to:
> * monitor growth over time, or growth of certain operations
> * monitor if garbage collection is effective
> * avoid out of disk space
> * estimate backup size
> * statistical purposes (for example, if there are many repositories, to group
> them by size)
> Datastore size: we could use the following heuristic: We could read the file
> sizes in ./datastore/00/00 (if it exists) and multiply by 65536; or
> ./datastore/00 and multiply by 256. That would give a rough estimation
> (within about 20% for repositories with datastore size > 50 GB).
> I think this is mainly important for the FileDataStore. The S3 datastore, if
> there is a simple and fast S3 API to read the size, then that would be good
> as well, but if there is none, then returning "unknown" is fine for me.
> As for the API, I would use something like this: {{long
> getEstimatedStorageSize(int accuracyLevel)}} with accuracyLevel 1 for
> inaccurate (fastest), 2 more accurate (slower),..., 9 precise (possibly very
> slow). Similar to
> [java.util.zip.Deflater.setLevel|https://docs.oracle.com/javase/7/docs/api/java/util/zip/Deflater.html#setLevel(int)].
> I would expect it takes up to 1 second for accuracyLevel 0, up to 5 seconds
> for accuracyLevel 1, and possibly hours for level 9. With level 1, I would
> read files in 00/00, with level 2 - 8 I would read files in 00, and with
> level 9 I would read all the files. For level 1, I wouldn't stop; for level
> 2, if it takes more than 5 seconds, I would stop and return the current best
> estimate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7167) 1.0: oak-lucene uses packages from oak-core that are not exported
[
https://issues.apache.org/jira/browse/OAK-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328730#comment-16328730
]
Chetan Mehrotra commented on OAK-7167:
--
{quote}just the test?{quote}
Yes
> 1.0: oak-lucene uses packages from oak-core that are not exported
> -
>
> Key: OAK-7167
> URL: https://issues.apache.org/jira/browse/OAK-7167
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Affects Versions: 1.0.40
>Reporter: Julian Reschke
>Priority: Major
> Fix For: 1.0.41
>
>
> See comments in https://issues.apache.org/jira/browse/OAK-5299.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7167) 1.0: oak-lucene uses packages from oak-core that are not exported
[ https://issues.apache.org/jira/browse/OAK-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328692#comment-16328692 ] Chetan Mehrotra commented on OAK-7167: -- Looks like we need to backport the test introduced in OAK-2402 which was introduced for very same reason! > 1.0: oak-lucene uses packages from oak-core that are not exported > - > > Key: OAK-7167 > URL: https://issues.apache.org/jira/browse/OAK-7167 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core >Affects Versions: 1.0.40 >Reporter: Julian Reschke >Priority: Major > Fix For: 1.0.41 > > > See comments in https://issues.apache.org/jira/browse/OAK-5299. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7147) Oak run LuceneIndexer indexes excluded parent nodes
[
https://issues.apache.org/jira/browse/OAK-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7147:
-
Labels: (was: candidate_oak_1_8)
Fix Version/s: 1.8.1
Merge
* 1.8 - 1820948
> Oak run LuceneIndexer indexes excluded parent nodes
> ---
>
> Key: OAK-7147
> URL: https://issues.apache.org/jira/browse/OAK-7147
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: indexing, run
>Affects Versions: 1.8.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.9.0, 1.10, 1.8.1
>
>
> {{LuceneIndexer}} currently indexes parent nodes which are not included by
> includedPaths. This happens because the LuceneIndexer#index does not check
> for path filter result and proceeds to index any node handed to it by the
> DocumentStoreIndexer
> As a fix it should check if the filter result is PathFilter.Result.INCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-7147) Oak run LuceneIndexer indexes excluded parent nodes
[
https://issues.apache.org/jira/browse/OAK-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7147:
-
Affects Version/s: 1.8.0
> Oak run LuceneIndexer indexes excluded parent nodes
> ---
>
> Key: OAK-7147
> URL: https://issues.apache.org/jira/browse/OAK-7147
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: indexing, run
>Affects Versions: 1.8.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.9.0, 1.10, 1.8.1
>
>
> {{LuceneIndexer}} currently indexes parent nodes which are not included by
> includedPaths. This happens because the LuceneIndexer#index does not check
> for path filter result and proceeds to index any node handed to it by the
> DocumentStoreIndexer
> As a fix it should check if the filter result is PathFilter.Result.INCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (OAK-7147) Oak run LuceneIndexer indexes excluded parent nodes
[
https://issues.apache.org/jira/browse/OAK-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-7147.
--
Resolution: Fixed
> Oak run LuceneIndexer indexes excluded parent nodes
> ---
>
> Key: OAK-7147
> URL: https://issues.apache.org/jira/browse/OAK-7147
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: indexing, run
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Labels: candidate_oak_1_8
> Fix For: 1.9.0, 1.10
>
>
> {{LuceneIndexer}} currently indexes parent nodes which are not included by
> includedPaths. This happens because the LuceneIndexer#index does not check
> for path filter result and proceeds to index any node handed to it by the
> DocumentStoreIndexer
> As a fix it should check if the filter result is PathFilter.Result.INCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-7147) Oak run LuceneIndexer indexes excluded parent nodes
[
https://issues.apache.org/jira/browse/OAK-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7147:
-
Fix Version/s: (was: 1.8.1)
> Oak run LuceneIndexer indexes excluded parent nodes
> ---
>
> Key: OAK-7147
> URL: https://issues.apache.org/jira/browse/OAK-7147
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: indexing, run
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Labels: candidate_oak_1_8
> Fix For: 1.9.0, 1.10
>
>
> {{LuceneIndexer}} currently indexes parent nodes which are not included by
> includedPaths. This happens because the LuceneIndexer#index does not check
> for path filter result and proceeds to index any node handed to it by the
> DocumentStoreIndexer
> As a fix it should check if the filter result is PathFilter.Result.INCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-7147) Oak run LuceneIndexer indexes excluded parent nodes
[
https://issues.apache.org/jira/browse/OAK-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-7147:
-
Labels: candidate_oak_1_8 (was: )
> Oak run LuceneIndexer indexes excluded parent nodes
> ---
>
> Key: OAK-7147
> URL: https://issues.apache.org/jira/browse/OAK-7147
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: indexing, run
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Labels: candidate_oak_1_8
> Fix For: 1.9.0, 1.10
>
>
> {{LuceneIndexer}} currently indexes parent nodes which are not included by
> includedPaths. This happens because the LuceneIndexer#index does not check
> for path filter result and proceeds to index any node handed to it by the
> DocumentStoreIndexer
> As a fix it should check if the filter result is PathFilter.Result.INCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-7147) Oak run LuceneIndexer indexes excluded parent nodes
[
https://issues.apache.org/jira/browse/OAK-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323563#comment-16323563
]
Chetan Mehrotra commented on OAK-7147:
--
Fixed with r1820947
> Oak run LuceneIndexer indexes excluded parent nodes
> ---
>
> Key: OAK-7147
> URL: https://issues.apache.org/jira/browse/OAK-7147
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: indexing, run
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.9.0, 1.10, 1.8.1
>
>
> {{LuceneIndexer}} currently indexes parent nodes which are not included by
> includedPaths. This happens because the LuceneIndexer#index does not check
> for path filter result and proceeds to index any node handed to it by the
> DocumentStoreIndexer
> As a fix it should check if the filter result is PathFilter.Result.INCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-7147) Oak run LuceneIndexer indexes excluded parent nodes
Chetan Mehrotra created OAK-7147:
Summary: Oak run LuceneIndexer indexes excluded parent nodes
Key: OAK-7147
URL: https://issues.apache.org/jira/browse/OAK-7147
Project: Jackrabbit Oak
Issue Type: Bug
Components: indexing, run
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Priority: Minor
Fix For: 1.9.0, 1.10, 1.8.1
{{LuceneIndexer}} currently indexes parent nodes which are not included by
includedPaths. This happens because the LuceneIndexer#index does not check for
path filter result and proceeds to index any node handed to it by the
DocumentStoreIndexer
As a fix it should check if the filter result is PathFilter.Result.INCLUDE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (OAK-7124) Support MemoryNodeStore with NodeStoreFixtureProvider
[
https://issues.apache.org/jira/browse/OAK-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-7124.
--
Resolution: Fixed
Done with 1820292
> Support MemoryNodeStore with NodeStoreFixtureProvider
> -
>
> Key: OAK-7124
> URL: https://issues.apache.org/jira/browse/OAK-7124
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: run
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.8, 1.7.15
>
>
> At times we need to use oak-run console to just execute some script (like
> OAK-7122). Currently oak-run console would require a working repository
> access. To support such cases we should enable support for using
> MemoryNodeStore. So following command can be used
> {noformat}
> java -jar oak-run-*.jar console memory
> {noformat}
> The memory NodeStore can be used to play with NodeStore API. Or this can just
> be used to enable launch of groovy script
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-7124) Support MemoryNodeStore with NodeStoreFixtureProvider
Chetan Mehrotra created OAK-7124:
Summary: Support MemoryNodeStore with NodeStoreFixtureProvider
Key: OAK-7124
URL: https://issues.apache.org/jira/browse/OAK-7124
Project: Jackrabbit Oak
Issue Type: Improvement
Components: run
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Fix For: 1.8, 1.7.15
At times we need to use oak-run console to just execute some script (like
OAK-7122). Currently oak-run console would require a working repository access.
To support such cases we should enable support for using MemoryNodeStore. So
following command can be used
{noformat}
java -jar oak-run-*.jar console memory
{noformat}
The memory NodeStore can be used to play with NodeStore API. Or this can just
be used to enable launch of groovy script
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (OAK-7122) Implement script to compare lucene indexes logically
[ https://issues.apache.org/jira/browse/OAK-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7122. -- Resolution: Done > Implement script to compare lucene indexes logically > > > Key: OAK-7122 > URL: https://issues.apache.org/jira/browse/OAK-7122 > Project: Jackrabbit Oak > Issue Type: Task > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8 > > > With Document Traversal based indexing we have implemented a newer indexing > logic. To validate that index produced by it is is same as one done by > existing indexing flow we need to implement a script which can enable > comparing the index content logically > This was recently discussed on lucene mailing list [1] and suggestion there > was it can be done by un-inverting the index. So to enable that we need to > implement a script which can > # Open a Lucene index > # Map the Lucene Document to path of node > # For each document determine what all fields are associated with it (stored > and non stored) > # Dump this content in file sorted by path and for each line field name > sorted by name > Then such dumps can be generated for old and new index and compared via > simple text diff > [1] http://lucene.markmail.org/thread/wt22gk6aufs4uz55 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7123) ChildNodeStateProvider does not return all immediate children
[
https://issues.apache.org/jira/browse/OAK-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-7123.
--
Resolution: Fixed
Done with 1820278
> ChildNodeStateProvider does not return all immediate children
> -
>
> Key: OAK-7123
> URL: https://issues.apache.org/jira/browse/OAK-7123
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: run
>Affects Versions: 1.7.14
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.8, 1.7.15
>
>
> Based on script implemented in OAK-7122 and running it against a test index
> it was observed that some of the relative fields were not getting indexed.
> This happens because the ChildNodeStateProvider#children does not handle the
> immediate children check properly. It would fail for case like
> {noformat}
> /a
> /a/b
> /a/b/c
> /a/d
> /a/d/e
> {noformat}
> Currently it would only report 'b' as child of 'a'.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (OAK-7122) Implement script to compare lucene indexes logically
[
https://issues.apache.org/jira/browse/OAK-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312866#comment-16312866
]
Chetan Mehrotra edited comment on OAK-7122 at 1/5/18 10:24 AM:
---
Implemented the script at [1]. Currently it build up the structure in memory.
If this proves to be problamatic for large index can look into building the
structure on file system
*Usage*
{code}
java -DindexPath=/path/to/indexing-result/indexes/lucene/data \
-jar oak-run-*.jar \
console /path/to/segmentstore \
":load
https://raw.githubusercontent.com/chetanmeh/oak-console-scripts/master/src/main/groovy/lucene/luceneIndexDumper.groovy;
{code}
[1]
https://github.com/chetanmeh/oak-console-scripts/tree/master/src/main/groovy/lucene
was (Author: chetanm):
Implemented the script at [1]. Currently it build up the structure in memory.
If this proves to be problamatic for large index can look into building the
structure on file system
[1]
https://github.com/chetanmeh/oak-console-scripts/tree/master/src/main/groovy/lucene
> Implement script to compare lucene indexes logically
>
>
> Key: OAK-7122
> URL: https://issues.apache.org/jira/browse/OAK-7122
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: run
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.8
>
>
> With Document Traversal based indexing we have implemented a newer indexing
> logic. To validate that index produced by it is is same as one done by
> existing indexing flow we need to implement a script which can enable
> comparing the index content logically
> This was recently discussed on lucene mailing list [1] and suggestion there
> was it can be done by un-inverting the index. So to enable that we need to
> implement a script which can
> # Open a Lucene index
> # Map the Lucene Document to path of node
> # For each document determine what all fields are associated with it (stored
> and non stored)
> # Dump this content in file sorted by path and for each line field name
> sorted by name
> Then such dumps can be generated for old and new index and compared via
> simple text diff
> [1] http://lucene.markmail.org/thread/wt22gk6aufs4uz55
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-7123) ChildNodeStateProvider does not return all immediate children
Chetan Mehrotra created OAK-7123:
Summary: ChildNodeStateProvider does not return all immediate
children
Key: OAK-7123
URL: https://issues.apache.org/jira/browse/OAK-7123
Project: Jackrabbit Oak
Issue Type: Bug
Components: run
Affects Versions: 1.7.14
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Fix For: 1.8, 1.7.15
Based on script implemented in OAK-7122 and running it against a test index it
was observed that some of the relative fields were not getting indexed. This
happens because the ChildNodeStateProvider#children does not handle the
immediate children check properly. It would fail for case like
{noformat}
/a
/a/b
/a/b/c
/a/d
/a/d/e
{noformat}
Currently it would only report 'b' as child of 'a'.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-7122) Implement script to compare lucene indexes logically
[ https://issues.apache.org/jira/browse/OAK-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312866#comment-16312866 ] Chetan Mehrotra commented on OAK-7122: -- Implemented the script at [1]. Currently it build up the structure in memory. If this proves to be problamatic for large index can look into building the structure on file system [1] https://github.com/chetanmeh/oak-console-scripts/tree/master/src/main/groovy/lucene > Implement script to compare lucene indexes logically > > > Key: OAK-7122 > URL: https://issues.apache.org/jira/browse/OAK-7122 > Project: Jackrabbit Oak > Issue Type: Task > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8 > > > With Document Traversal based indexing we have implemented a newer indexing > logic. To validate that index produced by it is is same as one done by > existing indexing flow we need to implement a script which can enable > comparing the index content logically > This was recently discussed on lucene mailing list [1] and suggestion there > was it can be done by un-inverting the index. So to enable that we need to > implement a script which can > # Open a Lucene index > # Map the Lucene Document to path of node > # For each document determine what all fields are associated with it (stored > and non stored) > # Dump this content in file sorted by path and for each line field name > sorted by name > Then such dumps can be generated for old and new index and compared via > simple text diff > [1] http://lucene.markmail.org/thread/wt22gk6aufs4uz55 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7122) Implement script to compare lucene indexes logically
Chetan Mehrotra created OAK-7122: Summary: Implement script to compare lucene indexes logically Key: OAK-7122 URL: https://issues.apache.org/jira/browse/OAK-7122 Project: Jackrabbit Oak Issue Type: Task Components: run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Fix For: 1.8 With Document Traversal based indexing we have implemented a newer indexing logic. To validate that index produced by it is is same as one done by existing indexing flow we need to implement a script which can enable comparing the index content logically This was recently discussed on lucene mailing list [1] and suggestion there was it can be done by un-inverting the index. So to enable that we need to implement a script which can # Open a Lucene index # Map the Lucene Document to path of node # For each document determine what all fields are associated with it (stored and non stored) # Dump this content in file sorted by path and for each line field name sorted by name Then such dumps can be generated for old and new index and compared via simple text diff [1] http://lucene.markmail.org/thread/wt22gk6aufs4uz55 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7115) Store NodeState json in bytes when storing in in-memory queue
[ https://issues.apache.org/jira/browse/OAK-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7115. -- Resolution: Fixed Fix Version/s: (was: 1.10) 1.7.15 1.8 Done with 1819936. It just stores the json in bytes but does not perform any compression. With this change the dumping time for 65M nodestates reduced from 2.632h to 2.230h i.e. saving of 24 mins! > Store NodeState json in bytes when storing in in-memory queue > - > > Key: OAK-7115 > URL: https://issues.apache.org/jira/browse/OAK-7115 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.8, 1.7.15 > > Attachments: OAK-7115-v1.patch > > > Currently TraverseWithSortStrategy stores the NodeStateEntry as json text in > the in-memory queue. We can save memory by storing it in byte array and > probably compressed which would allow storing more entries in-memory before > sorting and saving in the file -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-7115) Store NodeState json in bytes when storing in in-memory queue
[ https://issues.apache.org/jira/browse/OAK-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-7115: - Summary: Store NodeState json in bytes when storing in in-memory queue (was: Compress NodeStateEntry when storing in in-memory queue) > Store NodeState json in bytes when storing in in-memory queue > - > > Key: OAK-7115 > URL: https://issues.apache.org/jira/browse/OAK-7115 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.10 > > Attachments: OAK-7115-v1.patch > > > Currently TraverseWithSortStrategy stores the NodeStateEntry as json text in > the in-memory queue. We can save memory by storing it in byte array and > probably compressed which would allow storing more entries in-memory before > sorting and saving in the file -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-7115) Compress NodeStateEntry when storing in in-memory queue
[ https://issues.apache.org/jira/browse/OAK-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-7115: - Attachment: OAK-7115-v1.patch [patch|^OAK-7115-v1.patch] for the same. Perf test under run > Compress NodeStateEntry when storing in in-memory queue > --- > > Key: OAK-7115 > URL: https://issues.apache.org/jira/browse/OAK-7115 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.10 > > Attachments: OAK-7115-v1.patch > > > Currently TraverseWithSortStrategy stores the NodeStateEntry as json text in > the in-memory queue. We can save memory by storing it in byte array and > probably compressed which would allow storing more entries in-memory before > sorting and saving in the file -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-7116) Use JMX mode to reindex on SegmentNodeStore without requiring manual steps
[ https://issues.apache.org/jira/browse/OAK-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-7116: - Description: oak-run indexing for SegmentNodeStore currently require following steps while performing indexing against a repository which is in use [1] # Create checkpoint via MBean and pass it as part of cli args # Perform actual indexing with read only access to repo # Import the index via MBean operation As per current documented steps #1 and #3 are manual. This can potentially be simplified by directly using JMX operation from within oak-run as currently for accessing SegmentNodeStore oak-run needs to run on same host as actual application *JMX Access* JMX access can be done via following ways # Application using Oak has JMX remote ## Enabled and same info provided as cli args ## Not enabled - In such a case we can rely on ### pid and [local connection|https://stackoverflow.com/questions/13252914/how-to-connect-to-a-local-jmx-server-by-knowing-the-process-id] or [attach|https://github.com/nickman/jmxlocal] ### Or query all running java prcess jmx and check if SegmentNodeStore repo path is same as one provided in cli # Application using OAk *Proposed Approach* # Establish the JMX connection # Create checkpoint using the JMX connection programatically # Perform indexing with read only access # Import the index via JMX access With this indexing support for SegmentNodeStore would be at par with DocumentNodeStore in terms of ease of use [1] https://jackrabbit.apache.org/oak/docs/query/oak-run-indexing.html was: oak-run indexing for SegmentNodeStore currently require following steps while performing indexing against a repository which is in use [1] # Create checkpoint via MBean and pass it as part of cli args # Perform actual indexing with read only access to repo # Import the index via MBean operation As per current documented steps #1 and #3 are manual. This can potentially be simplified by directly using JMX operation from within oak-run as currently for accessing SegmentNodeStore oak-run needs to run on same host as actual application *JMX Access* JMX access can be done via following ways # Application using Oak has JMX remote ## Enabled and same info provided as cli args ## Not enabled - In such a case we can rely on ### pid and [local connection|https://stackoverflow.com/questions/13252914/how-to-connect-to-a-local-jmx-server-by-knowing-the-process-id] ### Or query all running java prcess jmx and check if SegmentNodeStore repo path is same as one provided in cli # Application using OAk *Proposed Approach* # Establish the JMX connection # Create checkpoint using the JMX connection programatically # Perform indexing with read only access # Import the index via JMX access With this indexing support for SegmentNodeStore would be at par with DocumentNodeStore in terms of ease of use [1] https://jackrabbit.apache.org/oak/docs/query/oak-run-indexing.html > Use JMX mode to reindex on SegmentNodeStore without requiring manual steps > -- > > Key: OAK-7116 > URL: https://issues.apache.org/jira/browse/OAK-7116 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.10 > > > oak-run indexing for SegmentNodeStore currently require following steps while > performing indexing against a repository which is in use [1] > # Create checkpoint via MBean and pass it as part of cli args > # Perform actual indexing with read only access to repo > # Import the index via MBean operation > As per current documented steps #1 and #3 are manual. This can potentially be > simplified by directly using JMX operation from within oak-run as currently > for accessing SegmentNodeStore oak-run needs to run on same host as actual > application > *JMX Access* > JMX access can be done via following ways > # Application using Oak has JMX remote > ## Enabled and same info provided as cli args > ## Not enabled - In such a case we can rely on > ### pid and [local > connection|https://stackoverflow.com/questions/13252914/how-to-connect-to-a-local-jmx-server-by-knowing-the-process-id] > or [attach|https://github.com/nickman/jmxlocal] > ### Or query all running java prcess jmx and check if SegmentNodeStore repo > path is same as one provided in cli > # Application using OAk > *Proposed Approach* > # Establish the JMX connection > # Create checkpoint using the JMX connection programatically > # Perform indexing with read only access > # Import the index via JMX access > With this indexing support for SegmentNodeStore would be at par with > DocumentNodeStore in terms of ease of use > [1] https://jackrabbit.apache.org/oak/docs/query/oak-run-indexing.html
[jira] [Commented] (OAK-7116) Use JMX mode to reindex on SegmentNodeStore without requiring manual steps
[ https://issues.apache.org/jira/browse/OAK-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307884#comment-16307884 ] Chetan Mehrotra commented on OAK-7116: -- [~catholicon] [~tmueller] [~dhasler] Thoughts? > Use JMX mode to reindex on SegmentNodeStore without requiring manual steps > -- > > Key: OAK-7116 > URL: https://issues.apache.org/jira/browse/OAK-7116 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.10 > > > oak-run indexing for SegmentNodeStore currently require following steps while > performing indexing against a repository which is in use [1] > # Create checkpoint via MBean and pass it as part of cli args > # Perform actual indexing with read only access to repo > # Import the index via MBean operation > As per current documented steps #1 and #3 are manual. This can potentially be > simplified by directly using JMX operation from within oak-run as currently > for accessing SegmentNodeStore oak-run needs to run on same host as actual > application > *JMX Access* > JMX access can be done via following ways > # Application using Oak has JMX remote > ## Enabled and same info provided as cli args > ## Not enabled - In such a case we can rely on > ### pid and [local > connection|https://stackoverflow.com/questions/13252914/how-to-connect-to-a-local-jmx-server-by-knowing-the-process-id] > > ### Or query all running java prcess jmx and check if SegmentNodeStore repo > path is same as one provided in cli > # Application using OAk > *Proposed Approach* > # Establish the JMX connection > # Create checkpoint using the JMX connection programatically > # Perform indexing with read only access > # Import the index via JMX access > With this indexing support for SegmentNodeStore would be at par with > DocumentNodeStore in terms of ease of use > [1] https://jackrabbit.apache.org/oak/docs/query/oak-run-indexing.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7116) Use JMX mode to reindex on SegmentNodeStore without requiring manual steps
Chetan Mehrotra created OAK-7116: Summary: Use JMX mode to reindex on SegmentNodeStore without requiring manual steps Key: OAK-7116 URL: https://issues.apache.org/jira/browse/OAK-7116 Project: Jackrabbit Oak Issue Type: Improvement Components: run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Fix For: 1.10 oak-run indexing for SegmentNodeStore currently require following steps while performing indexing against a repository which is in use [1] # Create checkpoint via MBean and pass it as part of cli args # Perform actual indexing with read only access to repo # Import the index via MBean operation As per current documented steps #1 and #3 are manual. This can potentially be simplified by directly using JMX operation from within oak-run as currently for accessing SegmentNodeStore oak-run needs to run on same host as actual application *JMX Access* JMX access can be done via following ways # Application using Oak has JMX remote ## Enabled and same info provided as cli args ## Not enabled - In such a case we can rely on ### pid and [local connection|https://stackoverflow.com/questions/13252914/how-to-connect-to-a-local-jmx-server-by-knowing-the-process-id] ### Or query all running java prcess jmx and check if SegmentNodeStore repo path is same as one provided in cli # Application using OAk *Proposed Approach* # Establish the JMX connection # Create checkpoint using the JMX connection programatically # Perform indexing with read only access # Import the index via JMX access With this indexing support for SegmentNodeStore would be at par with DocumentNodeStore in terms of ease of use [1] https://jackrabbit.apache.org/oak/docs/query/oak-run-indexing.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-7115) Compress NodeStateEntry when storing in in-memory queue
[ https://issues.apache.org/jira/browse/OAK-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-7115: - Summary: Compress NodeStateEntry when storing in in-memory queue (was: Compress NodeStateEntry when storing in in memory queue) > Compress NodeStateEntry when storing in in-memory queue > --- > > Key: OAK-7115 > URL: https://issues.apache.org/jira/browse/OAK-7115 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.10 > > > Currently TraverseWithSortStrategy stores the NodeStateEntry as json text in > the in-memory queue. We can save memory by storing it in byte array and > probably compressed which would allow storing more entries in-memory before > sorting and saving in the file -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7115) Compress NodeStateEntry when storing in in memory queue
Chetan Mehrotra created OAK-7115: Summary: Compress NodeStateEntry when storing in in memory queue Key: OAK-7115 URL: https://issues.apache.org/jira/browse/OAK-7115 Project: Jackrabbit Oak Issue Type: Improvement Components: run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Priority: Minor Fix For: 1.10 Currently TraverseWithSortStrategy stores the NodeStateEntry as json text in the in-memory queue. We can save memory by storing it in byte array and probably compressed which would allow storing more entries in-memory before sorting and saving in the file -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-6460) Index related tooling
[ https://issues.apache.org/jira/browse/OAK-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-6460. -- Resolution: Fixed > Index related tooling > - > > Key: OAK-6460 > URL: https://issues.apache.org/jira/browse/OAK-6460 > Project: Jackrabbit Oak > Issue Type: Epic > Components: indexing, run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8 > > > To enable better management for indexing related operation specially around > reindexing indexes on large repository setup we should implement some tooling > as part of oak-run. This epic is meant to track all work done in this area -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7106) Index Tooling for Oak 1.10
Chetan Mehrotra created OAK-7106: Summary: Index Tooling for Oak 1.10 Key: OAK-7106 URL: https://issues.apache.org/jira/browse/OAK-7106 Project: Jackrabbit Oak Issue Type: Epic Components: indexing, run Reporter: Chetan Mehrotra Fix For: 1.10 Epic to track tooling work for Oak 1.10 release -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-7105) Implement a traverse with sort strategy for DocumentStoreIndexer
[ https://issues.apache.org/jira/browse/OAK-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299858#comment-16299858 ] Chetan Mehrotra commented on OAK-7105: -- Switched the default with 1818900 > Implement a traverse with sort strategy for DocumentStoreIndexer > > > Key: OAK-7105 > URL: https://issues.apache.org/jira/browse/OAK-7105 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.15 > > > Currently the DocumentStoreIndexer logic uses a StoreAndSortStrategy in which > it first dumps all nodestates to a json file -> sort them in batches -> merge > the sorted file. In whole indexing the sorting phase is taking decent amount > of time (40 mins out of 3 hr run). > Further this approach suffers with potential OOM while ExternalSort creates > in memory batches where actual size of batch exceeds the estimated size > considerably. So we need to constant tweak the > "oak.indexer.maxSortMemoryInGB" (currently set to 2 GB) > As an improvement we can do following changes > # Implement a traverse with sort strategy - Here instead of first dumping all > nodestate in a single big json we instead add them to an in memory buffer and > then at some stage sort the batch and save it to file > # Use better memory checks - Use the approach as implemented in GCBarrier > i.e. monitor the current memory usage and if it goes below certain threshold > trigger the batch sort -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7105) Implement a traverse with sort strategy for DocumentStoreIndexer
[ https://issues.apache.org/jira/browse/OAK-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7105. -- Resolution: Fixed > Implement a traverse with sort strategy for DocumentStoreIndexer > > > Key: OAK-7105 > URL: https://issues.apache.org/jira/browse/OAK-7105 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.15 > > > Currently the DocumentStoreIndexer logic uses a StoreAndSortStrategy in which > it first dumps all nodestates to a json file -> sort them in batches -> merge > the sorted file. In whole indexing the sorting phase is taking decent amount > of time (40 mins out of 3 hr run). > Further this approach suffers with potential OOM while ExternalSort creates > in memory batches where actual size of batch exceeds the estimated size > considerably. So we need to constant tweak the > "oak.indexer.maxSortMemoryInGB" (currently set to 2 GB) > As an improvement we can do following changes > # Implement a traverse with sort strategy - Here instead of first dumping all > nodestate in a single big json we instead add them to an in memory buffer and > then at some stage sort the batch and save it to file > # Use better memory checks - Use the approach as implemented in GCBarrier > i.e. monitor the current memory usage and if it goes below certain threshold > trigger the batch sort -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-7105) Implement a traverse with sort strategy for DocumentStoreIndexer
[ https://issues.apache.org/jira/browse/OAK-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299800#comment-16299800 ] Chetan Mehrotra commented on OAK-7105: -- Implemented the above flow with 1818896 > Implement a traverse with sort strategy for DocumentStoreIndexer > > > Key: OAK-7105 > URL: https://issues.apache.org/jira/browse/OAK-7105 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.15 > > > Currently the DocumentStoreIndexer logic uses a StoreAndSortStrategy in which > it first dumps all nodestates to a json file -> sort them in batches -> merge > the sorted file. In whole indexing the sorting phase is taking decent amount > of time (40 mins out of 3 hr run). > Further this approach suffers with potential OOM while ExternalSort creates > in memory batches where actual size of batch exceeds the estimated size > considerably. So we need to constant tweak the > "oak.indexer.maxSortMemoryInGB" (currently set to 2 GB) > As an improvement we can do following changes > # Implement a traverse with sort strategy - Here instead of first dumping all > nodestate in a single big json we instead add them to an in memory buffer and > then at some stage sort the batch and save it to file > # Use better memory checks - Use the approach as implemented in GCBarrier > i.e. monitor the current memory usage and if it goes below certain threshold > trigger the batch sort -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7102) Refactor DocumentIndexer logic to enable different sort approaches
[ https://issues.apache.org/jira/browse/OAK-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7102. -- Resolution: Fixed Done with various commits in trunk > Refactor DocumentIndexer logic to enable different sort approaches > -- > > Key: OAK-7102 > URL: https://issues.apache.org/jira/browse/OAK-7102 > Project: Jackrabbit Oak > Issue Type: Task > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.7.14, 1.8 > > > DocumentStoreIndexer logic needs to be refactored to support plugging in > different sort approaches -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7105) Implement a traverse with sort strategy for DocumentStoreIndexer
Chetan Mehrotra created OAK-7105: Summary: Implement a traverse with sort strategy for DocumentStoreIndexer Key: OAK-7105 URL: https://issues.apache.org/jira/browse/OAK-7105 Project: Jackrabbit Oak Issue Type: Improvement Components: run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Fix For: 1.8, 1.7.15 Currently the DocumentStoreIndexer logic uses a StoreAndSortStrategy in which it first dumps all nodestates to a json file -> sort them in batches -> merge the sorted file. In whole indexing the sorting phase is taking decent amount of time (40 mins out of 3 hr run). Further this approach suffers with potential OOM while ExternalSort creates in memory batches where actual size of batch exceeds the estimated size considerably. So we need to constant tweak the "oak.indexer.maxSortMemoryInGB" (currently set to 2 GB) As an improvement we can do following changes # Implement a traverse with sort strategy - Here instead of first dumping all nodestate in a single big json we instead add them to an in memory buffer and then at some stage sort the batch and save it to file # Use better memory checks - Use the approach as implemented in GCBarrier i.e. monitor the current memory usage and if it goes below certain threshold trigger the batch sort -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7103) Enable compression by default on DocumentStoreIndexer logic
[ https://issues.apache.org/jira/browse/OAK-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7103. -- Resolution: Fixed Fix Version/s: (was: 1.7.14) 1.7.15 Done with 1818879 > Enable compression by default on DocumentStoreIndexer logic > --- > > Key: OAK-7103 > URL: https://issues.apache.org/jira/browse/OAK-7103 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.15 > > > While performing tests it appears that enabling end to end compression > reduces the sorting time by 14 mins (39.87 min to 26.44 min) and disk > consumption by 65GB (87GB to 12.5). Based on that we should enable > compression by default for > # Create compressed base store.json written by traversal > # Enable compression for intermediate files created while sorting > # Enable compression for finally sorted json file -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7104) Support read and write to compressed file in ExternalSort
[ https://issues.apache.org/jira/browse/OAK-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7104. -- Resolution: Fixed Fix Version/s: (was: 1.7.14) 1.7.15 Done with http://svn.apache.org/viewvc?rev=1818878=rev [~amjain] Please review the commit once > Support read and write to compressed file in ExternalSort > - > > Key: OAK-7104 > URL: https://issues.apache.org/jira/browse/OAK-7104 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: commons >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.7.15, 1.8 > > > Currently ExternalSort only support compression for intermediate file created > in merge phase. It would be good to also support reading and writing to > compressed file -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-7104) Support read and write to compressed file in ExternalSort
[ https://issues.apache.org/jira/browse/OAK-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-7104: - Summary: Support read and write to compressed file in ExternalSort (was: Support read and writing to compressed file in ExternalSort) > Support read and write to compressed file in ExternalSort > - > > Key: OAK-7104 > URL: https://issues.apache.org/jira/browse/OAK-7104 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: commons >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.7.14, 1.8 > > > Currently ExternalSort only support compression for intermediate file created > in merge phase. It would be good to also support reading and writing to > compressed file -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7104) Support read and writing to compressed file in ExternalSort
Chetan Mehrotra created OAK-7104: Summary: Support read and writing to compressed file in ExternalSort Key: OAK-7104 URL: https://issues.apache.org/jira/browse/OAK-7104 Project: Jackrabbit Oak Issue Type: Improvement Components: commons Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Fix For: 1.7.14, 1.8 Currently ExternalSort only support compression for intermediate file created in merge phase. It would be good to also support reading and writing to compressed file -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7103) Enable compression by default on DocumentStoreIndexer logic
Chetan Mehrotra created OAK-7103: Summary: Enable compression by default on DocumentStoreIndexer logic Key: OAK-7103 URL: https://issues.apache.org/jira/browse/OAK-7103 Project: Jackrabbit Oak Issue Type: Improvement Components: run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Fix For: 1.7.14, 1.8 While performing tests it appears that enabling end to end compression reduces the sorting time by 14 mins (39.87 min to 26.44 min) and disk consumption by 65GB (87GB to 12.5). Based on that we should enable compression by default for # Create compressed base store.json written by traversal # Enable compression for intermediate files created while sorting # Enable compression for finally sorted json file -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7102) Refactor DocumentIndexer logic to enable different sort approaches
Chetan Mehrotra created OAK-7102: Summary: Refactor DocumentIndexer logic to enable different sort approaches Key: OAK-7102 URL: https://issues.apache.org/jira/browse/OAK-7102 Project: Jackrabbit Oak Issue Type: Task Components: run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Fix For: 1.7.14, 1.8 DocumentStoreIndexer logic needs to be refactored to support plugging in different sort approaches -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7099) DocumentStoreIndexer should log estimate of ETA for dumping phase
[ https://issues.apache.org/jira/browse/OAK-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7099. -- Resolution: Fixed Done with 1818768 > DocumentStoreIndexer should log estimate of ETA for dumping phase > - > > Key: OAK-7099 > URL: https://issues.apache.org/jira/browse/OAK-7099 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.8, 1.7.14 > > > DocumentStoreIndexer currently does not log ETA for dumping phase -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7097) DocumentStoreIndexer should clear the index state prior to indexing
[ https://issues.apache.org/jira/browse/OAK-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7097. -- Resolution: Fixed Done with 1818758 > DocumentStoreIndexer should clear the index state prior to indexing > --- > > Key: OAK-7097 > URL: https://issues.apache.org/jira/browse/OAK-7097 > Project: Jackrabbit Oak > Issue Type: Bug > Components: run >Affects Versions: 1.7.13 >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.14 > > > DocumentStoreIndexer currently implements some part of logic which is present > in IndexUpdate. However it misses on 2 things > # Removing the hidden index state > # Resetting the reindexing flag > Those should be implemented -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7099) DocumentStoreIndexer should log estimate of ETA for dumping phase
Chetan Mehrotra created OAK-7099: Summary: DocumentStoreIndexer should log estimate of ETA for dumping phase Key: OAK-7099 URL: https://issues.apache.org/jira/browse/OAK-7099 Project: Jackrabbit Oak Issue Type: Improvement Components: run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Priority: Minor Fix For: 1.8, 1.7.14 DocumentStoreIndexer currently does not log ETA for dumping phase -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-7098) Refactor common logic between IndexUpdate and DocumentStoreIndexer
[ https://issues.apache.org/jira/browse/OAK-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-7098: - Summary: Refactor common logic between IndexUpdate and DocumentStoreIndexer (was: Refcator common logic between IndexUpdate and DocumentStoreIndexer) > Refactor common logic between IndexUpdate and DocumentStoreIndexer > -- > > Key: OAK-7098 > URL: https://issues.apache.org/jira/browse/OAK-7098 > Project: Jackrabbit Oak > Issue Type: Task > Components: indexing, run >Reporter: Chetan Mehrotra > Fix For: 1.10 > > > DocumentStoreIndexer implements an alternative way of indexing which differs > from diff based indexing done by IndexUpdate. However some part of logic is > commong > We should refactor them and abstract them out so both can share same logic -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7098) Refcator common logic between IndexUpdate and DocumentStoreIndexer
Chetan Mehrotra created OAK-7098: Summary: Refcator common logic between IndexUpdate and DocumentStoreIndexer Key: OAK-7098 URL: https://issues.apache.org/jira/browse/OAK-7098 Project: Jackrabbit Oak Issue Type: Task Components: indexing, run Reporter: Chetan Mehrotra Fix For: 1.10 DocumentStoreIndexer implements an alternative way of indexing which differs from diff based indexing done by IndexUpdate. However some part of logic is commong We should refactor them and abstract them out so both can share same logic -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-7097) DocumentStoreIndexer should clear the index state prior to indexing
[ https://issues.apache.org/jira/browse/OAK-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-7097: - Affects Version/s: 1.7.13 > DocumentStoreIndexer should clear the index state prior to indexing > --- > > Key: OAK-7097 > URL: https://issues.apache.org/jira/browse/OAK-7097 > Project: Jackrabbit Oak > Issue Type: Bug > Components: run >Affects Versions: 1.7.13 >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.14 > > > DocumentStoreIndexer currently implements some part of logic which is present > in IndexUpdate. However it misses on 2 things > # Removing the hidden index state > # Resetting the reindexing flag > Those should be implemented -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7097) DocumentStoreIndexer should clear the index state prior to indexing
Chetan Mehrotra created OAK-7097: Summary: DocumentStoreIndexer should clear the index state prior to indexing Key: OAK-7097 URL: https://issues.apache.org/jira/browse/OAK-7097 Project: Jackrabbit Oak Issue Type: Bug Components: run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Fix For: 1.8, 1.7.14 DocumentStoreIndexer currently implements some part of logic which is present in IndexUpdate. However it misses on 2 things # Removing the hidden index state # Resetting the reindexing flag Those should be implemented -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7095) NodeStoreFixtureProvider should use BlobStore from DocumentNodeStore if no DataStore configured
[ https://issues.apache.org/jira/browse/OAK-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7095. -- Resolution: Fixed Done with 1818751 > NodeStoreFixtureProvider should use BlobStore from DocumentNodeStore if no > DataStore configured > --- > > Key: OAK-7095 > URL: https://issues.apache.org/jira/browse/OAK-7095 > Project: Jackrabbit Oak > Issue Type: Bug > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.14 > > > NodeStoreFixtureProvider currently works fine for explicitly configured > BlobStore. However for setups like Mongo where is no external DataStore is > configured an implicit one is created then that BlobStore is not exposed. > So NodeStoreFixtureProvider should expose such a BlobStore -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7095) NodeStoreFixtureProvider should use BlobStore from DocumentNodeStore if no DataStore configured
Chetan Mehrotra created OAK-7095: Summary: NodeStoreFixtureProvider should use BlobStore from DocumentNodeStore if no DataStore configured Key: OAK-7095 URL: https://issues.apache.org/jira/browse/OAK-7095 Project: Jackrabbit Oak Issue Type: Bug Components: run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Fix For: 1.8, 1.7.14 NodeStoreFixtureProvider currently works fine for explicitly configured BlobStore. However for setups like Mongo where is no external DataStore is configured an implicit one is created then that BlobStore is not exposed. So NodeStoreFixtureProvider should expose such a BlobStore -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7094) Log cli arguments and vm arguments passed to indexer command
[ https://issues.apache.org/jira/browse/OAK-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7094. -- Resolution: Fixed Done with 1818746 > Log cli arguments and vm arguments passed to indexer command > > > Key: OAK-7094 > URL: https://issues.apache.org/jira/browse/OAK-7094 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.8, 1.7.14 > > > It would be useful to also log the cli arguments to the indexing.log as that > would help in analysing any customer reported issue -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7094) Log cli arguments and vm arguments passed to indexer command
Chetan Mehrotra created OAK-7094: Summary: Log cli arguments and vm arguments passed to indexer command Key: OAK-7094 URL: https://issues.apache.org/jira/browse/OAK-7094 Project: Jackrabbit Oak Issue Type: Improvement Components: run Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Priority: Minor Fix For: 1.7.14, 1.8 It would be useful to also log the cli arguments to the indexing.log as that would help in analysing any customer reported issue -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6353) Use Document order traversal for reindexing performed on DocumentNodeStore setups
[
https://issues.apache.org/jira/browse/OAK-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297924#comment-16297924
]
Chetan Mehrotra commented on OAK-6353:
--
Some performance numbers for reindexing done for repo having 255M Mongo Docs,
66M nodes under /content and having 4.2M assets
# Normal NodeStore traversal - 13.66 h
*Document Traversal*
A - Default setup
# Total time - 3.469 h
## Time in dumping - 2.405 h
## Time in sorting - 39.87 min
### Batch sorting - 19.13 min
### Merging - 20.17
## Indexing 24 mins
# Space consumed
#* dumped json - 43.6 GB
#* chunked files - 43.6 GB
#* index size - 2.5 GB
{noformat}
2017-12-15 16:48:34 Proceeding to index [/oak:index/damAssetLucene2] upto
checkpoint head {}
2017-12-15 19:12:55 Dumped 65472172 nodestates in json format in 2.405 h
2017-12-15 19:12:55 Compression enabled while sorting : false
(oak.indexer.useZip)
2017-12-15 19:12:55 Delete original dump from traversal : true
(oak.indexer.deleteOriginal)
2017-12-15 19:12:55 Max heap memory (GB) to be used for merge sort : 3
(oak.indexer.maxSortMemoryInGB)
2017-12-15 19:12:57 Sorting with memory 3.2 GB (estimated 12.6 GB)
2017-12-15 19:32:05 Batch sorting done in 19.13 min with 29 files of size 43.6
GB to merge
2017-12-15 19:32:05 Removing the original file temp/flat-file-store/store.json
2017-12-15 19:52:50 Merging of sorted files completed in 20.71 min
2017-12-15 19:52:50 Sorting completed in 39.87 min
2017-12-15 19:52:50 Estimated node count to be traversed for reindexing under /
is [65472172]
2017-12-15 20:16:35 Indexing report
- /oak:index/damAssetLucene2*(4407265)
2017-12-15 20:16:43 Indexing completed for indexes [/oak:index/damAssetLucene2]
in 3.469 h (12488171 ms)
{noformat}
B - Compression enabled in sorting
# Total time - 3.811 h
## Time in dumping - 2.929 h
## Time in sorting - 29.56 min
### Batch sorting - 17.67 min
### Merging - 11.87 min
## Indexing 24 mins
# Space consumed
#* dumped json - 43.6 GB
#* chunked files - 5.5 GB
#* index size - 2.5 GB
{noformat}
2017-12-19 10:56:00 Proceeding to index [/oak:index/damAssetLucene2] upto
checkpoint head {}
2017-12-19 13:51:50 oreBuilder - Dumped 65469575 nodestates in json format in
2.929 h (43.6 GB)
2017-12-19 13:51:50 oreBuilder - Compression enabled while sorting : true
(oak.indexer.useZip)
2017-12-19 13:51:50 oreBuilder - Delete original dump from traversal : true
(oak.indexer.deleteOriginal)
2017-12-19 13:51:50 oreBuilder - Max heap memory (GB) to be used for merge sort
: 3 (oak.indexer.maxSortMemoryInGB)
2017-12-19 13:51:52 Sorter - Sorting with memory 3.2 GB (estimated 12.6 GB)
2017-12-19 14:09:32 Sorter - Batch sorting done in 17.67 min with 29 files of
size 5.5 GB to merge
2017-12-19 14:09:32 Sorter - Removing the original file
temp/flat-file-store/store.json
2017-12-19 14:21:25 Sorter - Merging of sorted files completed in 11.87 min
2017-12-19 14:21:25 Sorter - Sorting completed in 29.56 min
2017-12-19 14:21:26 Estimated node count to be traversed for reindexing under /
is [65469575]
2017-12-19 14:44:30 Indexing report
- /oak:index/damAssetLucene2*(4407265)
2017-12-19 14:44:30 Reindexing completed
2017-12-19 14:44:30 Switched the async lane for indexes at
[/oak:index/damAssetLucene2] back to there original lanes
2017-12-19 14:44:39 Indexing completed for indexes [/oak:index/damAssetLucene2]
in 3.811 h (13718589 ms)
{noformat}
> Use Document order traversal for reindexing performed on DocumentNodeStore
> setups
> -
>
> Key: OAK-6353
> URL: https://issues.apache.org/jira/browse/OAK-6353
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: run
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.7.13, 1.8
>
> Attachments: OAK-6353-v1.patch, OAK-6353-v2.patch
>
>
> [~tmueller] suggested
> [here|https://issues.apache.org/jira/browse/OAK-6246?focusedCommentId=16034442=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16034442]
> that document order traversal can be faster compared to current mode of path
> based traversal. Initial test indicate that such a traversal can be order of
> magnitude faster.
> So this task is meant to implement such an approach and see if it can be a
> viable indexing mode used for DocumentNodeStore based setups
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
