[jira] [Comment Edited] (OAK-4675) SNFE thrown while testing FileStore.cleanup() running concurrently with writes
[
https://issues.apache.org/jira/browse/OAK-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426761#comment-15426761
]
Michael Dürig edited comment on OAK-4675 at 8/18/16 4:36 PM:
-
Initial patch for {{oak-segment}} [^OAK-4675-02.patch]. There is a problem
however. The test occasionally fails with
{code}
java.util.concurrent.ExecutionException: java.io.IOException: Stream Closed
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT.randomAccessFileConcurrentReadAndLength(CompactionAndCleanupIT.java:711)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.io.IOException: Stream Closed
at java.io.RandomAccessFile.length(Native Method)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileAccess$Random.length(FileAccess.java:101)
at
org.apache.jackrabbit.oak.plugins.segment.file.TarReader.loadGraph(TarReader.java:893)
at
org.apache.jackrabbit.oak.plugins.segment.file.TarReader.getGraph(TarReader.java:877)
at
org.apache.jackrabbit.oak.plugins.segment.file.TarReader.calculateForwardReferences(TarReader.java:721)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.includeForwardReferences(FileStore.java:948)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.cleanup(FileStore.java:878)
at
org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT$13.call(CompactionAndCleanupIT.java:690)
at
org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT$13.call(CompactionAndCleanupIT.java:687)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
{code}
This is caused by concurrent calls to {{FileStore.cleanup()}}.
[~dulceanu], I think we should avoid concurrent calls to
{{FileStore.cleanup()}} and also {{FileStore.compact()}} (and also make them
mutually exclusive). This deserves an issue in its own. Once we have this we
need to check whether {{randomAccessFileConcurrentReadAndLength()}} still
reproduces this issue and otherwise try to rework it.
Again I suggest we start with fixing this in {{oak-segment-tar}} followed by
{{oak-segment}} followed by the branches.
was (Author: mduerig):
Initial patch for {{oak-segment}} [^OAK-4675-02.patch]. There is a problem
however. The test occasionally fails with
{code}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at
[jira] [Comment Edited] (OAK-4675) SNFE thrown while testing FileStore.cleanup() running concurrently with writes
[
https://issues.apache.org/jira/browse/OAK-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426761#comment-15426761
]
Michael Dürig edited comment on OAK-4675 at 8/18/16 4:35 PM:
-
Initial patch for {{oak-segment}} [^OAK-4675-02.patch]. There is a problem
however. The test occasionally fails with
{code}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT.randomAccessFileConcurrentReadAndLength(CompactionAndCleanupIT.java:711)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.io.IOException: Stream Closed
at java.io.RandomAccessFile.length(Native Method)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileAccess$Random.length(FileAccess.java:101)
at
org.apache.jackrabbit.oak.plugins.segment.file.TarReader.loadGraph(TarReader.java:893)
at
org.apache.jackrabbit.oak.plugins.segment.file.TarReader.getGraph(TarReader.java:877)
at
org.apache.jackrabbit.oak.plugins.segment.file.TarReader.calculateForwardReferences(TarReader.java:721)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.includeForwardReferences(FileStore.java:948)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.cleanup(FileStore.java:878)
at
org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT$13.call(CompactionAndCleanupIT.java:690)
at
org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT$13.call(CompactionAndCleanupIT.java:687)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
{code}
This is caused by concurrent calls to {{FileStore.cleanup()}}.
[~dulceanu], I think we should avoid concurrent calls to
{{FileStore.cleanup()}} and also {{FileStore.compact()}} (and also make them
mutually exclusive). This deserves an issue in its own. Once we have this we
need to check whether {{randomAccessFileConcurrentReadAndLength()}} still
reproduces this issue and otherwise try to rework it.
Again I suggest we start with fixing this in {{oak-segment-tar}} followed by
{{oak-segment}} followed by the branches.
was (Author: mduerig):
Initial patch for [^OAK-4675-02.patch] {{oak-segment}}. There is a problem
however. The test occasionally fails with
{code}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
[jira] [Updated] (OAK-4675) SNFE thrown while testing FileStore.cleanup() running concurrently with writes
[
https://issues.apache.org/jira/browse/OAK-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4675:
---
Attachment: OAK-4675-02.patch
Initial patch for [^OAK-4675-02.patch] {{oak-segment}}. There is a problem
however. The test occasionally fails with
{code}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT.randomAccessFileConcurrentReadAndLength(CompactionAndCleanupIT.java:711)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.io.IOException: Stream Closed
at java.io.RandomAccessFile.length(Native Method)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileAccess$Random.length(FileAccess.java:101)
at
org.apache.jackrabbit.oak.plugins.segment.file.TarReader.loadGraph(TarReader.java:893)
at
org.apache.jackrabbit.oak.plugins.segment.file.TarReader.getGraph(TarReader.java:877)
at
org.apache.jackrabbit.oak.plugins.segment.file.TarReader.calculateForwardReferences(TarReader.java:721)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.includeForwardReferences(FileStore.java:948)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.cleanup(FileStore.java:878)
at
org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT$13.call(CompactionAndCleanupIT.java:690)
at
org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT$13.call(CompactionAndCleanupIT.java:687)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
{code}
This is caused by concurrent calls to {{FileStore.cleanup()}}.
[~dulceanu], I think we should avoid concurrent calls to
{{FileStore.cleanup()}} and also {{FileStore.compact()}} (and also make them
mutually exclusive). This deserves an issue in its own. Once we have this we
need to check whether {{randomAccessFileConcurrentReadAndLength()}} still
reproduces this issue and otherwise try to rework it.
Again I suggest we start with fixing this in {{oak-segment-tar}} followed by
{{oak-segment}} followed by the branches.
> SNFE thrown while testing FileStore.cleanup() running concurrently with writes
> --
>
> Key: OAK-4675
> URL: https://issues.apache.org/jira/browse/OAK-4675
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar,
[jira] [Commented] (OAK-4618) Align GCMonitorMBean MBean with new generation based GC
[
https://issues.apache.org/jira/browse/OAK-4618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426665#comment-15426665
]
Michael Dürig commented on OAK-4618:
I think we should go deeper than this and also remove the associated
{{TimeSeriesRecorder}} instances. And while we are at it, it seems like
{{gcCount}} is not used anywhere. We should either remove it or expose it as a
simple counter of the number of gc cycles.
> Align GCMonitorMBean MBean with new generation based GC
> ---
>
> Key: OAK-4618
> URL: https://issues.apache.org/jira/browse/OAK-4618
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Andrei Dulceanu
> Labels: production
> Fix For: Segment Tar 0.0.12
>
> Attachments: OAK-4618-01.patch
>
>
> The {{GCMonitorMBean}} MBean still dates back to the old {{oak-segment}}. We
> need to review its endpoints and only keep those that make sense for
> {{oak-segment-tar}}, adapt the others as necessary any add further
> functionality as required.
> Specifically I think we should get rid of the time series for
> {{getRepositorySize()}} and {{getReclaimedSize()}}.
> Also the name {{getRepositorySize()}} is confusing and we should change it.
> It leads callers to think it would return current size of the repository
> opposed to the size it had after the last cleanup. (There is
> {{FileStoreStatsMBean.getRepositorySize()}} for the latter.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3753) Test failure: HeavyWriteIT
[
https://issues.apache.org/jira/browse/OAK-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426643#comment-15426643
]
Michael Dürig commented on OAK-3753:
This looks more and more like a duplicate of OAK-4675. Porting the test
[~dulceanu] to {{oak-segment}} I am able to reproduce "Invalid segment format"
error:
{code}
Caused by: java.lang.IllegalStateException: Invalid segment format. Dumping
segment b3307395-6b84-4035-af39-52b487f817e7
64 61 74 61 30 30 30 30 30 61 2E 74 61 72 2E 69 data0a.tar.i
0010 64 78 00 00 00 00 00 00 00 00 00 00 00 00 00 00 dx..
0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0060 00 00 00 00 30 30 30 30 34 30 30 00 30 30 30 30 400.
{code}
> Test failure: HeavyWriteIT
> --
>
> Key: OAK-3753
> URL: https://issues.apache.org/jira/browse/OAK-3753
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
> Environment: Jenkins, Ubuntu:
> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Critical
> Labels: ci, jenkins
> Fix For: 1.6
>
> Attachments: build-1462283631.log.gz, build-1463134483.log.gz,
> build-1467622598.log.gz, build-1467624994.log.gz, build-1468934799.log.gz,
> build-1470736039.log.gz
>
>
> {{org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT}} failed on Jenkins:
> {noformat}
> heavyWrite[usePersistedMap:
> false](org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT) Time elapsed:
> 106.519 sec <<< ERROR!
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:134)
> at
> org.apache.jackrabbit.oak.plugins.segment.Segment.(Segment.java:214)
> at
> org.apache.jackrabbit.oak.plugins.segment.Segment.(Segment.java:198)
> at
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:1177)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.readSegment(SegmentTracker.java:224)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:149)
> at
> org.apache.jackrabbit.oak.plugins.segment.RecordId.getSegment(RecordId.java:88)
> at
> org.apache.jackrabbit.oak.plugins.segment.Segment.readTemplate(Segment.java:506)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getTemplate(SegmentNodeState.java:79)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getChildNode(SegmentNodeState.java:381)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$UnconnectedHead.update(MemoryNodeBuilder.java:651)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$ConnectedHead.update(MemoryNodeBuilder.java:729)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.head(MemoryNodeBuilder.java:171)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.access$300(MemoryNodeBuilder.java:88)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$UnconnectedHead.update(MemoryNodeBuilder.java:650)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$ConnectedHead.update(MemoryNodeBuilder.java:729)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.head(MemoryNodeBuilder.java:171)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.exists(MemoryNodeBuilder.java:273)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setProperty(MemoryNodeBuilder.java:506)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setProperty(MemoryNodeBuilder.java:515)
> at
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.createProperties(HeavyWriteIT.java:156)
> at
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.createNodes(HeavyWriteIT.java:148)
> at
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.createNodes(HeavyWriteIT.java:149)
> at
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.heavyWrite(HeavyWriteIT.java:129)
> {noformat}
> Seen at build 597
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4675) SNFE thrown while testing FileStore.cleanup() running concurrently with writes
[
https://issues.apache.org/jira/browse/OAK-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426625#comment-15426625
]
Michael Dürig commented on OAK-4675:
Fixed for {{oak-segment-tar}} at
http://svn.apache.org/viewvc?rev=1756809=rev
> SNFE thrown while testing FileStore.cleanup() running concurrently with writes
> --
>
> Key: OAK-4675
> URL: https://issues.apache.org/jira/browse/OAK-4675
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar, segmentmk
>Affects Versions: 1.0.32, 1.4.6, 1.2.18, Segment Tar 0.0.8
>Reporter: Andrei Dulceanu
>Assignee: Andrei Dulceanu
> Labels: cleanup, gc
> Fix For: Segment Tar 0.0.10
>
> Attachments: OAK-4675-01.patch, test-case.patch
>
>
> {{SegmentNotFoundException}} is thrown from time to time in the following
> scenario: plenty of concurrent writes (each creating a {{625 bytes}} blob)
> interrupted by a cleanup.
> Stack trace (including some debugging statements added by me):
> {code:java}
> Pre cleanup readers: []
> Before cleanup readers: [/Users/dulceanu/work/test-repo/data0a.tar]
> Initial size: 357.4 kB
> After cleanup readers: [/Users/dulceanu/work/test-repo/data0a.tar]
> After cleanup size: 357.4 kB
> Final size: 361.0 kB
> Exception in thread "pool-5-thread-74"
> org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Cannot copy
> record from a generation that has been gc'ed already
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1207)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1096)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1074)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:987)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.access$700(SegmentWriter.java:379)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$8.execute(SegmentWriter.java:337)
> at
> org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.execute(SegmentBufferWriterPool.java:105)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter.writeNode(SegmentWriter.java:334)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:111)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:550)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:571)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:627)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore.merge(SegmentNodeStore.java:287)
> at
> org.apache.jackrabbit.oak.segment.CompactionAndCleanupIT$1.run(CompactionAndCleanupIT.java:961)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.jackrabbit.oak.segment.SegmentNotFoundException:
> Segment 4fb637cc-5013-4925-ab13-0629c4406481 not found
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1341)
> at
> org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:123)
> at
> org.apache.jackrabbit.oak.segment.RecordId.getSegment(RecordId.java:94)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1199)
> ... 18 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IllegalStateException: Invalid segment format. Dumping segment
> 4fb637cc-5013-4925-ab13-0629c4406481
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0070 00 00 00 00 00 00 00 00 00 00 00 00
[jira] [Updated] (OAK-4618) Align GCMonitorMBean MBean with new generation based GC
[
https://issues.apache.org/jira/browse/OAK-4618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrei Dulceanu updated OAK-4618:
-
Attachment: OAK-4618-01.patch
Attached patch removing the two methods discussed.
[~mduerig] Could you have a look at it?
> Align GCMonitorMBean MBean with new generation based GC
> ---
>
> Key: OAK-4618
> URL: https://issues.apache.org/jira/browse/OAK-4618
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Andrei Dulceanu
> Labels: production
> Fix For: Segment Tar 0.0.12
>
> Attachments: OAK-4618-01.patch
>
>
> The {{GCMonitorMBean}} MBean still dates back to the old {{oak-segment}}. We
> need to review its endpoints and only keep those that make sense for
> {{oak-segment-tar}}, adapt the others as necessary any add further
> functionality as required.
> Specifically I think we should get rid of the time series for
> {{getRepositorySize()}} and {{getReclaimedSize()}}.
> Also the name {{getRepositorySize()}} is confusing and we should change it.
> It leads callers to think it would return current size of the repository
> opposed to the size it had after the last cleanup. (There is
> {{FileStoreStatsMBean.getRepositorySize()}} for the latter.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4618) Align GCMonitorMBean MBean with new generation based GC
[
https://issues.apache.org/jira/browse/OAK-4618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426592#comment-15426592
]
Michael Dürig commented on OAK-4618:
+1
> Align GCMonitorMBean MBean with new generation based GC
> ---
>
> Key: OAK-4618
> URL: https://issues.apache.org/jira/browse/OAK-4618
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Andrei Dulceanu
> Labels: production
> Fix For: Segment Tar 0.0.12
>
>
> The {{GCMonitorMBean}} MBean still dates back to the old {{oak-segment}}. We
> need to review its endpoints and only keep those that make sense for
> {{oak-segment-tar}}, adapt the others as necessary any add further
> functionality as required.
> Specifically I think we should get rid of the time series for
> {{getRepositorySize()}} and {{getReclaimedSize()}}.
> Also the name {{getRepositorySize()}} is confusing and we should change it.
> It leads callers to think it would return current size of the repository
> opposed to the size it had after the last cleanup. (There is
> {{FileStoreStatsMBean.getRepositorySize()}} for the latter.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4138) Decouple revision cleanup from the flush thread
[
https://issues.apache.org/jira/browse/OAK-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426586#comment-15426586
]
Michael Dürig commented on OAK-4138:
There is to parts to it:
* "Decouple revision cleanup from the flush thread": currently cleanup is
executed on the flush thread. This is problematic for the reasons given in this
issue's description. I would opt for putting cleanup into its own thread.
* "skip flush if already in progress": what I meant here is that instead of
synchronising removal of the old tar files on {{FileStore.pendingRemove}}, we
can just skip removal if already in progress.
> Decouple revision cleanup from the flush thread
> ---
>
> Key: OAK-4138
> URL: https://issues.apache.org/jira/browse/OAK-4138
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Andrei Dulceanu
> Labels: resilience
> Fix For: Segment Tar 0.0.12
>
>
> I suggest we decouple revision cleanup from the flush thread. With large
> repositories where cleanup can take several minutes to complete it blocks the
> flush thread from updating the journal and the persisted head thus resulting
> in larger then necessary data loss in case of a crash.
> /cc [~alex.parvulescu]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4679) Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344
[ https://issues.apache.org/jira/browse/OAK-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426568#comment-15426568 ] Dominique Jäggi commented on OAK-4679: -- backport of OAK-4087 in 1.2 branch done in r1756800. dependencies: - depends on OAK-4384 - depends on OAK-4385 > Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344 > -- > > Key: OAK-4679 > URL: https://issues.apache.org/jira/browse/OAK-4679 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: auth-external, auth-ldap, core >Reporter: Dominique Jäggi >Assignee: Dominique Jäggi > Fix For: 1.4.7, 1.2.19 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-4679) Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344
[ https://issues.apache.org/jira/browse/OAK-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominique Jäggi resolved OAK-4679. -- Resolution: Fixed > Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344 > -- > > Key: OAK-4679 > URL: https://issues.apache.org/jira/browse/OAK-4679 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: auth-external, auth-ldap, core >Reporter: Dominique Jäggi >Assignee: Dominique Jäggi > Fix For: 1.4.7, 1.2.19 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4385) Benchmarks: proper init of ExternalPrincipalConfiguration with dynamicMembership
[
https://issues.apache.org/jira/browse/OAK-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4385:
-
Fix Version/s: 1.2.19
1.4.7
> Benchmarks: proper init of ExternalPrincipalConfiguration with
> dynamicMembership
>
>
> Key: OAK-4385
> URL: https://issues.apache.org/jira/browse/OAK-4385
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-external, run
>Reporter: angela
>Assignee: angela
> Fix For: 1.5.3, 1.4.7, 1.2.19
>
>
> in order to produce realistic results the {{AbstractExternalTest}} benchmark
> base needs to initialize the {{ExternalPrincipalConfiguration}} such that
> upon login the {{ExternalGroupPrincipalProvider}} is enabled.
> And while doing so: the {{BenchmarkRunner}} currently always sets
> dynamicMembeship to false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4384) Benchmarks: add support 'automembership' config option
[ https://issues.apache.org/jira/browse/OAK-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominique Jäggi updated OAK-4384: - Fix Version/s: 1.4.7 > Benchmarks: add support 'automembership' config option > -- > > Key: OAK-4384 > URL: https://issues.apache.org/jira/browse/OAK-4384 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: angela >Assignee: angela > Fix For: 1.5.3, 1.4.7, 1.2.19 > > > - add 'automembership' option to the benchmark runner > - extend auth-external benchmark constructors with automembership list > - create groups listed in automembership option before running the suite -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4384) Benchmarks: add support 'automembership' config option
[ https://issues.apache.org/jira/browse/OAK-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominique Jäggi updated OAK-4384: - Fix Version/s: 1.2.19 > Benchmarks: add support 'automembership' config option > -- > > Key: OAK-4384 > URL: https://issues.apache.org/jira/browse/OAK-4384 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: angela >Assignee: angela > Fix For: 1.5.3, 1.4.7, 1.2.19 > > > - add 'automembership' option to the benchmark runner > - extend auth-external benchmark constructors with automembership list > - create groups listed in automembership option before running the suite -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4679) Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344
[ https://issues.apache.org/jira/browse/OAK-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426563#comment-15426563 ] Dominique Jäggi commented on OAK-4679: -- backport of OAK-4344 completed via OAK-4678 > Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344 > -- > > Key: OAK-4679 > URL: https://issues.apache.org/jira/browse/OAK-4679 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: auth-external, auth-ldap, core >Reporter: Dominique Jäggi >Assignee: Dominique Jäggi > Fix For: 1.4.7, 1.2.19 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4679) Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344
[ https://issues.apache.org/jira/browse/OAK-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426540#comment-15426540 ] Dominique Jäggi commented on OAK-4679: -- backport of OAK-4087 in 1.4 branch done in r1756792. dependencies: - depends on OAK-4364 - depends on OAK-4384 - depends on OAK-4385 > Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344 > -- > > Key: OAK-4679 > URL: https://issues.apache.org/jira/browse/OAK-4679 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: auth-external, auth-ldap, core >Reporter: Dominique Jäggi >Assignee: Dominique Jäggi > Fix For: 1.4.7, 1.2.19 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4682) ConcurrentModificationException in JournalEntry.TreeNode
[
https://issues.apache.org/jira/browse/OAK-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426527#comment-15426527
]
Marcel Reutegger commented on OAK-4682:
---
The modification is caused by a concurrent commit updating
{{DocumentNodeStore#changes}}.
> ConcurrentModificationException in JournalEntry.TreeNode
>
>
> Key: OAK-4682
> URL: https://issues.apache.org/jira/browse/OAK-4682
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, documentmk
>Affects Versions: 1.5.6
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
> Fix For: 1.6
>
>
> The JournalDiffLoader introduced in OAK-4528 sometimes causes a
> ConcurrentModificationException in JournalEntry.TreeNode:
> {noformat}
> java.util.ConcurrentModificationException: null
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
> at java.util.HashMap$EntryIterator.next(HashMap.java:1463)
> at java.util.HashMap$EntryIterator.next(HashMap.java:1461)
> at
> org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:503)
> at
> org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
> at
> org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
> at
> org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
> at
> org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
> at
> org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
> at
> org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
> at
> org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
> at
> org.apache.jackrabbit.oak.plugins.document.JournalEntry.addTo(JournalEntry.java:312)
> at
> org.apache.jackrabbit.oak.plugins.document.JournalDiffLoader.readTrunkChanges(JournalDiffLoader.java:150)
> at
> org.apache.jackrabbit.oak.plugins.document.JournalDiffLoader.call(JournalDiffLoader.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-4682) ConcurrentModificationException in JournalEntry.TreeNode
Marcel Reutegger created OAK-4682:
-
Summary: ConcurrentModificationException in JournalEntry.TreeNode
Key: OAK-4682
URL: https://issues.apache.org/jira/browse/OAK-4682
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, documentmk
Affects Versions: 1.5.6
Reporter: Marcel Reutegger
Assignee: Marcel Reutegger
Fix For: 1.6
The JournalDiffLoader introduced in OAK-4528 sometimes causes a
ConcurrentModificationException in JournalEntry.TreeNode:
{noformat}
java.util.ConcurrentModificationException: null
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
at java.util.HashMap$EntryIterator.next(HashMap.java:1463)
at java.util.HashMap$EntryIterator.next(HashMap.java:1461)
at
org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:503)
at
org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
at
org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
at
org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
at
org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
at
org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
at
org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
at
org.apache.jackrabbit.oak.plugins.document.JournalEntry$TreeNode.accept(JournalEntry.java:504)
at
org.apache.jackrabbit.oak.plugins.document.JournalEntry.addTo(JournalEntry.java:312)
at
org.apache.jackrabbit.oak.plugins.document.JournalDiffLoader.readTrunkChanges(JournalDiffLoader.java:150)
at
org.apache.jackrabbit.oak.plugins.document.JournalDiffLoader.call(JournalDiffLoader.java:74)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4138) Decouple revision cleanup from the flush thread
[
https://issues.apache.org/jira/browse/OAK-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426519#comment-15426519
]
Andrei Dulceanu commented on OAK-4138:
--
Not sure I understood the comment inside {{FileStore.flush}}: "Decouple
revision cleanup from the flush thread: instead of synchronising, skip flush if
already in progress". My take on this one would be to move the cleanup in a
separate thread through another {{PeriodicOperation}}. [~mduerig] WDYT?
//cc [~alexparvulescu]
> Decouple revision cleanup from the flush thread
> ---
>
> Key: OAK-4138
> URL: https://issues.apache.org/jira/browse/OAK-4138
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Andrei Dulceanu
> Labels: resilience
> Fix For: Segment Tar 0.0.12
>
>
> I suggest we decouple revision cleanup from the flush thread. With large
> repositories where cleanup can take several minutes to complete it blocks the
> flush thread from updating the journal and the persisted head thus resulting
> in larger then necessary data loss in case of a crash.
> /cc [~alex.parvulescu]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-4670) Release Oak 1.5.8
[ https://issues.apache.org/jira/browse/OAK-4670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella resolved OAK-4670. --- Resolution: Fixed > Release Oak 1.5.8 > - > > Key: OAK-4670 > URL: https://issues.apache.org/jira/browse/OAK-4670 > Project: Jackrabbit Oak > Issue Type: Task >Reporter: Davide Giannella >Assignee: Davide Giannella > Fix For: 1.6 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3275) DefaultSyncConfig: User membership expiration time not working under some circumstances
[
https://issues.apache.org/jira/browse/OAK-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426459#comment-15426459
]
Konrad Windszus commented on OAK-3275:
--
Please either fix this inconsistency or remove the membership expiration
completely. Otherwise there is a known bug in the implementation which may lead
to issues (because for some configuration values membership is never updated,
contrary to the expectation of every user, please check again the description
of this bug).
> DefaultSyncConfig: User membership expiration time not working under some
> circumstances
> ---
>
> Key: OAK-3275
> URL: https://issues.apache.org/jira/browse/OAK-3275
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Affects Versions: 1.3.5
>Reporter: Konrad Windszus
>Priority: Minor
>
> Currently the user expiration and the user membership expiration can be set
> independently of each other in the OSGi configuration for the
> {{DefaultSyncConfigImpl}}.
> In reality this is not true though:
> Not only can the membership not be updated more often than the other user
> properties (compare with OAK-3274).
> Also the property which is used to mark the last successfull sync is the same
> for both synchronisations
> (https://github.com/apache/jackrabbit-oak/blob/trunk/oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/basic/DefaultSyncContext.java#L433
> and
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/basic/DefaultSyncContext.java#L422).
> That is a problem if e.g. the user expiration time is 10 minutes but the user
> membership expiration time is 1 hour. Then every 10 minutes the property
> {{rep:lastSynced}} would be updated to the current time and the expiration
> check for the membership expiration would never return true
> (https://github.com/apache/jackrabbit-oak/blob/trunk/oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/basic/DefaultSyncContext.java#L433).
> Therefore memberships would never be updated!
> I suggest to completely get rid of user membership expiration time and only
> have one expiration time for both the user properties and the memberships.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4618) Align GCMonitorMBean MBean with new generation based GC
[
https://issues.apache.org/jira/browse/OAK-4618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426455#comment-15426455
]
Andrei Dulceanu commented on OAK-4618:
--
[~mduerig], since OAK-4657 handled most of the concerns raised in the issue
description, I'd propose a patch in which only to get rid of the two time
series, {{getRepositorySize}} and {{getReclaimedSize}}. WDYT?
> Align GCMonitorMBean MBean with new generation based GC
> ---
>
> Key: OAK-4618
> URL: https://issues.apache.org/jira/browse/OAK-4618
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Andrei Dulceanu
> Labels: production
> Fix For: Segment Tar 0.0.12
>
>
> The {{GCMonitorMBean}} MBean still dates back to the old {{oak-segment}}. We
> need to review its endpoints and only keep those that make sense for
> {{oak-segment-tar}}, adapt the others as necessary any add further
> functionality as required.
> Specifically I think we should get rid of the time series for
> {{getRepositorySize()}} and {{getReclaimedSize()}}.
> Also the name {{getRepositorySize()}} is confusing and we should change it.
> It leads callers to think it would return current size of the repository
> opposed to the size it had after the last cleanup. (There is
> {{FileStoreStatsMBean.getRepositorySize()}} for the latter.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4630) Create segment and segment tar directory only when it's missing
[
https://issues.apache.org/jira/browse/OAK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4630.
-
Bulk close for 1.5.8
> Create segment and segment tar directory only when it's missing
> ---
>
> Key: OAK-4630
> URL: https://issues.apache.org/jira/browse/OAK-4630
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: upgrade
>Reporter: Arek Kita
>Priority: Minor
> Labels: patch
> Fix For: 1.6, 1.5.8
>
> Attachments:
> OAK-4630-create-segment-tar-directory-only-when-it-is-missing.patch
>
>
> Currently using oak-upgrade we are dealing with source directory which should
> naturally exist and destination directory which in most of the cases
> (especially for segment-tar) does not exist at all.
> It's awkward to demand from oak-upgrade caller to always create such
> directory as the upgrade process should create everything it needs and it can
> simply do this on its own in majority of simple cases.
> In some sophisticated cases user should be able to create that directory on
> her/his own and mount i.e. external filesystem to it before using
> {{oak-upgrade}} but that's very special case that still should work out of
> the box. In that case we should never demand that the path cannot exist. It
> should work in both cases whether directory is created or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4660) TokenProviderImpl.getTokenParent may return non-existing tree
[
https://issues.apache.org/jira/browse/OAK-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4660.
-
Bulk close for 1.5.8
> TokenProviderImpl.getTokenParent may return non-existing tree
> -
>
> Key: OAK-4660
> URL: https://issues.apache.org/jira/browse/OAK-4660
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Fix For: 1.5.8
>
>
> while writing additional tests for the {{TokenProviderImpl}} I found that
> {{getTokenParent}} in fact return a {{NodeUtil}} referring to a non-existing
> tree as the local {{tokenParent}} field doesn't get reset to {{null}} if (in
> case of {{CommitFailedException}} an attempt is made to retrieve the parent
> that may have been created by another session.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4301) Missing protection for system-maintained rep:externalId
[
https://issues.apache.org/jira/browse/OAK-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4301.
-
Bulk close for 1.5.8
> Missing protection for system-maintained rep:externalId
>
>
> Key: OAK-4301
> URL: https://issues.apache.org/jira/browse/OAK-4301
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Critical
> Labels: security
> Fix For: 1.5.8
>
> Attachments: OAK-4301.patch
>
>
> while working on OAK-4101 i noticed that the current implementation doesn't
> provide any protection for the system maintained property {{rep:externalId}},
> which is intended to be an identifier for a given synchronized user/group
> within an external IDP.
> in other words:
> - the system doesn't assert the uniqueness of a given external-id
> - the external-id properties can be changed using regular JCR API
> up to now i didn't manage to exploit the missing protection with the current
> default implementation but i found that minor (legitimate) changes have the
> potential to turn this into a critical vulnerability.
> therefore I would strongly recommend to change the default implementation
> such that the rep:externalId really becomes system-maintained and prevent any
> unintentional or malicious modification outside of the scope of the
> sync-operations. furthermore uniqueness of this property should be asserted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4661) TokenLoginModule: improve log output
[
https://issues.apache.org/jira/browse/OAK-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4661.
-
Bulk close for 1.5.8
> TokenLoginModule: improve log output
>
>
> Key: OAK-4661
> URL: https://issues.apache.org/jira/browse/OAK-4661
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: angela
>Assignee: angela
>Priority: Trivial
> Fix For: 1.5.8
>
>
> In case {{TokenLoginModule.commit}} fails to create a new token the
> associated log output uses the {{userId}}, which usually has not been set. If
> missing the login name as present on the shared state should be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4566) Multiplexing store support in Lucene Indexes
[
https://issues.apache.org/jira/browse/OAK-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4566.
-
Bulk close for 1.5.8
> Multiplexing store support in Lucene Indexes
>
>
> Key: OAK-4566
> URL: https://issues.apache.org/jira/browse/OAK-4566
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6, 1.5.8
>
>
> Similar to OAK-3403 we need support multiplexing store in Lucene indexes.
> This can be done by having multiple directories under given index definition.
> For e.g. currently the Lucene indexes for an index /oak:index/assetIndex are
> stored in node /oak:index/assetIndex/:dir. For supporting multiple indexes
> which get stored in different stores we can have structure like
> {noformat}
> /oak:index/assetIndex
> + :oak:mount1-dir
> + :dir
> {noformat}
> In above structure index content for paths which are part of mount1 would be
> store in Lucene files stores under {{:oak:mount1-dir}} while the rest go in
> default location {{:dir}
> # *Writing* - At the time of indexing the {{LuceneIndexEditor}} should pick
> up correct writer i.e. one which is mapped to right directory node in
> repository
> # *Reading* - For reading we would have one {{IndexSearcher}} per directory
> node and then query would be executed against both and a joined cursor would
> be made
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4129) Use CredentialsSupport in TokenConfigurationImpl and TokenProviderImpl
[
https://issues.apache.org/jira/browse/OAK-4129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4129.
-
Bulk close for 1.5.8
> Use CredentialsSupport in TokenConfigurationImpl and TokenProviderImpl
> --
>
> Key: OAK-4129
> URL: https://issues.apache.org/jira/browse/OAK-4129
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Fix For: 1.5.8
>
>
> the default token implementation could make use of the {{CredentialsSupport}}
> thus not limiting the functionality to {{SimpleCredentials}}, which could
> still be used as default.
> in order to do so the {{CredentialsSupport}} would need to be extended with
> the ability to setAttibutes (i.e. an attribute map) to credentials
> corresponding to the getAttributes method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4662) Include Commons Lang Math 3 as a dependency in oak-run
[ https://issues.apache.org/jira/browse/OAK-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-4662. - Bulk close for 1.5.8 > Include Commons Lang Math 3 as a dependency in oak-run > -- > > Key: OAK-4662 > URL: https://issues.apache.org/jira/browse/OAK-4662 > Project: Jackrabbit Oak > Issue Type: Bug > Components: run >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: 1.6, 1.5.8 > > > The module oak-run should include Commons Lang Math 3 as a dependency, since > it is currently used by oak-segment-tar to compute various statistics at run > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (OAK-4585) Text extraction: runtime status monitoring
[ https://issues.apache.org/jira/browse/OAK-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-4585. - Bulk close for 1.5.8 > Text extraction: runtime status monitoring > -- > > Key: OAK-4585 > URL: https://issues.apache.org/jira/browse/OAK-4585 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Reporter: Thomas Mueller >Assignee: Thomas Mueller > Fix For: 1.5.8, 1.4.7 > > > Text extraction is sometimes slow, and, in case of a bug in the text > extraction library, can even get stuck in an endless loop. > Right now, it is not easy to understand what is going on, even when looking > at full thread dumps. (Debug) log information about the current state of text > extraction would be nice as well. > I suggest we add debug level logging for the current extracted binary > (content identity). For larger binaries, we can also temporarily set the > thread name (append "Extracting "). That way, it is > relatively easy to see if text extraction is stuck simply looking at full > thread dumps, without having to change the log level and then reindex. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (OAK-4196) EventListener gets removed event for denied node
[ https://issues.apache.org/jira/browse/OAK-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-4196. - Bulk close for 1.5.8 > EventListener gets removed event for denied node > > > Key: OAK-4196 > URL: https://issues.apache.org/jira/browse/OAK-4196 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core, jcr, security >Affects Versions: 1.4 >Reporter: Marcel Reutegger >Priority: Minor > Fix For: 1.5.8 > > > An EventListener of a session that does not have read access to a node may > get a node removed event when the node is removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (OAK-4626) Use oak-upgrade to initialize the DocumentMK local cache nodestore
[ https://issues.apache.org/jira/browse/OAK-4626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-4626. - Bulk close for 1.5.8 > Use oak-upgrade to initialize the DocumentMK local cache nodestore > -- > > Key: OAK-4626 > URL: https://issues.apache.org/jira/browse/OAK-4626 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: documentmk, upgrade >Reporter: Tomek Rękawek >Assignee: Tomek Rękawek > Fix For: 1.6, 1.5.8 > > > OAK-4654 introduces the possibility to use a node store (eg. segment node > store) as a local subtree for the remote document node store. The local store > has to be initialised, preferably with the content already existing in the > main document node store. oak-upgrade, with its --include-paths option can be > used to do that. However, nodes in the local store has to be extended with > some extra properties. The sidegrade process should optionally add these > properties. > The new feature should be available after setting system property > "oak.upgrade.addSecondaryMetadata" to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (OAK-4641) Using same index definition for both async and sync indexing
[
https://issues.apache.org/jira/browse/OAK-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4641.
-
Bulk close for 1.5.8
> Using same index definition for both async and sync indexing
>
>
> Key: OAK-4641
> URL: https://issues.apache.org/jira/browse/OAK-4641
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: query
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6, 1.5.8
>
> Attachments: OAK-4641.patch
>
>
> Currently one can set "async" flag on an index definition to indicate wether
> given index should be effective for synchronous commit or to be used for
> async indexing. For Hybrid Lucene indexing case OAK-4412 we need to have a
> way where same index definition gets used in both.
> As discussed on DL [1] for this to be done following changes need to be done
> # Making {{async}} as a multi value property.
> # Introducing a new IndexEditorProvider interface which can accept an
> IndexingContext instance (OAK-4642). This provides access to
> ## indexing mode - sync or async
> ## index path of the index (see OAK-4152)
> ## CommitInfo (see OAK-4640)
> [1]
> http://mail-archives.apache.org/mod_mbox/jackrabbit-oak-dev/201608.mbox/%3CCAHCW-mk1HzAxy8fk17SzYDcfLYY%3D0HUp93FCYoxpTP37cNgb%2Bg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4642) Provide a way to pass indexing related state to IndexEditorProvider
[
https://issues.apache.org/jira/browse/OAK-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4642.
-
Bulk close for 1.5.8
> Provide a way to pass indexing related state to IndexEditorProvider
> ---
>
> Key: OAK-4642
> URL: https://issues.apache.org/jira/browse/OAK-4642
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6, 1.5.8
>
> Attachments: OAK-4642-o4-v1.patch, OAK-4642-o4-v2.patch
>
>
> An index editor needs access to some more data points like below in the
> indexing process
> # reindexing - Currently the index implementation use some heuristic like
> check before root state being empty to determine if they are running in
> reindexing mode
> # indexing mode - sync or async
> # index path of the index (see OAK-4152)
> # CommitInfo (see OAK-4640)
> For this we need to provide a way to pass these data points from indexing
> flow. We have following options
> * O1 - Introduce a new interface which takes an {{IndexingContext}} instance
> which provide access to such datapoints. This would require some broader
> change
> ** Whereever the IndexEditorProvider is invoked it would need to check if the
> instance implements new interface. If yes then new method needs to be used
> * O2 - Here we can introduce such data points as part of callback interface.
> With this we would need to implement such methods in places where code
> constructs the callback
> * O3 - Make a backward incompatible change and just modify the existing
> interface and adapt the various implementation
> * O4 - Similar to O2 but here instead of modifying the existing
> {{IndexUpdateCallback}} we can introduce a new interface
> {{ContextualCallback}} which extends {{IndexUpdateCallback}} and provide
> access to {{IndexingContext}}. Editor provider implementation can then check
> if the callback implements this new interface and then cast it and access the
> context. So only those client which are interested in new capability make use
> of this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4640) Provide a way for commit hook to record meta data for a given commit
[
https://issues.apache.org/jira/browse/OAK-4640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4640.
-
Bulk close for 1.5.8
> Provide a way for commit hook to record meta data for a given commit
>
>
> Key: OAK-4640
> URL: https://issues.apache.org/jira/browse/OAK-4640
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6, 1.5.8
>
> Attachments: OAK-4640-v1.patch
>
>
> Currently as part of commit the caller can provide a CommitInfo instance
> which captures some metadata related to commit being performed. Note that
> CommitInfo instance passed to NodeStore is immutable.
> For some usecases we need a way to add some more metadata to on going
> commit from within the CommitHook
> * OAK-4586 - Here we need to record nodetypes of nodes which got modified as
> part of current commit
> * OAK-4412 - Here we want to generate Documents for modified nodestate (per
> index definition) and "attach" it to current commit
> This meta information would later be used by Observer.
> To enable such cases some mutable state needs to be exposed as part of
> CommitInfo which can be used by CommitHooks.
> As per discussion of DL [1] this can be done by adding a new
> {{CommitAttributes}} instance which can be accessed from {{CommitInfo}}
> [1]
> http://mail-archives.apache.org/mod_mbox/jackrabbit-oak-dev/201608.mbox/%3CCAHCW-mk%3DTMQ_q20dPSmdr7Q%3DcR8OfMDdJceDTd-pGs0pk_cK0g%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4628) Using non default dir name causes index directory to be deleted
[
https://issues.apache.org/jira/browse/OAK-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4628.
-
Bulk close for 1.5.8
> Using non default dir name causes index directory to be deleted
> ---
>
> Key: OAK-4628
> URL: https://issues.apache.org/jira/browse/OAK-4628
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.5.7
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6, 1.5.8
>
>
> With changes done in OAK-4607 its possible that when multiple dir names are
> used (so far only default dir is used as there was single index) then the
> whole local index directory can get deleted.
> This happens because {{indexPathVersionMapping}} does not account for dir name
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4633) Multiplexing store support for Node type indexes
[ https://issues.apache.org/jira/browse/OAK-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-4633. - Bulk close for 1.5.8 > Multiplexing store support for Node type indexes > > > Key: OAK-4633 > URL: https://issues.apache.org/jira/browse/OAK-4633 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: query >Reporter: Alex Parvulescu >Assignee: Alex Parvulescu > Fix For: 1.6, 1.5.8 > > > Support for PropertyIndexes was introduced as a part of OAK-3403, but was > incomplete, the query bits for the node type index are not properly wired > with the mount provider. Indexing looks fine, but the info is not used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (OAK-4652) indexName logged in QueryImpl is wrong in case of multiple indexes satisfying the query
[
https://issues.apache.org/jira/browse/OAK-4652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4652.
-
Bulk close for 1.5.8
> indexName logged in QueryImpl is wrong in case of multiple indexes satisfying
> the query
> ---
>
> Key: OAK-4652
> URL: https://issues.apache.org/jira/browse/OAK-4652
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: query
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.6, 1.5.8
>
>
> QueryImpl currently computes the indexName like
> {code}
> for (IndexPlan p : ipList) {
> // TODO limit is after all conditions
> long entryCount = Math.min(maxEntryCount,
> p.getEstimatedEntryCount());
> double c = p.getCostPerExecution() + entryCount *
> p.getCostPerEntry();
> if (c < cost) {
> cost = c;
> if (p.getPlanName() != null) {
> indexName += "[" + p.getPlanName() + "]";
> }
> indexPlan = p;
> }
> }
> {code}
> If there are multiple index plans then indexName contains multiple plan names
> which causes confusion
> {noformat}
>cost for
> lucene-property[/oak:index/keymatchKeywords][/oak:index/newImport] is 786.0
> {noformat}
> Instead of concatenating the indexName should be generated outside
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (OAK-4663) Include Commons Lang Math 3 as a dependency in oak-upgrade
[ https://issues.apache.org/jira/browse/OAK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-4663. - Bulk close for 1.5.8 > Include Commons Lang Math 3 as a dependency in oak-upgrade > -- > > Key: OAK-4663 > URL: https://issues.apache.org/jira/browse/OAK-4663 > Project: Jackrabbit Oak > Issue Type: Bug > Components: upgrade >Reporter: Tomek Rękawek >Assignee: Tomek Rękawek > Fix For: 1.6, 1.5.8 > > > The module oak-upgrade should include Commons Lang Math 3 as a dependency, > since it is currently used by oak-segment-tar to compute various statistics > at run time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (OAK-4466) Incorrect description for "Simple Inheritance with Restrictions" inthe Permission Evaluation Page
[ https://issues.apache.org/jira/browse/OAK-4466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-4466. - Bulk close for 1.5.8 > Incorrect description for "Simple Inheritance with Restrictions" inthe > Permission Evaluation Page > - > > Key: OAK-4466 > URL: https://issues.apache.org/jira/browse/OAK-4466 > Project: Jackrabbit Oak > Issue Type: Documentation > Components: doc >Reporter: Opkar Gill >Assignee: angela > Fix For: 1.5.8 > > > In the page > http://jackrabbit.apache.org/oak/docs/security/permission/evaluation.html > there is a description for "Simple Inheritance with Restrictions" > the text reads: > everyone is cannot read the complete tree defined by /content except for > properties named ‘prop1’ or ‘prop2’ which are explicitly denied by the > restricting entry. > It should be: > everyone can read the complete tree defined by /content except for properties > named ‘prop1’ or ‘prop2’ which are explicitly denied by the restricting entry. > i.e. "is cannot" should be changed to "can" in the above text -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (OAK-4658) Outer join: name(b) and localname(b) can throw a NPE
[
https://issues.apache.org/jira/browse/OAK-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-4658.
-
Bulk close for 1.5.8
> Outer join: name(b) and localname(b) can throw a NPE
>
>
> Key: OAK-4658
> URL: https://issues.apache.org/jira/browse/OAK-4658
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: query
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Fix For: 1.5.8, 1.4.7
>
>
> Outer join queries with a condition of "name(joinSelector) = 'x'" or
> "localname(joinSelector) = 'x'" can result in a NullPointerException. This is
> because the code doesn't take the "no-match" case into account (that the
> joined selector doesn't have a node). Example queries:
> {noformat}
> select a.[jcr:path] from [nt:base] as a
> left outer join [nt:base] as b on ischildnode(b, a)
> where name(b) = 'b'
> select a.[jcr:path] from [nt:base] as a
> left outer join [nt:base] as b on ischildnode(b, a)
> where localname(b) = 'b'
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3274) DefaultSyncConfigImpl: add information to "user.membershipExpTime" about minimum expiration time
[
https://issues.apache.org/jira/browse/OAK-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426430#comment-15426430
]
Konrad Windszus commented on OAK-3274:
--
There is a typo in this patch: "aways" instead of "always".
> DefaultSyncConfigImpl: add information to "user.membershipExpTime" about
> minimum expiration time
>
>
> Key: OAK-3274
> URL: https://issues.apache.org/jira/browse/OAK-3274
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-external
>Affects Versions: 1.3.5
>Reporter: Konrad Windszus
>Priority: Trivial
> Fix For: 1.4, 1.3.16, 1.2.19
>
> Attachments: OAK-3274.patch
>
>
> The {{user.membershipExpTime}} property cannot have a value which is less
> than the value of the {{user.expirationTime}}. Please add this information to
> the OSGi property description. Otherwise it is hard to debug issues here.
> The reason why {{user.expirationTime}} must be less or equal to
> {{user.membershipExpTime}} is in
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/basic/DefaultSyncContext.java#L421.
> Since {{syncMembership}} is only called after the {{user.expirationTime}}
> guard, it cannot be updated more often than the user itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4364) Initial Benchmarks for oak-auth-external
[ https://issues.apache.org/jira/browse/OAK-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominique Jäggi updated OAK-4364: - Fix Version/s: 1.2.19 > Initial Benchmarks for oak-auth-external > > > Key: OAK-4364 > URL: https://issues.apache.org/jira/browse/OAK-4364 > Project: Jackrabbit Oak > Issue Type: Task > Components: auth-external, run >Reporter: angela >Assignee: angela > Fix For: 1.5.3, 1.4.7, 1.2.19 > > > in order to properly measure performance improvements we should have > benchmarks for oak-auth-external module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4248) More tests for the exposed 'basic' package
[
https://issues.apache.org/jira/browse/OAK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4248:
-
Fix Version/s: 1.2.19
> More tests for the exposed 'basic' package
> --
>
> Key: OAK-4248
> URL: https://issues.apache.org/jira/browse/OAK-4248
> Project: Jackrabbit Oak
> Issue Type: Test
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Fix For: 1.5.2, 1.4.7, 1.2.19
>
>
> As {{org.apache.jackrabbit.oak.spi.security.authentication.external.basic}}
> is exported it would be good to have specific tests for the functionality and
> classes provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4231) DefaultSyncContext creates Value of type String for Binary|Inputstream Object
[
https://issues.apache.org/jira/browse/OAK-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4231:
-
Fix Version/s: 1.2.19
> DefaultSyncContext creates Value of type String for Binary|Inputstream Object
> -
>
> Key: OAK-4231
> URL: https://issues.apache.org/jira/browse/OAK-4231
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Fix For: 1.5.2, 1.4.7, 1.2.19
>
>
> while writing tests for the {{DefaultSyncContext}} I found that the type of
> {{Value}} created for a {{Binary}} and {{InputStream}} parameter is
> {{PropertyType.String}} instead of {{PropertyType.Binary}} as I had expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4302) DefaultSyncContextTest contains duplicate test
[
https://issues.apache.org/jira/browse/OAK-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4302:
-
Fix Version/s: 1.2.19
> DefaultSyncContextTest contains duplicate test
> --
>
> Key: OAK-4302
> URL: https://issues.apache.org/jira/browse/OAK-4302
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: Francesco Mari
>Assignee: angela
>Priority: Trivial
> Fix For: 1.5.2, 1.4.7, 1.2.19
>
>
> Some test methods in {{DefaultSyncContextTests}} are the same when compared
> line-by-line:
> * {{testCreateValueFromBytesArray}}, {{testCreateValueFromBinary}} and
> {{testCreateValueFromInputStream}};
> * {{testGetIdentityRefSyncGroup}} and {{testGetIdentityRefSyncUser}}.
> If these methods are actually testing different things because of a lucky
> state of the instance variable and an even luckier running order, they should
> be reworked to be independent. Otherwise, some of them should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4224) DefaultSyncContext.sync(ExternalIdentity) should verify IDP
[
https://issues.apache.org/jira/browse/OAK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4224:
-
Fix Version/s: 1.2.19
> DefaultSyncContext.sync(ExternalIdentity) should verify IDP
> ---
>
> Key: OAK-4224
> URL: https://issues.apache.org/jira/browse/OAK-4224
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Fix For: 1.5.2, 1.4.7, 1.2.19
>
> Attachments: OAK-4224.patch, OAK-4224_2.patch
>
>
> while writing more test for {{DefaultSyncContext}} i realized that the
> implementation of {{sync(ExternalIdentity)}} doesn't verify that the given
> external identity belongs to the same IDP than the one associated with the
> context instance.
> IMHO this would be needed and useful particularly when multiple IDPs are
> combined. also, the {{DefaultSyncContext}} is a public exposed class, I
> would prefer if it would guard against mixing up sync of external identities
> from different sources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4226) Improve testing of DefaultSyncContext
[ https://issues.apache.org/jira/browse/OAK-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominique Jäggi updated OAK-4226: - Fix Version/s: 1.2.19 > Improve testing of DefaultSyncContext > - > > Key: OAK-4226 > URL: https://issues.apache.org/jira/browse/OAK-4226 > Project: Jackrabbit Oak > Issue Type: Test > Components: auth-external >Reporter: angela >Assignee: angela > Fix For: 1.5.2, 1.4.7, 1.2.19 > > > - additional test-coverage > - more tests for edge cases and special scenario (see for example OAK-4224) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4219) ExternalLoginModuleTestBase doesn't remove synced User/Group accounts
[
https://issues.apache.org/jira/browse/OAK-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4219:
-
Fix Version/s: 1.2.19
> ExternalLoginModuleTestBase doesn't remove synced User/Group accounts
> -
>
> Key: OAK-4219
> URL: https://issues.apache.org/jira/browse/OAK-4219
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
> Fix For: 1.5.2, 1.4.7, 1.2.19
>
>
> Looking at the {{ExternalLoginModuleTestBase}} I got the impression that the
> cleanup of user/group accounts is effectively broken.
> The current code looks as follows:
> - in the before method the list of existing authorizables is collected
> - in the after method the following code is executed:
> {code}
> UserManager userManager = getUserManager(root);
> Iterator iter =
> userManager.findAuthorizables("jcr:primaryType", null);
> while (iter.hasNext()) {
> ids.remove(iter.next().getID());
> }
> for (String id : ids) {
> Authorizable a = userManager.getAuthorizable(id);
> if (a != null) {
> a.remove();
> }
> }
> {code}
> I might be totally mistaken but IMHO looks troublesome. Introducing an
> assertion after this verifying that the user with the external-test-id has
> been removed will actually fail... and I assume that this would have been the
> expected outcome.
> So, I would have expected the after-method to remove all users/groups
> _except_ those gather in the before-call, which would be considered built-in
> to the repository.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4267) SyncedIdentity for foreign authorizable always has isGroup set to false
[
https://issues.apache.org/jira/browse/OAK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4267:
-
Fix Version/s: 1.2.19
> SyncedIdentity for foreign authorizable always has isGroup set to false
> ---
>
> Key: OAK-4267
> URL: https://issues.apache.org/jira/browse/OAK-4267
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Trivial
> Fix For: 1.5.7, 1.4.7, 1.2.19
>
>
> Calling {{DefaultSyncContext.sync(String userId)}} for an authorizable that
> exists but belongs to a different IDP, will return a result with
> {{SyncedIdentity}} that always returns {{false}} for
> {{SyncedIdentity.isGroup}}.
> I guess this is a copy-paste mistake from the result generated for a {{null}}
> authorizable above. The fix would be trivial by passing
> {{Authorizable.isGroup}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3518) Consistently add annotations to DefaultSync* classes
[
https://issues.apache.org/jira/browse/OAK-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-3518:
-
Fix Version/s: 1.2.19
> Consistently add annotations to DefaultSync* classes
>
>
> Key: OAK-3518
> URL: https://issues.apache.org/jira/browse/OAK-3518
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Trivial
> Fix For: 1.4, 1.3.9, 1.2.19
>
> Attachments: OAK-3518.patch
>
>
> the public classes in
> {{org.apache.jackrabbit.oak.spi.security.authentication.external.basic}}
> package should have consistent annotations such {{@Nonnull}},
> {{@CheckForNull}} (etc) as well as {{@Override}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3522) DefaultSyncContext exposes internal path-utility method
[
https://issues.apache.org/jira/browse/OAK-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-3522:
-
Fix Version/s: 1.2.19
> DefaultSyncContext exposes internal path-utility method
> ---
>
> Key: OAK-3522
> URL: https://issues.apache.org/jira/browse/OAK-3522
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
> Fix For: 1.4, 1.3.11, 1.2.19
>
> Attachments: OAK-3522.patch, OAK-3522_2.patch, OAK-3522_3.patch
>
>
> the {{DefaultSyncContext}} defines a public static method {{joinPaths}},
> which IMHO should not be exposed as it is unrelated to the functionality
> defined by the {{DefaultSyncContext}}.
> i would therefore suggest to make it private (and increase the package
> version to 2.0.0)... we can then still decide to move this utility to
> {{PathUtils}} in oak-commons.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3523) DefaultSyncContext catches ClassCastException
[
https://issues.apache.org/jira/browse/OAK-3523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-3523:
-
Fix Version/s: 1.2.19
> DefaultSyncContext catches ClassCastException
> -
>
> Key: OAK-3523
> URL: https://issues.apache.org/jira/browse/OAK-3523
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
> Labels: technical_debt
> Fix For: 1.4, 1.3.9, 1.2.19
>
> Attachments: OAK-3523_classcast_only.patch
>
>
> the group and group-member sync code in the {{DefaultSyncContext}} twice
> catches {{ClassCastException}} and swallows exception situations, where a
> user is found, when actually a {{Group}} was expected.
> i would suggest to
> - explicitly test the assumption wrt {{ExternalIdentity}} being a group
> (instead of waiting for the exception)
> - make use of {{UserManager.getAuthorizable(String, Class) to explicitly
> retrieve a Group with a given ID, when this is actually expected. This method
> will throws an {{AuthorizableTypeException}} if there exists a {{User}} with
> that ID as thus properly raise the unexpected behavior instead of swallowing
> with a log-warning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3563) Improve DefaultSyncContext
[
https://issues.apache.org/jira/browse/OAK-3563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-3563:
-
Fix Version/s: 1.2.19
> Improve DefaultSyncContext
> --
>
> Key: OAK-3563
> URL: https://issues.apache.org/jira/browse/OAK-3563
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-external
>Reporter: angela
>Assignee: angela
> Labels: technical_debt
> Fix For: 1.5.0, 1.4.7, 1.2.19
>
> Attachments: OAK-3523_4.patch
>
>
> cloning from OAK-3523 in order to not mix the {{ClassCastException}} issue
> with additional improvements i suggested in the patch, which include e.g.
> minor refactoring to remove code duplication and improve overall readability
> (IMO).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3003) Improve login performance with huge group membership
[
https://issues.apache.org/jira/browse/OAK-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-3003:
-
Fix Version/s: 1.2.19
> Improve login performance with huge group membership
>
>
> Key: OAK-3003
> URL: https://issues.apache.org/jira/browse/OAK-3003
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: angela
>Assignee: angela
> Labels: performance
> Fix For: 1.4, 1.3.4, 1.2.19
>
> Attachments: OAK-3003.patch, OAK-3003_2.patch, OAK-3003_3.patch,
> OAK-3003_4.patch, group_cache_in_userprincipalprovider.txt
>
>
> As visible when running {{LoginWithMembershipTest}} default login performance
> (which uses the {{o.a.j.oak.security.principal.PrincipalProviderImpl}} to
> lookup the complete set of principals) suffers when a given user is member of
> a huge number of groups (see also OAK-2690 for benchmark data).
> While using dynamic group membership (and thus a custom {{PrincipalProvider}}
> would be the preferable way to deal with this, we still want to optimize
> {{PrincipalProvider.getPrincipals(String userId}} for the default
> implementation.
> With the introduction of a less generic implementation in OAK-2690, we might
> be able to come up with an optimization that makes use of the very
> implementation details of the user management while at the same time being
> able to properly secure it as we won't need to extend the public API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3211) Document External Identity Management
[ https://issues.apache.org/jira/browse/OAK-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominique Jäggi updated OAK-3211: - Fix Version/s: 1.2.19 > Document External Identity Management > - > > Key: OAK-3211 > URL: https://issues.apache.org/jira/browse/OAK-3211 > Project: Jackrabbit Oak > Issue Type: Task > Components: auth-external, doc >Reporter: angela >Assignee: angela > Fix For: 1.5.6, 1.4.7, 1.2.19 > > > we still lack proper documentation of the external identity management. there > is a initial stub at > http://svn.apache.org/repos/asf/jackrabbit/oak/trunk/oak-doc/src/site/markdown/security/authentication/identitymanagement.md. > [~djaeggi], would this be something you could (and wanted) to work on? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3274) DefaultSyncConfigImpl: add information to "user.membershipExpTime" about minimum expiration time
[
https://issues.apache.org/jira/browse/OAK-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-3274:
-
Fix Version/s: 1.2.19
> DefaultSyncConfigImpl: add information to "user.membershipExpTime" about
> minimum expiration time
>
>
> Key: OAK-3274
> URL: https://issues.apache.org/jira/browse/OAK-3274
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-external
>Affects Versions: 1.3.5
>Reporter: Konrad Windszus
>Priority: Trivial
> Fix For: 1.4, 1.3.16, 1.2.19
>
> Attachments: OAK-3274.patch
>
>
> The {{user.membershipExpTime}} property cannot have a value which is less
> than the value of the {{user.expirationTime}}. Please add this information to
> the OSGi property description. Otherwise it is hard to debug issues here.
> The reason why {{user.expirationTime}} must be less or equal to
> {{user.membershipExpTime}} is in
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/basic/DefaultSyncContext.java#L421.
> Since {{syncMembership}} is only called after the {{user.expirationTime}}
> guard, it cannot be updated more often than the user itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4679) Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344
[ https://issues.apache.org/jira/browse/OAK-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426417#comment-15426417 ] Dominique Jäggi commented on OAK-4679: -- backport of OAK-4101 in 1.2 branch done in r1756777. dependencies: - depends on OAK-3003 - depends on OAK-3211 - depends on OAK-3274 - depends on OAK-3518 - depends on OAK-3522 - depends on OAK-3523 - depends on OAK-3563 - depends on OAK-4001 - depends on OAK-4219 - depends on OAK-4224 - depends on OAK-4226 - depends on OAK-4231 - depends on OAK-4248 - depends on OAK-4267 - depends on OAK-4302 - depends on OAK-4364 > Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344 > -- > > Key: OAK-4679 > URL: https://issues.apache.org/jira/browse/OAK-4679 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: auth-external, auth-ldap, core >Reporter: Dominique Jäggi >Assignee: Dominique Jäggi > Fix For: 1.4.7, 1.2.19 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4675) SNFE thrown while testing FileStore.cleanup() running concurrently with writes
[
https://issues.apache.org/jira/browse/OAK-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426411#comment-15426411
]
Andrei Dulceanu commented on OAK-4675:
--
BTW, I chose to change {{FileAccess.length()}} signature to {code}public
synchronized int length() throws IOException{code}, aligning it to the other
methods in the class.
[~mduerig] Could you take a look at the provided patch, please?
> SNFE thrown while testing FileStore.cleanup() running concurrently with writes
> --
>
> Key: OAK-4675
> URL: https://issues.apache.org/jira/browse/OAK-4675
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar, segmentmk
>Affects Versions: 1.0.32, 1.4.6, 1.2.18, Segment Tar 0.0.8
>Reporter: Andrei Dulceanu
>Assignee: Andrei Dulceanu
> Labels: cleanup, gc
> Fix For: Segment Tar 0.0.10
>
> Attachments: OAK-4675-01.patch, test-case.patch
>
>
> {{SegmentNotFoundException}} is thrown from time to time in the following
> scenario: plenty of concurrent writes (each creating a {{625 bytes}} blob)
> interrupted by a cleanup.
> Stack trace (including some debugging statements added by me):
> {code:java}
> Pre cleanup readers: []
> Before cleanup readers: [/Users/dulceanu/work/test-repo/data0a.tar]
> Initial size: 357.4 kB
> After cleanup readers: [/Users/dulceanu/work/test-repo/data0a.tar]
> After cleanup size: 357.4 kB
> Final size: 361.0 kB
> Exception in thread "pool-5-thread-74"
> org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Cannot copy
> record from a generation that has been gc'ed already
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1207)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1096)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1074)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:987)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.access$700(SegmentWriter.java:379)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$8.execute(SegmentWriter.java:337)
> at
> org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.execute(SegmentBufferWriterPool.java:105)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter.writeNode(SegmentWriter.java:334)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:111)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:550)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:571)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:627)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore.merge(SegmentNodeStore.java:287)
> at
> org.apache.jackrabbit.oak.segment.CompactionAndCleanupIT$1.run(CompactionAndCleanupIT.java:961)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.jackrabbit.oak.segment.SegmentNotFoundException:
> Segment 4fb637cc-5013-4925-ab13-0629c4406481 not found
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1341)
> at
> org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:123)
> at
> org.apache.jackrabbit.oak.segment.RecordId.getSegment(RecordId.java:94)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1199)
> ... 18 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IllegalStateException: Invalid segment format. Dumping segment
> 4fb637cc-5013-4925-ab13-0629c4406481
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0050 00 00 00 00 00 00 00 00
[jira] [Updated] (OAK-4675) SNFE thrown while testing FileStore.cleanup() running concurrently with writes
[
https://issues.apache.org/jira/browse/OAK-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrei Dulceanu updated OAK-4675:
-
Attachment: OAK-4675-01.patch
I started writing a test which would reproduce on every run the described
behaviour, but this proved a little bit tricky since there were runs in which
everything went perfectly fine.
Previously the race condition was obtained by calling {{FileStore.cleanup()}}
which in turn called {{TarReader.sweep()}} which at some point had a debug
statement in which {{FileAccess.length()}} was called. In another thread
{{SegmentNodeStore.merge()}} triggered
{{SegmentNodeWriter.writeNodeUncached()}} which had a check which caused the
segment generation to be read from the segment, which was loaded via
{{FileAccess.read()}}. Since {{SegmentCache}} was also involved, this prevented
the calls to {{readSegment()}} to go directly to disk to fetch the bytes.
Therefore I chose a different approach trying to concurrently trigger calls to
{{FileAccess.length()}} and {{FileAccess.read()}}. Through
{{FileStore.collectBlobReferences()}} I managed to call
{{TarReader.loadBinaryReferences()}} which perfectly fit the job, since it
contains calls to {{FileAccess.length()}} and {{FileAccess.read()}} in
consecutive lines.
Having lots of threads, some doing cleanup, some writing and some collecting
the references proved the right recipe. It always throws either
{{java.nio.BufferUnderflowException}}, either {{SegmentNotFoundException}}
depending on which task failed, cleanup or reference collection. The important
thing here is that the root cause is the same, only the place where it happens
differs.
Below, the full stack trace obtained before synchronising access to
{{FileAccess.length()}}:
{code}
java.util.concurrent.ExecutionException: java.nio.BufferUnderflowException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.jackrabbit.oak.segment.CompactionAndCleanupIT.randomAccessFileConcurrentReadAndLength(CompactionAndCleanupIT.java:1098)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:498)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:406)
at
org.apache.jackrabbit.oak.segment.file.TarReader.parseBinaryReferences(TarReader.java:1054)
at
org.apache.jackrabbit.oak.segment.file.TarReader.getBinaryReferences(TarReader.java:1000)
at
org.apache.jackrabbit.oak.segment.file.TarReader.collectBlobReferences(TarReader.java:744)
at
org.apache.jackrabbit.oak.segment.file.FileStore.collectBlobReferences(FileStore.java:888)
at
[jira] [Updated] (OAK-4267) SyncedIdentity for foreign authorizable always has isGroup set to false
[
https://issues.apache.org/jira/browse/OAK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4267:
-
Fix Version/s: 1.5.7
> SyncedIdentity for foreign authorizable always has isGroup set to false
> ---
>
> Key: OAK-4267
> URL: https://issues.apache.org/jira/browse/OAK-4267
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Trivial
> Fix For: 1.5.7, 1.4.7
>
>
> Calling {{DefaultSyncContext.sync(String userId)}} for an authorizable that
> exists but belongs to a different IDP, will return a result with
> {{SyncedIdentity}} that always returns {{false}} for
> {{SyncedIdentity.isGroup}}.
> I guess this is a copy-paste mistake from the result generated for a {{null}}
> authorizable above. The fix would be trivial by passing
> {{Authorizable.isGroup}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4302) DefaultSyncContextTest contains duplicate test
[
https://issues.apache.org/jira/browse/OAK-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4302:
-
Fix Version/s: 1.4.7
> DefaultSyncContextTest contains duplicate test
> --
>
> Key: OAK-4302
> URL: https://issues.apache.org/jira/browse/OAK-4302
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: Francesco Mari
>Assignee: angela
>Priority: Trivial
> Fix For: 1.5.2, 1.4.7
>
>
> Some test methods in {{DefaultSyncContextTests}} are the same when compared
> line-by-line:
> * {{testCreateValueFromBytesArray}}, {{testCreateValueFromBinary}} and
> {{testCreateValueFromInputStream}};
> * {{testGetIdentityRefSyncGroup}} and {{testGetIdentityRefSyncUser}}.
> If these methods are actually testing different things because of a lucky
> state of the instance variable and an even luckier running order, they should
> be reworked to be independent. Otherwise, some of them should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4267) SyncedIdentity for foreign authorizable always has isGroup set to false
[
https://issues.apache.org/jira/browse/OAK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4267:
-
Fix Version/s: (was: 1.5.2)
1.4.7
> SyncedIdentity for foreign authorizable always has isGroup set to false
> ---
>
> Key: OAK-4267
> URL: https://issues.apache.org/jira/browse/OAK-4267
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Trivial
> Fix For: 1.5.7, 1.4.7
>
>
> Calling {{DefaultSyncContext.sync(String userId)}} for an authorizable that
> exists but belongs to a different IDP, will return a result with
> {{SyncedIdentity}} that always returns {{false}} for
> {{SyncedIdentity.isGroup}}.
> I guess this is a copy-paste mistake from the result generated for a {{null}}
> authorizable above. The fix would be trivial by passing
> {{Authorizable.isGroup}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4364) Initial Benchmarks for oak-auth-external
[ https://issues.apache.org/jira/browse/OAK-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominique Jäggi updated OAK-4364: - Fix Version/s: 1.4.7 > Initial Benchmarks for oak-auth-external > > > Key: OAK-4364 > URL: https://issues.apache.org/jira/browse/OAK-4364 > Project: Jackrabbit Oak > Issue Type: Task > Components: auth-external, run >Reporter: angela >Assignee: angela > Fix For: 1.5.3, 1.4.7 > > > in order to properly measure performance improvements we should have > benchmarks for oak-auth-external module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4226) Improve testing of DefaultSyncContext
[ https://issues.apache.org/jira/browse/OAK-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominique Jäggi updated OAK-4226: - Fix Version/s: 1.4.7 > Improve testing of DefaultSyncContext > - > > Key: OAK-4226 > URL: https://issues.apache.org/jira/browse/OAK-4226 > Project: Jackrabbit Oak > Issue Type: Test > Components: auth-external >Reporter: angela >Assignee: angela > Fix For: 1.5.2, 1.4.7 > > > - additional test-coverage > - more tests for edge cases and special scenario (see for example OAK-4224) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4219) ExternalLoginModuleTestBase doesn't remove synced User/Group accounts
[
https://issues.apache.org/jira/browse/OAK-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4219:
-
Fix Version/s: 1.4.7
> ExternalLoginModuleTestBase doesn't remove synced User/Group accounts
> -
>
> Key: OAK-4219
> URL: https://issues.apache.org/jira/browse/OAK-4219
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
> Fix For: 1.5.2, 1.4.7
>
>
> Looking at the {{ExternalLoginModuleTestBase}} I got the impression that the
> cleanup of user/group accounts is effectively broken.
> The current code looks as follows:
> - in the before method the list of existing authorizables is collected
> - in the after method the following code is executed:
> {code}
> UserManager userManager = getUserManager(root);
> Iterator iter =
> userManager.findAuthorizables("jcr:primaryType", null);
> while (iter.hasNext()) {
> ids.remove(iter.next().getID());
> }
> for (String id : ids) {
> Authorizable a = userManager.getAuthorizable(id);
> if (a != null) {
> a.remove();
> }
> }
> {code}
> I might be totally mistaken but IMHO looks troublesome. Introducing an
> assertion after this verifying that the user with the external-test-id has
> been removed will actually fail... and I assume that this would have been the
> expected outcome.
> So, I would have expected the after-method to remove all users/groups
> _except_ those gather in the before-call, which would be considered built-in
> to the repository.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4001) ExternalLoginModule: Make max sync attempts configurable
[
https://issues.apache.org/jira/browse/OAK-4001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4001:
-
Fix Version/s: 1.4.7
> ExternalLoginModule: Make max sync attempts configurable
>
>
> Key: OAK-4001
> URL: https://issues.apache.org/jira/browse/OAK-4001
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-external
>Reporter: angela
>Priority: Minor
> Fix For: 1.4.7
>
>
> improvement request reflecting open TODO in
> {{ExternalLoginModule#MAX_SYNC_ATTEMPTS}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4248) More tests for the exposed 'basic' package
[
https://issues.apache.org/jira/browse/OAK-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4248:
-
Fix Version/s: 1.4.7
> More tests for the exposed 'basic' package
> --
>
> Key: OAK-4248
> URL: https://issues.apache.org/jira/browse/OAK-4248
> Project: Jackrabbit Oak
> Issue Type: Test
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Fix For: 1.5.2, 1.4.7
>
>
> As {{org.apache.jackrabbit.oak.spi.security.authentication.external.basic}}
> is exported it would be good to have specific tests for the functionality and
> classes provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4224) DefaultSyncContext.sync(ExternalIdentity) should verify IDP
[
https://issues.apache.org/jira/browse/OAK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominique Jäggi updated OAK-4224:
-
Fix Version/s: 1.4.7
> DefaultSyncContext.sync(ExternalIdentity) should verify IDP
> ---
>
> Key: OAK-4224
> URL: https://issues.apache.org/jira/browse/OAK-4224
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Fix For: 1.5.2, 1.4.7
>
> Attachments: OAK-4224.patch, OAK-4224_2.patch
>
>
> while writing more test for {{DefaultSyncContext}} i realized that the
> implementation of {{sync(ExternalIdentity)}} doesn't verify that the given
> external identity belongs to the same IDP than the one associated with the
> context instance.
> IMHO this would be needed and useful particularly when multiple IDPs are
> combined. also, the {{DefaultSyncContext}} is a public exposed class, I
> would prefer if it would guard against mixing up sync of external identities
> from different sources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-4679) Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344
[ https://issues.apache.org/jira/browse/OAK-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426205#comment-15426205 ] Dominique Jäggi edited comment on OAK-4679 at 8/18/16 10:08 AM: backport of OAK-4101 in 1.4 branch done in r1756752. dependencies: - depends on OAK-3211 - depends on OAK-3563 - depends on OAK-4001 - depends on OAK-4219 - depends on OAK-4224 - depends on OAK-4226 - depends on OAK-4231 - depends on OAK-4248 - depends on OAK-4267 - depends on OAK-4302 - depends on OAK-4364 was (Author: dominique.jaeggi): backport of OAK-4101 in 1.4 branch done in r1756752. dependencies: - depends on OAK-3211 - depends on OAK-3274 - depends on OAK-3518 - depends on OAK-3522 - depends on OAK-3523 - depends on OAK-3563 - depends on OAK-4001 - depends on OAK-4219 - depends on OAK-4224 - depends on OAK-4226 - depends on OAK-4231 - depends on OAK-4248 - depends on OAK-4267 - depends on OAK-4302 - depends on OAK-4364 > Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344 > -- > > Key: OAK-4679 > URL: https://issues.apache.org/jira/browse/OAK-4679 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: auth-external, auth-ldap, core >Reporter: Dominique Jäggi >Assignee: Dominique Jäggi > Fix For: 1.4.7, 1.2.19 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3211) Document External Identity Management
[ https://issues.apache.org/jira/browse/OAK-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominique Jäggi updated OAK-3211: - Fix Version/s: 1.4.7 > Document External Identity Management > - > > Key: OAK-3211 > URL: https://issues.apache.org/jira/browse/OAK-3211 > Project: Jackrabbit Oak > Issue Type: Task > Components: auth-external, doc >Reporter: angela >Assignee: angela > Fix For: 1.5.6, 1.4.7 > > > we still lack proper documentation of the external identity management. there > is a initial stub at > http://svn.apache.org/repos/asf/jackrabbit/oak/trunk/oak-doc/src/site/markdown/security/authentication/identitymanagement.md. > [~djaeggi], would this be something you could (and wanted) to work on? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4679) Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344
[ https://issues.apache.org/jira/browse/OAK-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426205#comment-15426205 ] Dominique Jäggi commented on OAK-4679: -- backport of OAK-4101 in 1.4 branch done in r1756752. dependencies: - depends on OAK-3211 - depends on OAK-3274 - depends on OAK-3518 - depends on OAK-3522 - depends on OAK-3523 - depends on OAK-3563 - depends on OAK-4001 - depends on OAK-4219 - depends on OAK-4224 - depends on OAK-4226 - depends on OAK-4231 - depends on OAK-4248 - depends on OAK-4267 - depends on OAK-4302 - depends on OAK-4364 > Backport OAK-4119, OAK-4101, OAK-4087 and OAK-4344 > -- > > Key: OAK-4679 > URL: https://issues.apache.org/jira/browse/OAK-4679 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: auth-external, auth-ldap, core >Reporter: Dominique Jäggi >Assignee: Dominique Jäggi > Fix For: 1.4.7, 1.2.19 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4681) Automatically convert *all* "or" queries to "union" for SQL-2
[ https://issues.apache.org/jira/browse/OAK-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426174#comment-15426174 ] Vikas Saurabh commented on OAK-4681: Just to mention, breaking query into UNION would imply execution of 2 separate queries where the merge is done in QE. While that's probably completely ok if each of the sub-queries are answered by prop indices. But, a single lucene index could very well be handling both OR clauses and splitting into UNION might end up choosing entirely different index (prop or lucene). Moreover, with OR clauses lucene automatically ranks more clause-matches above... this is usually a good ordering... splitting would remove that too. > Automatically convert *all* "or" queries to "union" for SQL-2 > - > > Key: OAK-4681 > URL: https://issues.apache.org/jira/browse/OAK-4681 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: query >Reporter: Thomas Mueller > Fix For: 1.6 > > > Currently, in OAK-1617, simple SQL-2 queries that contain "or" are converted > to "union" if the cost is lower. However, more complex queries are not > converted, see AndImpl.java, convertToUnion(), "in this case prefer to be > conservative and don't optimize. This could happen when for example: WHERE (a > OR b) AND (c OR d)." > It is implemented for XPath, and works fine there, so I think it is > reasonable to do that for SQL-2 as well for trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4623) Log more information when null DocumentNodeState is read for a child while fetching children
[
https://issues.apache.org/jira/browse/OAK-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426169#comment-15426169
]
Marcel Reutegger commented on OAK-4623:
---
I think the underlying document for the node state would be useful to have,
both cached and uncached. It would show whether the children or the state cache
entry is wrong.
> Log more information when null DocumentNodeState is read for a child while
> fetching children
>
>
> Key: OAK-4623
> URL: https://issues.apache.org/jira/browse/OAK-4623
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.6
>
>
> {{DocumentNodeStore#getChildNodes}} validates node states of children in the
> returned iterator and throws exception if a child is read as {{null}}.
> This is clearly an unexpected case (although, it has been observed in running
> systems). The more probable cause for such a scenario would be to have
> inconsistent value in {{nodeChildrenCache}}. It'd be useful to log more
> information about cache state to help investigating such cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2498) Root record references provide too little context for parsing a segment
[
https://issues.apache.org/jira/browse/OAK-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426104#comment-15426104
]
Michael Dürig commented on OAK-2498:
While we introduce new record types, we should also think of adding a type
{{RecordType.ROOT_NODE}} marking the roots of the content trees. This could be
helpful for implementing OAK-4103 (mind the
[disclaimer|https://issues.apache.org/jira/browse/OAK-4103?focusedCommentId=15426101=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15426101]
though). Furthermore it could simplify {{oak-run tarmkrecovery}} and at the
same time make it more reliable (see OAK-3804).
/cc [~alex.parvulescu] re. OAK-3804.
> Root record references provide too little context for parsing a segment
> ---
>
> Key: OAK-2498
> URL: https://issues.apache.org/jira/browse/OAK-2498
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Andrei Dulceanu
> Labels: tools
> Fix For: Segment Tar 0.0.12
>
>
> According to the [documentation |
> http://jackrabbit.apache.org/oak/docs/nodestore/segmentmk.html] the root
> record references in a segment header provide enough context for parsing all
> records within this segment without any external information.
> Turns out this is not true: if a root record reference turns e.g. to a list
> record. The items in that list are record ids of unknown type. So even though
> those records might live in the same segment, we can't parse them as we don't
> know their type.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4103) Replace journal.log with an in place journal
[
https://issues.apache.org/jira/browse/OAK-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426101#comment-15426101
]
Michael Dürig commented on OAK-4103:
The simplest way to implement this would probably be to introduce a special
kind record type (e.g. {{RecordType.ROOT_NODE}}). Then we could just scan
backwards through the segments until we find the first root node. However, we
need to be very careful here as this could easily re-introduce OAK-4291 through
the back door.
> Replace journal.log with an in place journal
>
>
> Key: OAK-4103
> URL: https://issues.apache.org/jira/browse/OAK-4103
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Andrei Dulceanu
>Priority: Minor
> Labels: resilience
>
> Instead of writing the current head revision to the {{journal.log}} file we
> could make it an integral part of the node states: as OAK-3804 demonstrates
> we already have very good heuristics to reconstruct a lost journal. If we add
> the right annotations to the root node states this could replace the current
> approach. The latter is problematic as it relies on the flush thread properly
> and timely updating {{journal.log}}. See e.g. OAK-3303.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4681) Automatically convert *all* "or" queries to "union" for SQL-2
[ https://issues.apache.org/jira/browse/OAK-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller updated OAK-4681: Component/s: query > Automatically convert *all* "or" queries to "union" for SQL-2 > - > > Key: OAK-4681 > URL: https://issues.apache.org/jira/browse/OAK-4681 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: query >Reporter: Thomas Mueller > Fix For: 1.6 > > > Currently, in OAK-1617, simple SQL-2 queries that contain "or" are converted > to "union" if the cost is lower. However, more complex queries are not > converted, see AndImpl.java, convertToUnion(), "in this case prefer to be > conservative and don't optimize. This could happen when for example: WHERE (a > OR b) AND (c OR d)." > It is implemented for XPath, and works fine there, so I think it is > reasonable to do that for SQL-2 as well for trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-4681) Automatically convert *all* "or" queries to "union" for SQL-2
Thomas Mueller created OAK-4681: --- Summary: Automatically convert *all* "or" queries to "union" for SQL-2 Key: OAK-4681 URL: https://issues.apache.org/jira/browse/OAK-4681 Project: Jackrabbit Oak Issue Type: New Feature Reporter: Thomas Mueller Currently, in OAK-1617, simple SQL-2 queries that contain "or" are converted to "union" if the cost is lower. However, more complex queries are not converted, see AndImpl.java, convertToUnion(), "in this case prefer to be conservative and don't optimize. This could happen when for example: WHERE (a OR b) AND (c OR d)." It is implemented for XPath, and works fine there, so I think it is reasonable to do that for SQL-2 as well for trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4681) Automatically convert *all* "or" queries to "union" for SQL-2
[ https://issues.apache.org/jira/browse/OAK-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller updated OAK-4681: Fix Version/s: 1.6 > Automatically convert *all* "or" queries to "union" for SQL-2 > - > > Key: OAK-4681 > URL: https://issues.apache.org/jira/browse/OAK-4681 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: query >Reporter: Thomas Mueller > Fix For: 1.6 > > > Currently, in OAK-1617, simple SQL-2 queries that contain "or" are converted > to "union" if the cost is lower. However, more complex queries are not > converted, see AndImpl.java, convertToUnion(), "in this case prefer to be > conservative and don't optimize. This could happen when for example: WHERE (a > OR b) AND (c OR d)." > It is implemented for XPath, and works fine there, so I think it is > reasonable to do that for SQL-2 as well for trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-4667) Create IT to validate that reclaimed size after cleanup is always positive
[
https://issues.apache.org/jira/browse/OAK-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-4667.
Resolution: Fixed
Done at http://svn.apache.org/viewvc?rev=1756739=rev
Thanks [~dulceanu] for sending a patch. I edited it a bit so it would also fail
in case there is a failure in one of the concurrently executed tasks.
> Create IT to validate that reclaimed size after cleanup is always positive
> --
>
> Key: OAK-4667
> URL: https://issues.apache.org/jira/browse/OAK-4667
> Project: Jackrabbit Oak
> Issue Type: Test
> Components: segment-tar
>Affects Versions: Segment Tar 0.0.10
>Reporter: Andrei Dulceanu
>Assignee: Andrei Dulceanu
>Priority: Minor
> Fix For: Segment Tar 0.0.10
>
> Attachments: OAK-4667-01.patch, OAK-4667-02.patch
>
>
> As OAK-4106 reworked the way {{reclaimedSize}} is computed, it would be nice
> to have an IT to validate the assumption that under the new circumstances
> this value is always positive. Of particular interest here is the scenario in
> which concurrent writes interweave with the cleanup process.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-4669) Cleanup creates new generation of tar file without removing any segments
[ https://issues.apache.org/jira/browse/OAK-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig resolved OAK-4669. Resolution: Fixed Fixed at http://svn.apache.org/viewvc?rev=1756738=rev Thanks [~dulceanu] for sending a patch. I edited it slightly before applying. > Cleanup creates new generation of tar file without removing any segments > - > > Key: OAK-4669 > URL: https://issues.apache.org/jira/browse/OAK-4669 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar >Affects Versions: Segment Tar 0.0.8 >Reporter: Andrei Dulceanu >Assignee: Andrei Dulceanu >Priority: Minor > Labels: cleanup, gc > Fix For: Segment Tar 0.0.10 > > Attachments: OAK-4669-01.patch, test case.patch > > > On some deployments I have seen tar files with a quite hight generation > post-fix (e.g. 'v'). From the log files I could deduce that this particular > tar file was rewritten multiple times without actually any segment being > removed. > I assume this is caused by the 25% gain threshold not taking the sizes > contributed by the index and the graph entries into account. > The attached test case can be used to verify the above hypothesis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-4106) Reclaimed size reported by FileStore.cleanup is off
[ https://issues.apache.org/jira/browse/OAK-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig resolved OAK-4106. Resolution: Fixed Fixed at http://svn.apache.org/viewvc?rev=1756737=rev Thanks [~dulceanu] for sending a patch. > Reclaimed size reported by FileStore.cleanup is off > --- > > Key: OAK-4106 > URL: https://issues.apache.org/jira/browse/OAK-4106 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar >Reporter: Michael Dürig >Assignee: Andrei Dulceanu >Priority: Minor > Labels: cleanup, gc > Fix For: Segment Tar 0.0.10 > > Attachments: OAK-4106-01.patch, OAK-4106-02.patch, OAK-4106-03.patch > > > The current implementation simply reports the difference between the > repository size before cleanup to the size after cleanup. As cleanup runs > concurrently to other commits, the size increase contributed by those is not > accounted for. In the extreme case where cleanup cannot reclaim anything this > can even result in negative values being reported. > We should either change the wording of the respective log message and speak > of before and after sizes or adjust our calculation of reclaimed size > (preferred). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4669) Cleanup creates new generation of tar file without removing any segments
[ https://issues.apache.org/jira/browse/OAK-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426057#comment-15426057 ] Michael Dürig commented on OAK-4669: Patch looks good, will commit it in a minute. > Cleanup creates new generation of tar file without removing any segments > - > > Key: OAK-4669 > URL: https://issues.apache.org/jira/browse/OAK-4669 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar >Affects Versions: Segment Tar 0.0.8 >Reporter: Andrei Dulceanu >Assignee: Andrei Dulceanu >Priority: Minor > Labels: cleanup, gc > Fix For: Segment Tar 0.0.10 > > Attachments: OAK-4669-01.patch, test case.patch > > > On some deployments I have seen tar files with a quite hight generation > post-fix (e.g. 'v'). From the log files I could deduce that this particular > tar file was rewritten multiple times without actually any segment being > removed. > I assume this is caused by the 25% gain threshold not taking the sizes > contributed by the index and the graph entries into account. > The attached test case can be used to verify the above hypothesis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4658) Outer join: name(b) and localname(b) can throw a NPE
[
https://issues.apache.org/jira/browse/OAK-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426046#comment-15426046
]
Thomas Mueller commented on OAK-4658:
-
http://svn.apache.org/r1756732 (formatting; trunk)
http://svn.apache.org/r1756733 (formatting; 1.4 branch)
> Outer join: name(b) and localname(b) can throw a NPE
>
>
> Key: OAK-4658
> URL: https://issues.apache.org/jira/browse/OAK-4658
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: query
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Fix For: 1.5.8, 1.4.7
>
>
> Outer join queries with a condition of "name(joinSelector) = 'x'" or
> "localname(joinSelector) = 'x'" can result in a NullPointerException. This is
> because the code doesn't take the "no-match" case into account (that the
> joined selector doesn't have a node). Example queries:
> {noformat}
> select a.[jcr:path] from [nt:base] as a
> left outer join [nt:base] as b on ischildnode(b, a)
> where name(b) = 'b'
> select a.[jcr:path] from [nt:base] as a
> left outer join [nt:base] as b on ischildnode(b, a)
> where localname(b) = 'b'
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4106) Reclaimed size reported by FileStore.cleanup is off
[ https://issues.apache.org/jira/browse/OAK-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426028#comment-15426028 ] Michael Dürig commented on OAK-4106: Patch LGTM, will apply it in a minute. > Reclaimed size reported by FileStore.cleanup is off > --- > > Key: OAK-4106 > URL: https://issues.apache.org/jira/browse/OAK-4106 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar >Reporter: Michael Dürig >Assignee: Andrei Dulceanu >Priority: Minor > Labels: cleanup, gc > Fix For: Segment Tar 0.0.10 > > Attachments: OAK-4106-01.patch, OAK-4106-02.patch, OAK-4106-03.patch > > > The current implementation simply reports the difference between the > repository size before cleanup to the size after cleanup. As cleanup runs > concurrently to other commits, the size increase contributed by those is not > accounted for. In the extreme case where cleanup cannot reclaim anything this > can even result in negative values being reported. > We should either change the wording of the respective log message and speak > of before and after sizes or adjust our calculation of reclaimed size > (preferred). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4196) EventListener gets removed event for denied node
[ https://issues.apache.org/jira/browse/OAK-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426020#comment-15426020 ] Marcel Reutegger commented on OAK-4196: --- Documenting the behaviour works for me. I still find it more intuitive if the event was not delivered. After all the session was explicitly denied read access to the node in question. > EventListener gets removed event for denied node > > > Key: OAK-4196 > URL: https://issues.apache.org/jira/browse/OAK-4196 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core, jcr, security >Affects Versions: 1.4 >Reporter: Marcel Reutegger >Priority: Minor > Fix For: 1.5.8 > > > An EventListener of a session that does not have read access to a node may > get a node removed event when the node is removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4675) SNFE thrown while testing FileStore.cleanup() running concurrently with writes
[
https://issues.apache.org/jira/browse/OAK-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426019#comment-15426019
]
Michael Dürig commented on OAK-4675:
Another way to fix this would be to keep the file's size in an field
initialised from the constructor.
> SNFE thrown while testing FileStore.cleanup() running concurrently with writes
> --
>
> Key: OAK-4675
> URL: https://issues.apache.org/jira/browse/OAK-4675
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar, segmentmk
>Affects Versions: 1.0.32, 1.4.6, 1.2.18, Segment Tar 0.0.8
>Reporter: Andrei Dulceanu
>Assignee: Andrei Dulceanu
> Labels: cleanup, gc
> Fix For: Segment Tar 0.0.10
>
> Attachments: test-case.patch
>
>
> {{SegmentNotFoundException}} is thrown from time to time in the following
> scenario: plenty of concurrent writes (each creating a {{625 bytes}} blob)
> interrupted by a cleanup.
> Stack trace (including some debugging statements added by me):
> {code:java}
> Pre cleanup readers: []
> Before cleanup readers: [/Users/dulceanu/work/test-repo/data0a.tar]
> Initial size: 357.4 kB
> After cleanup readers: [/Users/dulceanu/work/test-repo/data0a.tar]
> After cleanup size: 357.4 kB
> Final size: 361.0 kB
> Exception in thread "pool-5-thread-74"
> org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Cannot copy
> record from a generation that has been gc'ed already
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1207)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1096)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1074)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:987)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.access$700(SegmentWriter.java:379)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$8.execute(SegmentWriter.java:337)
> at
> org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.execute(SegmentBufferWriterPool.java:105)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter.writeNode(SegmentWriter.java:334)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:111)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:550)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:571)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:627)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore.merge(SegmentNodeStore.java:287)
> at
> org.apache.jackrabbit.oak.segment.CompactionAndCleanupIT$1.run(CompactionAndCleanupIT.java:961)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.jackrabbit.oak.segment.SegmentNotFoundException:
> Segment 4fb637cc-5013-4925-ab13-0629c4406481 not found
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1341)
> at
> org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:123)
> at
> org.apache.jackrabbit.oak.segment.RecordId.getSegment(RecordId.java:94)
> at
> org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1199)
> ... 18 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IllegalStateException: Invalid segment format. Dumping segment
> 4fb637cc-5013-4925-ab13-0629c4406481
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0070 00 00 00 00 00 00 00 00 00 00
