BUILD FAILURE: Jackrabbit Oak - Build # 357 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #357)

Status: Still Failing

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/357/ to 
view the results.

Changes:
[angela] OAK-6277 : UserQueryManager: redundant check for colliding bound and 
offset 
OAK-6278 : UserQueryManager: scope filtering for everyone groupId compares to 
principal name
OAK-5882 : Improve coverage for oak.security code in oak-core (wip)

 

Test results:
1 tests failed.
FAILED:  
org.apache.jackrabbit.oak.plugins.blob.UploadStagingCacheTest.testUpgrade

Error Message:
expected null, but 
was:

Stack Trace:
java.lang.AssertionError: expected null, but 
was:
at 
org.apache.jackrabbit.oak.plugins.blob.UploadStagingCacheTest.testUpgrade(UploadStagingCacheTest.java:608)

Re: [discuss] expose way to detect "eventual consistency delay"

[Stefan Egli]

On 30/05/17 14:51, "Stefan Egli"  wrote:

>on how Oak could "expose a way to detect the eventual delay".

... "to detect the eventual consistency delay" ...

of course ...

[discuss] expose way to detect "eventual consistency delay"

[Stefan Egli]

Hi all,

I'd like to invite those interested to join a discussion in

https://issues.apache.org/jira/browse/OAK-6276

on how Oak could "expose a way to detect the eventual delay".

This is a requirement coming from the integration with an external messaging
system in an Oak-based application.

One way suggested so far is that this could simply be done by exposing a
"normalized head revision vector" via a repository descriptor.

But let's discuss over in OAK-6276.

Thanks,
Cheers,
Stefan

BUILD FAILURE: Jackrabbit Oak - Build # 356 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #356)

Status: Still Failing

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/356/ to 
view the results.

Changes:
[rombert] OAK-6196 - Improve Javadoc of multiplexing SPI

Expand the Javadoc for Mount.isMounted

 

Test results:
1 tests failed.
FAILED:  
org.apache.jackrabbit.oak.upgrade.cli.JdbcToSegmentTest.validateMigration

Error Message:
Failed to copy content

Stack Trace:
javax.jcr.RepositoryException: Failed to copy content
Caused by: java.lang.IllegalStateException: Branch with failed reset
Caused by: org.apache.jackrabbit.oak.api.CommitFailedException: OakOak0100: 
Branch reset failed
Caused by: org.apache.jackrabbit.oak.plugins.document.DocumentStoreException: 
Empty branch cannot be reset

[ANNOUNCE] Apache Jackrabbit Oak 1.7.0 released

[Davide Giannella]

The Apache Jackrabbit community is pleased to announce the release of
Apache Jackrabbit Oak. The release is available for download at:

http://jackrabbit.apache.org/downloads.html

See the full release notes below for details about this release:

Release Notes -- Apache Jackrabbit Oak -- Version 1.7.0

Introduction


Jackrabbit Oak is a scalable, high-performance hierarchical content
repository designed for use as the foundation of modern world-class
web sites and other demanding content applications.

Apache Jackrabbit Oak 1.7.0 is an unstable release cut directly from
Jackrabbit Oak trunk, with a focus on new features and other
improvements. For production use we recommend the latest stable 1.6.x
release.

The Oak effort is a part of the Apache Jackrabbit project.
Apache Jackrabbit is a project of the Apache Software Foundation.

Changes in Oak 1.7.0
-

Sub-task

[OAK-5869] - Annotate documents with branch commits
[OAK-5964] - Invalidate documents through journal
[OAK-5968] - Introduce RevisionContext.getClock()

Technical task

[OAK-3690] - Decouple SegmentBufferWriter from SegmentStore
[OAK-5554] - RDB*Store: update postgresql JDBC driver reference to
9.4.1212
[OAK-] - RDB*Store: update Tomcat JDBC pool dependency to
7.0.73
[OAK-5627] - RDBDocumentStore: improve long query logging
[OAK-5652] - RDB*Store: update Oracle JDBC driver reference to
12.1.0.2.0
[OAK-5653] - RDB*Store: update Derby to release 10.13
[OAK-5667] - RDBDocumentStore: remove support for DBs without
support for CASE statements in SELECT
[OAK-5751] - RDBDocumentStore: properly handle null values for
system properties
[OAK-5852] - RDB*Store: update Tomcat JDBC pool dependency to
7.0.75
[OAK-5855] - RDBDocumentStore: improve query support for VersionGC
[OAK-5977] - Document enhancements in S3DataStore in 1.6
[OAK-5981] - SegmentTar version check with disabled mmaping
[OAK-6083] - RDBDocumentStore: implement support for
VersionGCSupport extensions added for OAK-4780
[OAK-6134] - RDB*Store: update PostgreSQL JDBC
[OAK-6140] - Create RDB-specific BlobReferenceIterator
[OAK-6143] - RDB*store fixtures: shorten table name prefixes for
Oracle
[OAK-6176] - Service to provide access to async indexer state
[OAK-6192] - Lucene IndexInfoProvider implementation
[OAK-6207] - RDBDocumentStore: allow schema evolution part 2:
record schema version when updating/inserting rows
[OAK-6216] - Property IndexInfoProvider implementation
[OAK-6224] - Enable dumping index definitions and stats via
oak-run
[OAK-6226] - RDBDocumentStoreDB: missing @Override statements
[OAK-6228] - Enable index consistency check via oak-run
[OAK-6231] - Enable dumping index content via oak-run
[OAK-6236] - Improve the help output from oak-run commands
[OAK-6244] - RDB*Store: update postgresql JDBC driver reference to
42.1.1
[OAK-6247] - RDB*Store: update Tomcat JDBC pool dependency to
7.0.78

Bug

[OAK-4390] - DocumentStoreStatsIT.update fails when RDB's append
mode is disabled
[OAK-4529] - DocumentNodeStore does not have a repository software
version range check
[OAK-5301] - Possible null dereference in MapRecord
[OAK-5355] - Too eager refreshing of tree permissions in
SecureNodeBuilder
[OAK-5357] - StringUtils conversion functions can throw
NullPointerException
[OAK-5441] - Test failure: BasicServerTest.testServerOk() Address
already in use
[OAK-5450] - Documented example for relativeNode in index
aggregation does not work.
[OAK-5500] - Oak Standalone throws ClassNotFoundException:
remoting/protectedHandlersConfig.xml
[OAK-5501] - Oak Standalone: Webdav configuration is set to
remoting mode by default
[OAK-5536] - Facets on relative properties do not work properly
[OAK-5557] - incomplete diffManyChildren during commitHook
evaluation in a persisted branch
[OAK-5573] -
org.apache.jackrabbit.oak.segment.standby.StandbyTestIT.testSyncLoop
[OAK-5580] - Show statistics about I/O operations in the check
command
[OAK-5590] - The check command doesn't do any check when "deep"
option is not provided
[OAK-5601] - documentMk backgroundRead should handle missing
journal entries
[OAK-5619] - withIncludeAncestorsRemove reports unrelated
top-level node deletion
[OAK-5626] - ChangeProcessor doesn't reset 'blocking' flag when
items from queue gets removed and commit-rate-limiter is null
[OAK-5636] - potential NPE in ReplicaSetInfo
[OAK-5649] - Error in RefreshPolicy can lead to IndexNode lock
leak
[OAK-5651] - java.lang.IllegalStateException logged when migrating
Segment to Document
[OAK-5657] - leverage project.version in oak-examples
[OAK-5668] - Test failure:
observation.ObservationQueueFullWarnTest.warnOnRepeatedQueueFull

[RESULT][VOTE] Release Apache Jackrabbit Oak 1.7.0

[Davide Giannella]

Hello Team,

the vote passes as follows:

+1 Davide Giannella
+1 Alex Parvulescu
+1 Michael Dürig

Thanks for voting. I'll push the release out.

-- Davide

Re: copy on write node store

[Julian Sedding]

Slightly off topic: the thought that the copy on read/write indexing
features may need to be explicitly managed in such a setup just
occurred to me.

I.e. when an instance is switched to the copy on write node store, the
local index directory will deviate from the "mainline" node store.
Upon switching the instance back to the "mainline" (i.e. disabling
copy on write node store), the local index directory may need to be
deleted? Or maybe it is already resilient enough to automatically
recover.

Regards
Julian


On Tue, May 30, 2017 at 10:05 AM, Michael Dürig  wrote:
>
>
> On 30.05.17 09:34, Tomek Rekawek wrote:
>>
>> Hello Michael,
>>
>> thanks for the reply!
>>
>>> On 30 May 2017, at 09:18, Michael Dürig  wrote:
>>> AFAIU from your mail and from looking at the patch this is about a node
>>> store implementation that can be rolled back to a previous state.
>>>
>>> If this is the case, a simpler way to achieve this might be to use the
>>> TarMK and and add functionality for rolling it back.
>>
>>
>> Indeed, it would be much simpler. However, the main purpose of the new
>> feature is testing the blue-green Sling deployments. That’s why we need the
>> DocumentMK to support it as well.
>
>
> Ok I see. I think the fact that these classes are not for production use
> should be stated in the Javadoc along with what clarifications of what can
> be expected from the store wrt. interleaving of calls to various mutators
> (e.g. enableCopyOnWrite() / disableCopyOnWrite() / merge(), etc.). I foresee
> a couple of very sneaky race conditions here.
>
> Michael

Re: failure building oak-upgrade

[Angela Schreiber]

hi tomek

confirming that the issue is gone. thanks a lot
angela

On 29/05/17 20:45, "Tomek Rekawek"  wrote:

>Hi,
>
>regression fixed, sorry for that.
>
>Regards,
>Tomek
>
>-- 
>Tomek Rękawek | Adobe Research | www.adobe.com
>reka...@adobe.com
>
>> On 29 May 2017, at 20:11, Vikas Saurabh  wrote:
>> 
>> Hi Angela,
>> 
>>> do others experience the same issue? and if yes, is anybody working on
>>> this?
>>> 
>> Yes, this seems to affecting generally (OAK-6273). I guess Tomek would
>> check it out.
>> 
>> Thanks,
>> Vikas
>

Re: copy on write node store

[Michael Dürig]

On 30.05.17 09:34, Tomek Rekawek wrote:

Hello Michael,

thanks for the reply!


On 30 May 2017, at 09:18, Michael Dürig  wrote:
AFAIU from your mail and from looking at the patch this is about a node store 
implementation that can be rolled back to a previous state.

If this is the case, a simpler way to achieve this might be to use the TarMK 
and and add functionality for rolling it back.


Indeed, it would be much simpler. However, the main purpose of the new feature 
is testing the blue-green Sling deployments. That’s why we need the DocumentMK 
to support it as well.


Ok I see. I think the fact that these classes are not for production use 
should be stated in the Javadoc along with what clarifications of what 
can be expected from the store wrt. interleaving of calls to various 
mutators (e.g. enableCopyOnWrite() / disableCopyOnWrite() / merge(), 
etc.). I foresee a couple of very sneaky race conditions here.


Michael

Re: copy on write node store

[Tomek Rekawek]

Hello Michael,

thanks for the reply!

> On 30 May 2017, at 09:18, Michael Dürig  wrote:
> AFAIU from your mail and from looking at the patch this is about a node store 
> implementation that can be rolled back to a previous state.
> 
> If this is the case, a simpler way to achieve this might be to use the TarMK 
> and and add functionality for rolling it back.

Indeed, it would be much simpler. However, the main purpose of the new feature 
is testing the blue-green Sling deployments. That’s why we need the DocumentMK 
to support it as well.

Regards,
Tomek

-- 
Tomek Rękawek | Adobe Research | www.adobe.com
reka...@adobe.com
> 
> 
> 
> On 29.05.17 10:50, Tomek Rekawek wrote:
>> Hello,
>> in the OAK-6220 I’m exploring a topic of having a switchable copy-on-write 
>> node store implementation. The idea is that the “main” node store (eg. 
>> DocumentMK) is wrapped with an extra layer (copy-on-write node store), which 
>> can be turned on/off in the runtime. When the copy-on-write is turned on, 
>> all the new changes are not merged with the main store, but kept in a 
>> separate, volatile store.
>> The new mode is meant to be used for testing - so we can perform even 
>> destructible tests and then reverse all the changes seamlessly. It’s 
>> especially useful in the blue-green deployments with CompositeNodeStore and 
>> DocumentMK, since we can test the new version of the application on the new 
>> (green) instance, even if the tests requires changes in the node schema. The 
>> changes won’t be propagated to the old (blue) instance as long as the COW 
>> mode is on.
>> Together with other people involved in the issue we had 3 ideas how this can 
>> be implemented:
>> 1. By copying the / node and its subtree into some private location and then 
>> mount the COW store on top of it.
>> This works fine for the SegmentMK (supporting the copy-by-reference), but 
>> not with the DocumentMK (which actually copied the whole tree). Since the 
>> new feature is more useful with DocumentMK, we needed to find something else.
>> 2. By storing the data in a NodeBuilder taken from the store without merging 
>> it back to the main repository.
>> This seemed to worked fine, but because of the DocumentMK limitations 
>> (OAK-1838) this wasn’t reliable.
>> 3. By creating a MemoryNodeStore on a top of the recent root state
>> This is the current implementation, it works fine [1]. The newly created 
>> MemoryNodeStore didn’t contain any checkpoints, so some extra layer 
>> (BranchNodeStore) was introduced to inherit the already existing checkpoints 
>> from the main store. Another layer (CowNodeStore) is being used to 
>> dynamically switch between the main and the branch node store.
>> Potential limitation here is that the changes have to fit into memory. 
>> Switching the repository into COW mode and forgetting about this is not a 
>> good idea.
>> I’d like to merge the [1], so the blue-green Sling deployments can be tested 
>> in the more robust way. Any thoughts?
>> Regards,
>> Tomek
>> [1] 
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2Fattachment%2F12868273%2FOAK-6220-3.patch=02%7C01%7C%7Cb4d5c84e0fc141ccc03b08d4a72c2496%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636317255425935320=i0XxBrunLWPWgIAhMqkr4F8Pfo1ktHNFlJ2V1H%2FpmQg%3D=0

Re: copy on write node store

[Michael Dürig]

Hi Tomek,

AFAIU from your mail and from looking at the patch this is about a node 
store implementation that can be rolled back to a previous state.


If this is the case, a simpler way to achieve this might be to use the 
TarMK and and add functionality for rolling it back. The cleanest way 
would be to add a method SegmentNodeStore.rollBack(String checkpoint), 
whose implementation is basically a call to Revisions.setHead(...) 
passing the state of the checkpoint.
Implementation wise there might be a few hoops as there is this probably 
needs to be looped through the Scheduler. But I guess Andrei might be 
able to help out on the details here.


Michael


On 29.05.17 10:50, Tomek Rekawek wrote:

Hello,

in the OAK-6220 I’m exploring a topic of having a switchable copy-on-write node 
store implementation. The idea is that the “main” node store (eg. DocumentMK) 
is wrapped with an extra layer (copy-on-write node store), which can be turned 
on/off in the runtime. When the copy-on-write is turned on, all the new changes 
are not merged with the main store, but kept in a separate, volatile store.

The new mode is meant to be used for testing - so we can perform even 
destructible tests and then reverse all the changes seamlessly. It’s especially 
useful in the blue-green deployments with CompositeNodeStore and DocumentMK, 
since we can test the new version of the application on the new (green) 
instance, even if the tests requires changes in the node schema. The changes 
won’t be propagated to the old (blue) instance as long as the COW mode is on.

Together with other people involved in the issue we had 3 ideas how this can be 
implemented:

1. By copying the / node and its subtree into some private location and then 
mount the COW store on top of it.

This works fine for the SegmentMK (supporting the copy-by-reference), but not 
with the DocumentMK (which actually copied the whole tree). Since the new 
feature is more useful with DocumentMK, we needed to find something else.

2. By storing the data in a NodeBuilder taken from the store without merging it 
back to the main repository.

This seemed to worked fine, but because of the DocumentMK limitations 
(OAK-1838) this wasn’t reliable.

3. By creating a MemoryNodeStore on a top of the recent root state

This is the current implementation, it works fine [1]. The newly created 
MemoryNodeStore didn’t contain any checkpoints, so some extra layer 
(BranchNodeStore) was introduced to inherit the already existing checkpoints 
from the main store. Another layer (CowNodeStore) is being used to dynamically 
switch between the main and the branch node store.

Potential limitation here is that the changes have to fit into memory. 
Switching the repository into COW mode and forgetting about this is not a good 
idea.

I’d like to merge the [1], so the blue-green Sling deployments can be tested in 
the more robust way. Any thoughts?

Regards,
Tomek

[1] https://issues.apache.org/jira/secure/attachment/12868273/OAK-6220-3.patch

BUILD FAILURE: Jackrabbit Oak - Build # 355 - Failure

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #355)

Status: Failure

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/355/ to 
view the results.

Changes:
[mduerig] OAK-6272: AbstractNodeState.toString does not scale to many child 
nodes
Prefix system property with "oak."

 

Test results:
1 tests failed.
FAILED:  
org.apache.jackrabbit.oak.segment.standby.ExternalPrivateStoreIT.testProxyFlippedIntermediateByteChange

Error Message:
expected:<{ root = { ... } }> but was:<{ root : { } }>

Stack Trace:
java.lang.AssertionError: expected:<{ root = { ... } }> but was:<{ root : { } }>

Re: svn commit: r1796624 - /jackrabbit/oak/trunk/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/spi/state/AbstractNodeState.java

[Michael Dürig]

On 30.05.17 07:00, Chetan Mehrotra wrote:

On Mon, May 29, 2017 at 6:50 PM,   wrote:

+private static final int CHILDREN_CAP = getInteger("children.cap", 100);


Better to have system property prefix with 'oak' i.e. 'oak.children.cap'



Agreed. Done at  http://svn.apache.org/viewvc?rev=1796728=rev

Michael

Re: codahale metrics Jmx reporter

[Chetan Mehrotra]

On Tue, May 30, 2017 at 11:53 AM, Andrei Kalfas
 wrote:
> Looks to me that there is a dependency on oak functionality.

Ian can confirm but I believe thats not required now (the component
does not get activated) and was only a temporary workaround. Oak
publishes the MetricRegistry instance in OSGi service registry and
then any component can look up that service and configure a reporter
for it

Chetan Mehrotra

Re: codahale metrics Jmx reporter

[Andrei Kalfas]

https://github.com/ieb/statsd-reporter-osgi/blob/master/src/main/java/org/apache/sling/statsd/OakStatisticsProviderShim.java#L24
 


Looks to me that there is a dependency on oak functionality.

Anyhow, thank you for the clarifications.

Thanks,
Andrei

> On May 30, 2017, at 9:13 AM, Chetan Mehrotra  
> wrote:
> 
> On Tue, May 30, 2017 at 11:30 AM, Andrei Kalfas
>  wrote:
>> I thought that oak-core is the best position to say what metrics are needed, 
>> and the way that we want to extend the behavior is by picking whatever 
>> reporter we say fits the environment - this is why I started the thread, I 
>> wanted to understand whats the intended design.
> 
> Oak is only concerned with publishing metrics to a MetricRegistry. How
> the metrics are consumed and reported are outside of Oak scope
> 
> Various reporters that Ian referred are generic and can support any
> MetricRegistry and have no dependency on Oak. Hence no needs for them
> to be in Oak
> 
> Chetan Mehrotra



smime.p7s
Description: S/MIME cryptographic signature

Re: codahale metrics Jmx reporter

[Chetan Mehrotra]

On Tue, May 30, 2017 at 11:30 AM, Andrei Kalfas
 wrote:
> I thought that oak-core is the best position to say what metrics are needed, 
> and the way that we want to extend the behavior is by picking whatever 
> reporter we say fits the environment - this is why I started the thread, I 
> wanted to understand whats the intended design.

Oak is only concerned with publishing metrics to a MetricRegistry. How
the metrics are consumed and reported are outside of Oak scope

Various reporters that Ian referred are generic and can support any
MetricRegistry and have no dependency on Oak. Hence no needs for them
to be in Oak

Chetan Mehrotra

Re: codahale metrics Jmx reporter

[Andrei Kalfas]

Hi,

Exposing the metrics registry is about exposing the data, while exposing the 
reporter would be exposing behavior. Doing so, the encouraged pattern here is 
that you get to tap into the metrics and pick whatever you like.

I thought that oak-core is the best position to say what metrics are needed, 
and the way that we want to extend the behavior is by picking whatever reporter 
we say fits the environment - this is why I started the thread, I wanted to 
understand whats the intended design.

I was not saying that reporting in JMX is a good thing, I was simply stating 
that it can be the default in order not to break already existing assumptions.

Why are the bellow mentioned reporters not in oak? 

Thanks,
Andrei


> On May 29, 2017, at 2:56 PM, Ian Boston  wrote:
> 
> Hi,
> Here are some reporters that work with Sling/Oak/AEM. [1]. They all look
> for components registered as implementing MetricsRegistry and then
> aggregate the data pumping it out to a reporter. They are all implemented
> as independent bundles. TBH I would avoid pumping the metrics into
> JMX as JMX was designed for management, and not metrics. It might be able
> to cope with trivial metrics sets, but will likely start to consume
> unreasonable JVM resources with a production set of metrics..
> 
> Most of the reporters in [1] are simple wrappers round other 3rd party
> Metrics reporters. If you have a target not included in that list, its
> trivial to follow the same pattern.
> 
> HTH
> Best Regards
> Ian
> 
> 1
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fieb%2Fstatsd-reporter-osgi=02%7C01%7C%7Cf54c602fdb634fb1adad08d4a689d350%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636316558271983739=1ypTPtDZV8lxw95HEYrd2llBjlu%2F4KO1e3xISA8Hd2Y%3D=0
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fieb%2Fprometheus-reporter-osgi=02%7C01%7C%7Cf54c602fdb634fb1adad08d4a689d350%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636316558271983739=so6%2F2Jn6m7zst5vZ3aCI25HOpOhPbk8RvomBPa7J%2B%2FA%3D=0
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fieb%2Finfluxdb-reporter-osgi=02%7C01%7C%7Cf54c602fdb634fb1adad08d4a689d350%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636316558271993752=qyBUqTAxLSkFUPJZGIUujBqFVOnCy5I9n4su12i%2BWSo%3D=0
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fieb%2Fgmond-osgi=02%7C01%7C%7Cf54c602fdb634fb1adad08d4a689d350%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636316558271993752=Wtvr1gs1nFRm5W81fGsA3l8QXUS6Cm8Z%2BMKE6%2F%2BEy2k%3D=0
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fieb%2Ftsdb-reporter-osgi=02%7C01%7C%7Cf54c602fdb634fb1adad08d4a689d350%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636316558271993752=V%2F%2BVdEsjyqiae1k%2BWSYGIecDt4v7loujSWdj8uue3jo%3D=0
> 
> On 29 May 2017 at 12:48, Andrei Kalfas  wrote:
> 
>> Hi,
>> 
>>> By default this is the only mode.
>> 
>> What would you guys rather prefer, have a different component peeks into
>> the metrics registry or change oak-core to deal with multiple reporters -
>> Jmx should be the default one.
>> 
>> Thanks,
>> Andrei
>> 
>> 



smime.p7s
Description: S/MIME cryptographic signature