[Oak origin/1.4] Apache Jackrabbit Oak matrix - Build # 1453 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build 
#1453)

Status: Still Failing

Check console output at 
https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1453/ to view 
the results.

Changes:
[reschke] OAK-5773: BlobCache does not implement Closeable (ported to 1.4)

[reschke] OAK-5751: RDBDocumentStore: properly handle null values for system

 

Test results:
All tests passed

Re: Supporting "resumable" operations on a large tree

[Thomas Mueller]

Hi,

My suggestion is to _not_ support "resumable" operations on a large tree,
but instead don't use large operations. But I wouldn't call my solution
"sharding", but more "bit-by-bit reindexing". Some more details: For
indexing (specially synchronous property indexes) I suggest to do the
following, for both a new index, and for reindexing:

1) Reindexing is writing to a new subtree, that is, ":index_1",
":index_2",... Which one is used is (automatically) set in the index
itself at /oak:index/<...>, in a hidden property ":writeNode"

2) Reading (queries) use the old subtree if there is any (":index" right
now). Which one it used is (automatically) set in the index itself at
/oak:index/<...>, in a hidden property ":liveNode"

3) Synchronous index updates are written to the index node defined at
":liveNode". Therefore, at the same time, reindex to a new subtree and
index updates to the old subtree can occur (actually more than that, see
below at 6).

4) To track reindexing/indexing progress, there is a "current position"
persisted at /oak:index/<...>, in a hidden property ":writePath". This
entry is automatically advanced by the asynchronous reindexing thread from
"/" to "/a" to "/b/a" to "/content/a/b", "/content/x", "/system/03/01",
"/system/04", "/system/0a",... and so on, until the whole repository is
indexed.

5) The asynchronous indexing thread reads all nodes of the repository
_after_ what is written at ":writePath" and indexes that to the new
subtree, and once there are 1000 changes to the index, the last indexed
path is written to ":writePath", plus the additions to the index, in one
transaction.

6) Synchronous index updates are also written to the index node defined at
":writeNode", if they affect nodes before the ":writePath".

7) After a restart, reindexing continues where it left off, using the last
":writePath".

8) After reindexing is done, ":livePath" is updated to ":index_2", and
":writePath" is removed. Also, the old index subtree is removed (1000
nodes per commit).

9) Sorting of path is needed, so that the repository can be processed bit
by bit by bit. For that, the following logic is used, recursively: read at
most 1000 child nodes. If there are more than 1000, then this subtree is
never split but processed in one step (so many child nodes can still lead
to large transactions, unfortunately). If less than 1000 child nodes, then
the names of all child nodes are read, and processed in sorted order
(sorted by node name).

Therefore, all reindexing operations use small transactions. Queries can
run concurrently. Reindexing can be paused. Reindexing can continue even
after a restart. While reindexing, the old index is maintained and
up-to-date. The branch-less commit mode is not needed. No conflicts
between the synchronous index updates and asynchronous reindexing thread
are possible.

Regards,
Thomas

SHA-1 collision

[Thomas Mueller]

Hi,

A SHA-1 collision has been published:
https://www.schneier.com/blog/archives/2017/02/sha-1_collision.html
https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html

Our FileDataStore and S3DataStore use SHA-1. For new binaries, we should use 
(for example) SHA-256.

Right now, a content management system that uses Oak as the repository can't 
serve those two files at the same time, if it uses the FileDataStore or the 
S3DataStore.

(The FileBlobStore, MongoDB BlobStore,..., are not affected)

Regards,
Thomas

[Oak origin/1.6] Apache Jackrabbit Oak matrix - Build # 1452 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build 
#1452)

Status: Still Failing

Check console output at 
https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1452/ to view 
the results.

Changes:
[reschke] OAK-5773: BlobCache does not implement Closeable (ported to 1.6)

[reschke] OAK-5751: RDBDocumentStore: properly handle null values for system

 

Test results:
All tests passed

Re: Strong all documents under Root - severe slowness on start-up

[Chetan Mehrotra]

Can you provide a thread dump around startup time where you see Oak is
reading all child nodes?
Chetan Mehrotra


On Fri, Feb 24, 2017 at 2:26 AM, Eugene Prystupa
 wrote:
> Thanks, Michael.
>
> I should have included more details in the original email.
> We are on 1.4.10 version of Jackrabbit Oak, we are using Mongo backend.
>
>
> On Thu, Feb 23, 2017 at 3:40 PM, Michael Dürig  wrote:
>
>>
>>
>> On 23.02.17 19:11, Eugene Prystupa wrote:
>>
>>> We are seeing severe delays on start-up (20 minutes+) when repository is
>>> created (new Jcr(oak).createRepository()).
>>>
>>
>> Regardless of the content structure, 20 min. seems off. What back-end are
>> you on? Which version of Oak is this?
>>
>> Michael
>>
>
>
>
> --
> Thanks,
> Eugene

[Oak origin/1.0] Apache Jackrabbit Oak matrix - Build # 1451 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build 
#1451)

Status: Still Failing

Check console output at 
https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1451/ to view 
the results.

Changes:
[reschke] OAK-5701: Update Oak 1.0 to Jackrabbit 2.8.5

[reschke] OAK-5773: BlobCache does not implement Closeable (ported to 1.0)

 

Test results:
25 tests failed.
FAILED:  
org.apache.jackrabbit.oak.security.authentication.ldap.LdapProviderTest.testGetUserByUserId

Error Message:
ERR_171 Failed to bind an LDAP service (1,024) to the service registry.

Stack Trace:
org.apache.directory.api.ldap.model.exception.LdapConfigurationException: 
ERR_171 Failed to bind an LDAP service (1,024) to the service registry.
at 
org.apache.directory.server.ldap.LdapServer.startNetwork(LdapServer.java:695)
at 
org.apache.directory.server.ldap.LdapServer.start(LdapServer.java:547)
at 
org.apache.jackrabbit.oak.security.authentication.ldap.AbstractServer.setUp(AbstractServer.java:227)
at 
org.apache.jackrabbit.oak.security.authentication.ldap.InternalLdapServer.setUp(InternalLdapServer.java:33)
at 
org.apache.jackrabbit.oak.security.authentication.ldap.LdapProviderTest.before(LdapProviderTest.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:463)
at sun.nio.ch.Net.bind(Net.java:455)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198)
at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51)
at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547)
at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68)
at 
org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422)
at 

How can I start ?

[Edward Pan]

Hello OAK development Team,

I'm trying to use OAK to save document in local file system.
I still get confused after read the all documents on
https://jackrabbit.apache.org/oak/docs/index.html,
Could you please give me some guide on it ?

Thank you and Regards,
Edward

[Oak origin/1.2] Apache Jackrabbit Oak matrix - Build # 1450 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build 
#1450)

Status: Still Failing

Check console output at 
https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1450/ to view 
the results.

Changes:
[reschke] OAK-5773: BlobCache does not implement Closeable (ported to 1.2)

 

Test results:
2 tests failed.
FAILED:  
org.apache.jackrabbit.oak.plugins.document.ConcurrentQueryAndUpdateIT.cacheUpdate[RDBFixture:
 RDB-H2(file)]

Error Message:
Unexpected revision timestamp for 1:/node-0 expected:<39> but was:<38>

Stack Trace:
java.lang.AssertionError: Unexpected revision timestamp for 1:/node-0 
expected:<39> but was:<38>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at 
org.apache.jackrabbit.oak.plugins.document.ConcurrentQueryAndUpdateIT.cacheUpdate(ConcurrentQueryAndUpdateIT.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)


FAILED:  
org.apache.jackrabbit.oak.run.osgi.DocumentNodeStoreConfigTest.testRDBDocumentStoreRestart

Error Message:
Service of type interface org.apache.jackrabbit.oak.spi.state.NodeStore was 
found. Expression: (sr == null). Values: sr = 
[org.apache.jackrabbit.oak.spi.state.NodeStore]

Stack Trace:
java.lang.AssertionError: Service of type interface 
org.apache.jackrabbit.oak.spi.state.NodeStore was found. Expression: (sr == 
null). Values: sr = [org.apache.jackrabbit.oak.spi.state.NodeStore]
at 
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:400)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:646)
at 
org.apache.jackrabbit.oak.run.osgi.AbstractRepositoryFactoryTest.assertNoService(AbstractRepositoryFactoryTest.groovy:89)
at 

Windows Jenkins issue flood

[Michael Dürig]

Hi,

Windows Jenkins was wreaking havoc so I disabled automatic issue 
creation again and deleted the last round of useless and noisy issues.



Michael

Re: Strong all documents under Root - severe slowness on start-up

[Eugene Prystupa]

Thanks, Michael.

I should have included more details in the original email.
We are on 1.4.10 version of Jackrabbit Oak, we are using Mongo backend.


On Thu, Feb 23, 2017 at 3:40 PM, Michael Dürig  wrote:

>
>
> On 23.02.17 19:11, Eugene Prystupa wrote:
>
>> We are seeing severe delays on start-up (20 minutes+) when repository is
>> created (new Jcr(oak).createRepository()).
>>
>
> Regardless of the content structure, 20 min. seems off. What back-end are
> you on? Which version of Oak is this?
>
> Michael
>



-- 
Thanks,
Eugene

Re: Strong all documents under Root - severe slowness on start-up

[Michael Dürig]

On 23.02.17 19:11, Eugene Prystupa wrote:

We are seeing severe delays on start-up (20 minutes+) when repository is
created (new Jcr(oak).createRepository()).


Regardless of the content structure, 20 min. seems off. What back-end 
are you on? Which version of Oak is this?


Michael

Re: Strong all documents under Root - severe slowness on start-up

[Michael Marth]

Hi Eugene,

There are 2 aspects here:
The repository definitely should not take such a long time for startup (or 
presumably read all/many children of root at startup).
But the reason why this was not seen before pertains to your second question:
It is definitely not advised to store that many document anywhere in the repo, 
let alone root. The preferred way to use the repo is by having some kind of 
hierarchy that puts the documents into different folders. This allows for 
access control on these folders. It also allows humans a more manageable way to 
browse the documents.
If there is on apparent hierarchy one can introduce an artificial one like 
day/month of creation or a hash of a property of the document, etc.

HTH
Michael




On 23/02/17 19:11, "Eugene Prystupa"  wrote:

>Can anyone confirm if this is a viable storage option? We have an
>Oak-backed repository with ~100,000 documents in it (and growing), and all
>documents are stored as children nodes of the root node.
>
>We are seeing severe delays on start-up (20 minutes+) when repository is
>created (new Jcr(oak).createRepository()). At appears that for whatever
>reason Oak is trying to read all direct root children on start-up and that
>takes a lot of time. Is storing documents not directly under root a better
>choice?
>
>-- 
>Thanks,
>Eugene

Re: Supporting "resumable" operations on a large tree

[Vikas Saurabh]

Hi,

A quick side-question related to what Stefan mentioned earlier:
> A stable traversal order at a given revision + node seems like a
prerequisite to me.

Javadoc of NodeState#getChildNodeEntries says:
" Multiple iterations are guaranteed to return the child nodes in
the same order, but the specific order used is implementation
dependent and may change across different states of the same node."

I'm not entirely sure if that's completely ambiguous - but afaik the
behavior of current stores (Tar - both versions, MongoMK and RDB),
they'd indeed to do stable iterations at a given revision even on
multiple calls to getChildNodeEntries. I wonder if we should call it
out explicitly too.

Btw, I think we'd still need to make a checkpoint to safeguard against
rev-gc/compaction.

Also, afaiu,
[Marcel]
> An intermediate commit (OAK-2556) would have to be annotated with the
> current path, while the checkpoint stays the same. For the re-index use case
> this probably also means an indirection for the index data tree is
> necessary.

and

[Thomas]
> For example use a "fromPath"
> .. "toPath" range, and only re-index part of the repository at a time

are different ideas. Marcel probably meant an intermediate commit with
some prop making current_path while Thomas is probably saying that we
should do indexing on sub-trees and hence have kind-of shards of the
index.

I don't think we should do shard-approach at least on lucene indices
as doing post-query-merge won't give right relevance ordering.
Also, I don't think we have a good heuristic to assert a shard-depth
(e.g. if we shard at each children of root node and most of the
content is under /content, then we won't save much)

Thanks,
Vikas

Strong all documents under Root - severe slowness on start-up

[Eugene Prystupa]

Can anyone confirm if this is a viable storage option? We have an
Oak-backed repository with ~100,000 documents in it (and growing), and all
documents are stored as children nodes of the root node.

We are seeing severe delays on start-up (20 minutes+) when repository is
created (new Jcr(oak).createRepository()). At appears that for whatever
reason Oak is trying to read all direct root children on start-up and that
takes a lot of time. Is storing documents not directly under root a better
choice?

-- 
Thanks,
Eugene

Re: Two-instance Oak cluster problems - data visibility latency

[Vikas Saurabh]

Hi Eugene,

>>> Is it possible for you to set up session/cookie based load balancer?
>
> This is probably possible, but before going through the trouble, I want to
> confirm that the behavior I described is by design (I hope not). I.e. does
> Jackrabbit Oak explicitly provides no guarantee on when the data is
> available in multi-instance cluster?

That (eventually consistent changes across cluster nodes) is indeed by
design. The background operations, that you were explicitly calling,
by default run in 1 s intervals. There are 2 background operations in
reality - backgroundUpdate and backgroundRead. "backgroundUpdate" one
essentially "publishes" (loosely speaking) the changes to be consumed
by other cluster nodes. "backgroundRead" one pull in "published"
changes by other cluster nodes. So, in a normally running system, in
worst cases, you can see 2s latency before the changes are visible
everywhere. The reason I said "normally running" is that background
threads might not get a chance every second on a busy setup.

Thanks,
Vikas

Re: Two-instance Oak cluster problems - data visibility latency

[Eugene Prystupa]

Thanks, Vikas.

>> Is it possible for you to set up session/cookie based load balancer?

This is probably possible, but before going through the trouble, I want to
confirm that the behavior I described is by design (I hope not). I.e. does
Jackrabbit Oak explicitly provides no guarantee on when the data is
available in multi-instance cluster?

On Thu, Feb 23, 2017 at 9:29 AM, Vikas Saurabh 
wrote:

> > We currently found a dirty workaround by
> > running DocumentNodewStore.runBackgroundOperations() on every new
> Session -
> > but this does seem like an intended way to work in a cluster.
> >
> Yes, that definitely is a bad idea - I don't recall the exact details
> atm, but I think background operations take exclusive locks so it'd
> drop concurrent access performance a lot. Moreover, I don't think
> background operations are supposed to be run concurrently - so, there
> might be other issue too.
>
> > HTTP servers are load balanced.
> >
> Is it possible for you to set up session/cookie based load balancer?
> For example the dispatcher (in AEM setups) sets up a rendid cookie so
> that related requests are sent same AEM instance.
>
> Thanks,
> Vikas
>



-- 
Thanks,
Eugene

Re: Two-instance Oak cluster problems - data visibility latency

[Vikas Saurabh]

> We currently found a dirty workaround by
> running DocumentNodewStore.runBackgroundOperations() on every new Session -
> but this does seem like an intended way to work in a cluster.
>
Yes, that definitely is a bad idea - I don't recall the exact details
atm, but I think background operations take exclusive locks so it'd
drop concurrent access performance a lot. Moreover, I don't think
background operations are supposed to be run concurrently - so, there
might be other issue too.

> HTTP servers are load balanced.
>
Is it possible for you to set up session/cookie based load balancer?
For example the dispatcher (in AEM setups) sets up a rendid cookie so
that related requests are sent same AEM instance.

Thanks,
Vikas

Two-instance Oak cluster problems - data visibility latency

[Eugene Prystupa]

I am wondering if anyone else experiences this problem, or have found a way
to deal with it...

We are using Jackrabbit Oak version 1.4.10, and running two HTTP
server-fronted instances in Prod. Both instances are configured to talk to
the same Mongo database. HTTP servers are load balanced.

The problems is when I create a node using instance 1, and after that
quickly access that same node using instance 2, instance 2 reports the node
as non-existent. It usually takes a second or two for the instance 2 to
"see" the added node.

This is very bad news for us, as create-and-immediately-retrieve pattern is
used a lot in our application. We currently found a dirty workaround by
running DocumentNodewStore.runBackgroundOperations() on every new Session -
but this does seem like an intended way to work in a cluster.

Any thoughts or suggestions appreciated.

-- 
Thanks,
Eugene

Oak 1.6.1 release plan

[Davide Giannella]

Hello team,

I'm planning to cut Oak on Thursday 2nd March.

If there are any objections please let me know. Otherwise I will
re-schedule any non-resolved issue for the next iteration.

Thanks
Davide

Re: CommitEditors looking for specific child node like oak:index, rep:cugPolicy leads to lots of redundant remote calls

[Chetan Mehrotra]

I realized now that I logged an issue for this recently OAK-5511 which
mentioned similar approach. So lets move this discussion there
Chetan Mehrotra


On Thu, Feb 23, 2017 at 7:06 PM, Thomas Mueller  wrote:
> Hi,
>
>>I like Marcel proposal for "enforcing" use of mixin on parent node to
>>indicate that it can have a child node of 'oak:index'. So we can
>>leverage mxin 'mix:indexable' (OAK-3725) to mark such parent nodes
>>(like root) and IndexUpdate would only look for 'oak:index' node if
>>current node has that mixin.
>
> Ah I didn't know about OAK-3725.
>
> I'm a bit worried that we mix different aspects together, not sure which
> is better.
>
> "oak:Indexable" is visible, so it can be added and _removed_ by the user.
> So when trying to remove that mixin, we would need to check there is no
> oak:index child node with nodetype oak:QueryIndexDefinition. We need to
> check the nodetype hierarchy. On the other hand, possibly we can enforce
> that the parent node of oak:index is oak:Indexable (can we?)
>
> I'm not saying with a hidden property hidden property ":hasOakIndex"
> (automatically set and removed) it would be painless. For example when
> moving an oak:index node to a new parent, the setting has to be changed at
> both the original and the new parents.
>
> Regards,
> Thomas
>
>

Re: Purge nodes from Oak

[Julian Reschke]

On 2017-02-23 14:38, Eugene Prystupa wrote:

Is there a way to permanently remove a node from Oak?

When I delete the node (java.jcr.Item.remove()), it is being marked as
deleted, but it's content and all revisions are still in the database. Is
there a way to purge it entirely?


It's supposed to be removed later on (*) by the VersionGarbageCollection.

Best regards, Julian

PS: between 24...48 hours with the standard AEM config settings

Purge nodes from Oak

[Eugene Prystupa]

Is there a way to permanently remove a node from Oak?

When I delete the node (java.jcr.Item.remove()), it is being marked as
deleted, but it's content and all revisions are still in the database. Is
there a way to purge it entirely?

-- 
Thanks,
Eugene

Re: CommitEditors looking for specific child node like oak:index, rep:cugPolicy leads to lots of redundant remote calls

[Thomas Mueller]

Hi,

>I like Marcel proposal for "enforcing" use of mixin on parent node to
>indicate that it can have a child node of 'oak:index'. So we can
>leverage mxin 'mix:indexable' (OAK-3725) to mark such parent nodes
>(like root) and IndexUpdate would only look for 'oak:index' node if
>current node has that mixin.

Ah I didn't know about OAK-3725.

I'm a bit worried that we mix different aspects together, not sure which
is better.

"oak:Indexable" is visible, so it can be added and _removed_ by the user.
So when trying to remove that mixin, we would need to check there is no
oak:index child node with nodetype oak:QueryIndexDefinition. We need to
check the nodetype hierarchy. On the other hand, possibly we can enforce
that the parent node of oak:index is oak:Indexable (can we?)

I'm not saying with a hidden property hidden property ":hasOakIndex"
(automatically set and removed) it would be painless. For example when
moving an oak:index node to a new parent, the setting has to be changed at
both the original and the new parents.

Regards,
Thomas

Re: CommitEditors looking for specific child node like oak:index, rep:cugPolicy leads to lots of redundant remote calls

[Chetan Mehrotra]

I like Marcel proposal for "enforcing" use of mixin on parent node to
indicate that it can have a child node of 'oak:index'. So we can
leverage mxin 'mix:indexable' (OAK-3725) to mark such parent nodes
(like root) and IndexUpdate would only look for 'oak:index' node if
current node has that mixin.

This would avoid the extra calls. For new setups we can enforce this
and for upgrade we can migrate the existing code by using nodetype
index to update all such "indexable" nodes
Chetan Mehrotra


On Thu, Feb 23, 2017 at 4:47 PM, Chetan Mehrotra
 wrote:
> On Wed, Feb 22, 2017 at 8:21 PM, Davide Giannella  wrote:
>> Did you mean for ALL the nodes, or only specific nodes?
>>
>> Any way you're suggesting something like the following flow:
>>
>> 1) user call nodebuilder.child(":index")
>> 2) lookup in hidden property
>> 3) if not there, leverage the existing code
>>
>> If so I guess the property has been already fetched and it does not
>> require roundtrips towards the DB. Am I right?
>
> Currently the lookup is being done for ALL nodes. So IndexUpdate class
> does following on each changed node
>
> --
>> @Override
> public void enter(NodeState before, NodeState after)
> throws CommitFailedException {
> collectIndexEditors(builder.getChildNode(INDEX_DEFINITIONS_NAME),
> before);
> --
>
> Which transalates into checking if the current node has a child node
> 'oak:index' and this leads to redudant calls.
>
> Chetan Mehrotra

Re: CommitEditors looking for specific child node like oak:index, rep:cugPolicy leads to lots of redundant remote calls

[Angela Schreiber]

hi

yes that mixin exists. but the CugValidatorProvider is performs additional
validation and doesn't allow for usage of the reserved names if the mixin
is not there... so we cannot limit the evaluation to nodes that have the
mixin type set (or inherited through super-type).
see http://jackrabbit.apache.org/oak/docs/security/authorization/cug.html
for details.

kind regards
angela

On 22/02/17 14:26, "Marcel Reutegger"  wrote:

>Hi,
>
>On 22/02/17 12:57, Chetan Mehrotra wrote:
>> One possible approach is to mark the parent with a specific hidden
>> property which has such a node upon addition. This would avoid the
>> negative lookup in case of updates
>
>Alternatively we could also mark the parent node with a mixin. This
>would be similar to nodes that can have a rep:policy child nodes. In
>that case the parent node is rep:AccessControllable.
>
>At least for rep:cugPolicy nodes this seems to be the case already. The
>node type registry of my Oak 1.6 repository says:
>
>[rep:CugMixin]
>   mixin
>   + rep:cugPolicy (rep:CugPolicy) protected ignore
>
>[rep:CugPolicy] > rep:Policy
>   - rep:principalNames (string) mandatory protected multiple ignore
>
>
>Regards
>  Marcel

Re: CommitEditors looking for specific child node like oak:index, rep:cugPolicy leads to lots of redundant remote calls

[Chetan Mehrotra]

On Wed, Feb 22, 2017 at 8:21 PM, Davide Giannella  wrote:
> Did you mean for ALL the nodes, or only specific nodes?
>
> Any way you're suggesting something like the following flow:
>
> 1) user call nodebuilder.child(":index")
> 2) lookup in hidden property
> 3) if not there, leverage the existing code
>
> If so I guess the property has been already fetched and it does not
> require roundtrips towards the DB. Am I right?

Currently the lookup is being done for ALL nodes. So IndexUpdate class
does following on each changed node

--
> @Override
public void enter(NodeState before, NodeState after)
throws CommitFailedException {
collectIndexEditors(builder.getChildNode(INDEX_DEFINITIONS_NAME),
before);
--

Which transalates into checking if the current node has a child node
'oak:index' and this leads to redudant calls.

Chetan Mehrotra

Re: CommitEditors looking for specific child node like oak:index, rep:cugPolicy leads to lots of redundant remote calls

[Thomas Mueller]

Hi,

For "oak:index" of type oak:QueryIndexDefinition, what about a hidden
property ":hasOakIndex" = true. That would be
NodeBuilder.hasProperty(":hasOakIndex").

Regards,
Thomas


On 22/02/17 12:57, "Chetan Mehrotra"  wrote:

>We have some CommitEditors in Oak which look for specific child node
>upon each commit like 'oak:index' and 'rep:cugPolicy'
>
>In most cases such child node does not exist and this leads to extra
>remote calls in case of DocumentNodeStore to determine if child with
>such a name exist or not. In case of updates to nodes where child data
>is not cached this quickly adds up and becomes a major portion of
>remote call made from Oak and something which we can avoid
>
>We should look into approaches where such child lookup can be avoided
>in critical write path.
>
>One possible approach is to mark the parent with a specific hidden
>property which has such a node upon addition. This would avoid the
>negative lookup in case of updates
>
>Chetan Mehrotra

[Oak origin/trunk] Apache Jackrabbit Oak matrix - Build # 1449 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build 
#1449)

Status: Still Failing

Check console output at 
https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1449/ to view 
the results.

Changes:
[stefanegli] OAK-5742 : making getListenerToString package-protected to limit 
its use

[mreutegg] OAK-5752: Remove duplicate code for background operation timing log

[thomasm] OAK-5624 Test failure:

[adulceanu] OAK-5600 - Test coverage for CheckCommand Added (disabled) test for

[reschke] OAK-5751: RDBDocumentStore: properly handle null values for system

[thomasm] OAK-5388 Test failure: persistentCache.BroadcastTest.broadcastTCP

[mreutegg] OAK-5761: Move commit value resolution to DocumentNodeStore

[reschke] OAK-5741: DocumentStore UpdateOp: support removal of properties

 

Test results:
2 tests failed.
FAILED:  
org.apache.jackrabbit.oak.security.authentication.ldap.LdapProviderTest.testSplitDNIntermediatePath2

Error Message:
ERR_171 Failed to bind an LDAP service (1,024) to the service registry.

Stack Trace:
org.apache.directory.api.ldap.model.exception.LdapConfigurationException: 
ERR_171 Failed to bind an LDAP service (1,024) to the service registry.
at 
org.apache.directory.server.ldap.LdapServer.startNetwork(LdapServer.java:695)
at 
org.apache.directory.server.ldap.LdapServer.start(LdapServer.java:547)
at 
org.apache.jackrabbit.oak.security.authentication.ldap.AbstractServer.setUp(AbstractServer.java:231)
at 
org.apache.jackrabbit.oak.security.authentication.ldap.InternalLdapServer.setUp(InternalLdapServer.java:33)
at 
org.apache.jackrabbit.oak.security.authentication.ldap.LdapProviderTest.before(LdapProviderTest.java:93)
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198)
at 
org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51)
at 

[ANNOUNCE] Apache Jackrabbit 2.8.5 released

[Julian Reschke]

The Apache Jackrabbit community is pleased to announce the release of
Apache Jackrabbit 2.8.5. The release is available for download at:

http://jackrabbit.apache.org/downloads.html

See the full release notes below for details about this release:

Release Notes -- Apache Jackrabbit -- Version 2.8.5

Introduction


This is Apache Jackrabbit(TM) 2.8.5, a fully compliant implementation of the
Content Repository for Java(TM) Technology API, version 2.0 (JCR 2.0) as
specified in the Java Specification Request 283 (JSR 283).

Apache Jackrabbit 2.8.5 is a patch release that contains fixes and
improvements over Jackrabbit 2.8. Jackrabbit 2.8.x releases are considered
stable and targeted for production use.

Changes since Jackrabbit 2.8.4
--

Improvement

[JCR-3801] - Improve ListenerTracker output
[JCR-3812] - Improvements to the GQL tools

Task

[JCR-4106] - fix JCR API docs link


In addition to the above-mentioned changes, this release contains
all the changes included up to the Apache Jackrabbit 2.8.3 release.

For more detailed information about all the changes in this and other
Jackrabbit releases, please see the Jackrabbit issue tracker at

https://issues.apache.org/jira/browse/JCR

Release Contents


This release consists of a single source archive packaged as a zip file.
The archive can be unpacked with the jar tool from your JDK installation.
See the README.txt file for instructions on how to build this release.

The source archive is accompanied by SHA1 and MD5 checksums and a PGP
signature that you can use to verify the authenticity of your download.
The public key used for the PGP signature can be found at
https://svn.apache.org/repos/asf/jackrabbit/dist/KEYS.

About Apache Jackrabbit
---

Apache Jackrabbit is a fully conforming implementation of the Content
Repository for Java Technology API (JCR). A content repository is a
hierarchical content store with support for structured and unstructured
content, full text search, versioning, transactions, observation, and
more.

For more information, visit http://jackrabbit.apache.org/

About The Apache Software Foundation


Established in 1999, The Apache Software Foundation provides organizational,
legal, and financial support for more than 140 freely-available,
collaboratively-developed Open Source projects. The pragmatic Apache License
enables individual and commercial users to easily deploy Apache software;
the Foundation's intellectual property framework limits the legal exposure
of its 3,800+ contributors.

For more information, visit http://www.apache.org/

Trademarks
--

Apache Jackrabbit, Jackrabbit, Apache, the Apache feather logo, and the 
Apache

Jackrabbit project logo are trademarks of The Apache Software Foundation.