Intent to backport OAK-8066 to 1.8 and 1.10

[Francesco Mari]

I intend to backport OAK-8066 to 1.8 and 1.10. The change prevents
excessive memory consumption, which can lead to OOME, when rebasing a node
with many children.

The backport is tracked by OAK-8559. I couldn't reopen the original issue,
so I created a new one to clearly signal that I want this change to be part
of the next releases on the 1.8 and 1.10 branches.

Re: Possible to open existing JCR repository using read-only FileStore?

[Francesco Mari]

The Jcr builder assumes write access to the repository. The read-only
FileStore is used in CLI utilities and tests that deal directly with the
NodeStore, completely bypassing the JCR API. As far as I know, we don't
have a way to create a read-only JCR Repository.

On Mon, 6 May 2019 at 10:49, Mark Adamcin  wrote:

> I'm trying to create a tool to export some JCR data (such as a list of
> nodetypes as a CND, and maybe even to run a query) from an existing oak
> repository directory without risking mutation of the filestore. While I am
> able to follow the patterns used by oak-run to get a NodeStoreFixture with
> a readonly FileStore, when I attempt to construct an Oak() instance and a
> Jcr() builder on top of it, the Oak.createNewContentRepository() method
> throws when it attempts to merge an "OakInitializer" commit to the
> SegmentNodeStore.
>
> Is it possible to avoid this commit to support a readonly JCR session? Or
> would this have to be mounted to a separate JCR repository using a
> CompositeNodeStore?
>
> Stack trace from oak 1.8.9:
>
> java.lang.UnsupportedOperationException: Cannot write to read-only store
>
> at
>
> org.apache.jackrabbit.oak.segment.DefaultSegmentWriterBuilder$1.execute(DefaultSegmentWriterBuilder.java:176)
> at
>
> org.apache.jackrabbit.oak.segment.DefaultSegmentWriter.writeNode(DefaultSegmentWriter.java:243)
> at
>
> org.apache.jackrabbit.oak.segment.SegmentWriter.writeNode(SegmentWriter.java:141)
> at
>
> org.apache.jackrabbit.oak.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:132)
> at
>
> org.apache.jackrabbit.oak.segment.scheduler.Commit.hasChanges(Commit.java:102)
> at
>
> org.apache.jackrabbit.oak.segment.scheduler.LockBasedScheduler.execute(LockBasedScheduler.java:258)
> at
>
> org.apache.jackrabbit.oak.segment.scheduler.LockBasedScheduler.schedule(LockBasedScheduler.java:236)
> at
>
> org.apache.jackrabbit.oak.segment.SegmentNodeStore.merge(SegmentNodeStore.java:195)
> at
> org.apache.jackrabbit.oak.OakInitializer.initialize(OakInitializer.java:54)
> at
> org.apache.jackrabbit.oak.Oak.createNewContentRepository(Oak.java:687)
> at org.apache.jackrabbit.oak.Oak.createContentRepository(Oak.java:675)
> at
> org.apache.jackrabbit.oak.jcr.Jcr.createContentRepository(Jcr.java:376)
> at org.apache.jackrabbit.oak.jcr.Jcr.createRepository(Jcr.java:385)
> at net.adamcin.oakpal.toolslib.JcrFactory.getJcr(JcrFactory.java:54)
>
> Thanks,
> Mark Adamcin
>

Re: [SUMMARY] Branching and release

[Francesco Mari]

On Thu, 28 Mar 2019 at 14:13, Davide Giannella  wrote:

> I saw we still have versions like 1.10. Do we want to overall go through
> a bit of sanitisation in Jira?
>

I think we should get rid of pseudo-versions like 1.10. I always found them
confusing. Why not using only concrete versions we are going to release?

Re: Why don't we move to Git?

[Francesco Mari]

Hi,

On Tue, 11 Dec 2018 at 10:56, Davide Giannella  wrote:

> Does git change always HEAD? Even if you merge something that fits
> "behind" HEAD.
>

Git always changes the head when a merge commit is performed. See [1] for
more details about how merging works in Git. I hope I understood your
concerns right.

[1]: https://www.atlassian.com/git/tutorials/using-branches/git-merge

Re: [Monitoring] - Provisioning StatisticsProvider in Oak

[Francesco Mari]

On Tue, 27 Nov 2018 at 11:45, Tommaso Teofili 
wrote:

> After having dig into the code for a while I can see following options:
> - have a dedicated OSGi service on the query side that can be then
> polled in a static way from QueryEngineImpl in order to obtain such a
> reference (see patch from OAK-7904 [1])
>

I suggest you don't pursue this option. I find that every service
introduces additional complexity and hinders the testability of the code.
In oak-segment-tar we pushed all the OSGi bits to a couple of services
(read below) and everything else is just plain Java code that explicitly
declares its dependencies. The API is clearer and the code is much more
testable. You don't want to resort on oak-it-osgi to test your stuff ;)


> - pass the StatisticsProvider instances down the whole stack from Oak
> class (or from MutableRoot/ImmubtableRoot)
>

In oak-segment-tar we opted for this option. We have two OSGi services that
represent the entry point for the oak-segment-tar bundle,
SegmentNodeStoreService and SegmentNodeStoreFactory. Both the services have
a reference to a StatisticsProvider, which is passed down the stack to the
bowels of the bundle.

Re: Intent to backport OAK-7867 to Oak 1.8 and 1.6

[Francesco Mari]

+1

On Mon, 19 Nov 2018 at 17:50, Michael Dürig  wrote:

> Hi,
>
> I intend to backport OAK-7867 to Oak 1.8 and 1.6. This fixes an issue
> that can cause sever data loss with the segment node store. There is a
> medium to high risk with this backport as it touches some of the core
> parts of the segment node store. To mitigate the risk we ran a 14 days
> longevity test internally (Adobe), which did not show any significant
> difference with any of the tracked metrics. Furthermore I plan to run
> each of those backports through the same longevity test before including
> them in a release.
>
> Michael
>

Re: Test failures in o.a.j.o.plugins.document

[Francesco Mari]

The problem seems to be caused by an incompatibility between the Docker
client shipped with docker-junit-rule and the latest version of Docker. I
created OAK-7874 to track this.

On Thu, 1 Nov 2018 at 11:26, Francesco Mari 
wrote:

> I experience multiple failures from tests in o.a.j.o.plugins.document. It
> looks like they have something to do with how the tests use Docker. I have
> Docker for Mac installed and working on my machine. Can somebody help with
> this?
>
> A sample of failures is the following:
>
> [INFO] Running
> org.apache.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteServiceCrashTest
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 0.072 s <<< FAILURE! - in
> org.apache.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteServiceCrashTest
> [ERROR]
> testTwoNodesWithCrashAndLongduringRecovery(org.apache.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteServiceCrashTest)
> Time elapsed: 0.035 s  <<< ERROR!
> java.lang.RuntimeException: Unable to start docker container:
> DockerConfig{name=MongoDB}
> Caused by: com.spotify.docker.client.exceptions.DockerException: Unable to
> pull docker image mongo:3.6
> Caused by: com.spotify.docker.client.exceptions.DockerException:
> com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException:
> Can not construct instance of
> com.spotify.docker.client.messages.RegistryAuth: no String-argument
> constructor/factory method to deserialize from String value ('swarm')
>  at [Source: N/A; line: -1, column: -1] (through reference chain:
> java.util.LinkedHashMap["stackOrchestrator"])
> Caused by:
> com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException:
> Can not construct instance of
> com.spotify.docker.client.messages.RegistryAuth: no String-argument
> constructor/factory method to deserialize from String value ('swarm')
>  at [Source: N/A; line: -1, column: -1] (through reference chain:
> java.util.LinkedHashMap["stackOrchestrator"])
>
> [ERROR]
> testTwoNodesWithCrashAndLongduringRecoveryAndBacklog(org.apache.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteServiceCrashTest)
> Time elapsed: 0.026 s  <<< ERROR!
> java.lang.RuntimeException: Unable to start docker container:
> DockerConfig{name=MongoDB}
> Caused by: com.spotify.docker.client.exceptions.DockerException: Unable to
> pull docker image mongo:3.6
> Caused by: com.spotify.docker.client.exceptions.DockerException:
> com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException:
> Can not construct instance of
> com.spotify.docker.client.messages.RegistryAuth: no String-argument
> constructor/factory method to deserialize from String value ('swarm')
>  at [Source: N/A; line: -1, column: -1] (through reference chain:
> java.util.LinkedHashMap["stackOrchestrator"])
> Caused by:
> com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException:
> Can not construct instance of
> com.spotify.docker.client.messages.RegistryAuth: no String-argument
> constructor/factory method to deserialize from String value ('swarm')
>  at [Source: N/A; line: -1, column: -1] (through reference chain:
> java.util.LinkedHashMap["stackOrchestrator"])
>
>

Test failures in o.a.j.o.plugins.document

[Francesco Mari]

I experience multiple failures from tests in o.a.j.o.plugins.document. It
looks like they have something to do with how the tests use Docker. I have
Docker for Mac installed and working on my machine. Can somebody help with
this?

A sample of failures is the following:

[INFO] Running
org.apache.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteServiceCrashTest
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
0.072 s <<< FAILURE! - in
org.apache.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteServiceCrashTest
[ERROR]
testTwoNodesWithCrashAndLongduringRecovery(org.apache.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteServiceCrashTest)
Time elapsed: 0.035 s  <<< ERROR!
java.lang.RuntimeException: Unable to start docker container:
DockerConfig{name=MongoDB}
Caused by: com.spotify.docker.client.exceptions.DockerException: Unable to
pull docker image mongo:3.6
Caused by: com.spotify.docker.client.exceptions.DockerException:
com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException:
Can not construct instance of
com.spotify.docker.client.messages.RegistryAuth: no String-argument
constructor/factory method to deserialize from String value ('swarm')
 at [Source: N/A; line: -1, column: -1] (through reference chain:
java.util.LinkedHashMap["stackOrchestrator"])
Caused by:
com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException:
Can not construct instance of
com.spotify.docker.client.messages.RegistryAuth: no String-argument
constructor/factory method to deserialize from String value ('swarm')
 at [Source: N/A; line: -1, column: -1] (through reference chain:
java.util.LinkedHashMap["stackOrchestrator"])

[ERROR]
testTwoNodesWithCrashAndLongduringRecoveryAndBacklog(org.apache.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteServiceCrashTest)
Time elapsed: 0.026 s  <<< ERROR!
java.lang.RuntimeException: Unable to start docker container:
DockerConfig{name=MongoDB}
Caused by: com.spotify.docker.client.exceptions.DockerException: Unable to
pull docker image mongo:3.6
Caused by: com.spotify.docker.client.exceptions.DockerException:
com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException:
Can not construct instance of
com.spotify.docker.client.messages.RegistryAuth: no String-argument
constructor/factory method to deserialize from String value ('swarm')
 at [Source: N/A; line: -1, column: -1] (through reference chain:
java.util.LinkedHashMap["stackOrchestrator"])
Caused by:
com.spotify.docker.client.shaded.com.fasterxml.jackson.databind.JsonMappingException:
Can not construct instance of
com.spotify.docker.client.messages.RegistryAuth: no String-argument
constructor/factory method to deserialize from String value ('swarm')
 at [Source: N/A; line: -1, column: -1] (through reference chain:
java.util.LinkedHashMap["stackOrchestrator"])

Re: Intend to backport OAK-7838

[Francesco Mari]

+1

On Tue, 30 Oct 2018 at 14:03, Michael Dürig  wrote:

>
>
> Hi,
>
> I intend to backport https://issues.apache.org/jira/browse/OAK-7838.
> This fixes an unclosed executor (by removing it). The fix only affects
> monitoring code and is thus rather low risk.
>
>
> Michael
>

Re: Intend to backport OAK-7837

[Francesco Mari]

+1

On Wed, 24 Oct 2018 at 11:42, Michael Dürig  wrote:

>
> Hi,
>
> I intend to backport https://issues.apache.org/jira/browse/OAK-7837.
> This is a simple fix in tooling (oak-run check) preventing it from crash
> under certain circumstances. The fix is simple and the risk is low.
>
> Michael
>

Re: Intend to backport OAK-7854

[Francesco Mari]

+1

On Wed, 24 Oct 2018 at 11:46, Michael Dürig  wrote:

>
> Hi,
>
> I intend to backport https://issues.apache.org/jira/browse/OAK-7854.
> This issues adds and additional monitoring endpoint to detect the case
> where the flush thread fails. Although the fix affects the file store,
> the changes are simple and low risk: addition of a timer update whenever
> the a flush is scheduled.
>
> Michael
>

Re: Jira component for oak-segmemt-azure

[Francesco Mari]

+1

On Thu, 4 Oct 2018 at 11:43, Michael Dürig  wrote:

>
> Hi,
>
> With more work going into the Azure Segment Store, should we start
> tracking this via a dedicated Jira component?
>
> If there are no objections I suggest to add a segment-azure component.
>
> Michael
>

Intent to backport OAK-6890

[Francesco Mari]

I intend to backport OAK-6890 to the 1.6 branch. The keeps some background
threads alive in the face of unexpected failures. All of these threads are
critical for the correctness of the Segment Store.

Intent to backport OAK-7721

[Francesco Mari]

I intend to backport OAK-7721 to the 1.8 and 1.6 branches. The fix prevents
the segment buffers from being corrupted when too big records are
persisted. The corruption is only detected when the buffer is flushed to
disk, when it's too late to detect the code path that led to the
corruption. The fix fails earlier and louder, preventing the segment buffer
from being corrupted and allowing us to identify the defective code path.

Intent to backport OAK-7720

[Francesco Mari]

I intend to backport OAK-7720 to the 1.8 and 1.6 branch. The fix makes the
system log a dump of too big segments to ease debugging the corruption of
the segment buffer.

Re: Intent to backport OAK-6648

[Francesco Mari]

+1

On Fri, 17 Aug 2018 at 09:09, Michael Dürig  wrote:

>
> Hi,
>
> I would like to backport OAK-6648 to Oak 1.6. This fixes an issue with
> offline revision cleanup causing tar files to not being removed under
> some circumstances.
>
> The risk is low as it only contains minor changes to code that is
> executed when the repository is shut down.
>
> Michael
>
>
>

Re: Release plan Oak 1.9.7

[Francesco Mari]

On Tue, 7 Aug 2018 at 20:14, Manfred Baedke 
wrote:

> Hi Francesco,
>
> are you sure you need the factoryPid?
>

You are right. There is no need to repeat the PID in the `factoryPid`
attribute in `@ObjectClassDefinition`. It's sufficient to set `factory` to
true in `@Designate`. In summary, the fix seems to be to drop the `factory`
attribute from `@Component`.


> I found this on migration of config factories from Felix to new OSGi
> annotations:
>
> Felix variation:
> https://techrevel.blog/2017/04/12/felix-configuration-factory/
> OSGi variation:
> https://techrevel.blog/2017/08/23/osgi-annotations-configuration-factory/
>
> On 8/7/2018 8:02 PM, Francesco Mari wrote:
>
> Hi Manfred,
>
> Thanks for you suggestion. I'm currently experimenting with dropping the
> `factory` attribute from `@Component` and setting the `factoryPid`
> attribute in `@ObjectClassDefinition`. This seems to produce the wanted
> result in both the component and metatype descriptors. I will run more
> tests tomorrow.
>
> On Tue, 7 Aug 2018 at 19:56, Manfred Baedke 
> wrote:
>
>> Hi Francesco,
>>
>> I think that you don't want to make this a factory component but instead
>> you want to use a factory configuration (the difference being a
>> component lifecycle control thing, if I got that right - ask a real OSGi
>> expert :)). Just try dropping the factory attribute from the Component
>> annotation and you should be fine.
>>
>> On 8/7/2018 6:32 PM, Francesco Mari wrote:
>> > The weird situation I'm looking into is that before my commit (e.g. at
>> > 01c07cfec2), the component descriptor for SegmentNodeStoreFactory,
>> > generated with the old annotations, contained a declaration like the
>> > following (whitespaces included for clarity):
>> >
>> > > >  xmlns:scr="http://www.osgi.org/xmlns/scr/v1.2.0;
>> >  name="org.apache.jackrabbit.oak.segment.SegmentNodeStoreFactory"
>> >  configuration-policy="require"
>> >  activate="activate"
>> >  deactivate="deactivate">
>> >
>> > After my commit, in trunk, using the new annotations, the component
>> > descriptor contains the following:
>> >
>> > > >  xmlns:scr="http://www.osgi.org/xmlns/scr/v1.3.0;
>> >
>> name="org.apache.jackrabbit.oak.segment.SegmentNodeStoreFactory"
>> >  configuration-policy="require"
>> >
>> factory="org.apache.jackrabbit.oak.segment.SegmentNodeStoreFactory"
>> >  activate="activate"
>> >  deactivate="deactivate">
>> >
>> > According to the declarative services specification [1], "a factory
>> > component is used if the 'factory' attribute of the 'component' element
>> is
>> > set to a factory identifier". Isn't then my commit actually fixing the
>> > component descriptor?
>> >
>> > [1]:
>> >
>> https://osgi.org/specification/osgi.cmpn/7.0.0/service.component.html#service.component-factorycomponent
>> >
>> >
>> > On Tue, 7 Aug 2018 at 16:00, Francesco Mari 
>> > wrote:
>> >
>> >> The issue is weird. The activate method of SegmentNodeStoreFactory is
>> not
>> >> even called. As far as I know, all the other tests in oak-pojosr are
>> >> working correctly. I will look into it.
>> >>
>> >> On Tue, 7 Aug 2018 at 15:19, Marcel Reutegger
>>  
>> >> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> On 07.08.18 14:57, Davide Giannella wrote:
>> >>>> Hello team, there are no blockers for the issues as of now. If none
>> will
>> >>>> come I'll proceed with the cut tomorrow Wednesday 8th August.
>> >>> I'd like to highlight OAK-6770. I just re-opened that issue, which has
>> >>> changes that would go into the release.
>> >>>
>> >>> Regards
>> >>>Marcel
>> >>>
>>
>>
>

Re: Release plan Oak 1.9.7

[Francesco Mari]

Hi Manfred,

Thanks for you suggestion. I'm currently experimenting with dropping the
`factory` attribute from `@Component` and setting the `factoryPid`
attribute in `@ObjectClassDefinition`. This seems to produce the wanted
result in both the component and metatype descriptors. I will run more
tests tomorrow.

On Tue, 7 Aug 2018 at 19:56, Manfred Baedke 
wrote:

> Hi Francesco,
>
> I think that you don't want to make this a factory component but instead
> you want to use a factory configuration (the difference being a
> component lifecycle control thing, if I got that right - ask a real OSGi
> expert :)). Just try dropping the factory attribute from the Component
> annotation and you should be fine.
>
> On 8/7/2018 6:32 PM, Francesco Mari wrote:
> > The weird situation I'm looking into is that before my commit (e.g. at
> > 01c07cfec2), the component descriptor for SegmentNodeStoreFactory,
> > generated with the old annotations, contained a declaration like the
> > following (whitespaces included for clarity):
> >
> >  >  xmlns:scr="http://www.osgi.org/xmlns/scr/v1.2.0;
> >  name="org.apache.jackrabbit.oak.segment.SegmentNodeStoreFactory"
> >  configuration-policy="require"
> >  activate="activate"
> >  deactivate="deactivate">
> >
> > After my commit, in trunk, using the new annotations, the component
> > descriptor contains the following:
> >
> >  >  xmlns:scr="http://www.osgi.org/xmlns/scr/v1.3.0;
> >  name="org.apache.jackrabbit.oak.segment.SegmentNodeStoreFactory"
> >  configuration-policy="require"
> >
> factory="org.apache.jackrabbit.oak.segment.SegmentNodeStoreFactory"
> >  activate="activate"
> >  deactivate="deactivate">
> >
> > According to the declarative services specification [1], "a factory
> > component is used if the 'factory' attribute of the 'component' element
> is
> > set to a factory identifier". Isn't then my commit actually fixing the
> > component descriptor?
> >
> > [1]:
> >
> https://osgi.org/specification/osgi.cmpn/7.0.0/service.component.html#service.component-factorycomponent
> >
> >
> > On Tue, 7 Aug 2018 at 16:00, Francesco Mari 
> > wrote:
> >
> >> The issue is weird. The activate method of SegmentNodeStoreFactory is
> not
> >> even called. As far as I know, all the other tests in oak-pojosr are
> >> working correctly. I will look into it.
> >>
> >> On Tue, 7 Aug 2018 at 15:19, Marcel Reutegger
> 
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> On 07.08.18 14:57, Davide Giannella wrote:
> >>>> Hello team, there are no blockers for the issues as of now. If none
> will
> >>>> come I'll proceed with the cut tomorrow Wednesday 8th August.
> >>> I'd like to highlight OAK-6770. I just re-opened that issue, which has
> >>> changes that would go into the release.
> >>>
> >>> Regards
> >>>Marcel
> >>>
>
>

Re: Release plan Oak 1.9.7

[Francesco Mari]

The weird situation I'm looking into is that before my commit (e.g. at
01c07cfec2), the component descriptor for SegmentNodeStoreFactory,
generated with the old annotations, contained a declaration like the
following (whitespaces included for clarity):

http://www.osgi.org/xmlns/scr/v1.2.0;
name="org.apache.jackrabbit.oak.segment.SegmentNodeStoreFactory"
configuration-policy="require"
activate="activate"
deactivate="deactivate">

After my commit, in trunk, using the new annotations, the component
descriptor contains the following:

http://www.osgi.org/xmlns/scr/v1.3.0;
name="org.apache.jackrabbit.oak.segment.SegmentNodeStoreFactory"
configuration-policy="require"
factory="org.apache.jackrabbit.oak.segment.SegmentNodeStoreFactory"
activate="activate"
deactivate="deactivate">

According to the declarative services specification [1], "a factory
component is used if the 'factory' attribute of the 'component' element is
set to a factory identifier". Isn't then my commit actually fixing the
component descriptor?

[1]:
https://osgi.org/specification/osgi.cmpn/7.0.0/service.component.html#service.component-factorycomponent


On Tue, 7 Aug 2018 at 16:00, Francesco Mari 
wrote:

> The issue is weird. The activate method of SegmentNodeStoreFactory is not
> even called. As far as I know, all the other tests in oak-pojosr are
> working correctly. I will look into it.
>
> On Tue, 7 Aug 2018 at 15:19, Marcel Reutegger 
> wrote:
>
>> Hi,
>>
>> On 07.08.18 14:57, Davide Giannella wrote:
>> > Hello team, there are no blockers for the issues as of now. If none will
>> > come I'll proceed with the cut tomorrow Wednesday 8th August.
>>
>> I'd like to highlight OAK-6770. I just re-opened that issue, which has
>> changes that would go into the release.
>>
>> Regards
>>   Marcel
>>
>

Re: Release plan Oak 1.9.7

[Francesco Mari]

The issue is weird. The activate method of SegmentNodeStoreFactory is not
even called. As far as I know, all the other tests in oak-pojosr are
working correctly. I will look into it.

On Tue, 7 Aug 2018 at 15:19, Marcel Reutegger 
wrote:

> Hi,
>
> On 07.08.18 14:57, Davide Giannella wrote:
> > Hello team, there are no blockers for the issues as of now. If none will
> > come I'll proceed with the cut tomorrow Wednesday 8th August.
>
> I'd like to highlight OAK-6770. I just re-opened that issue, which has
> changes that would go into the release.
>
> Regards
>   Marcel
>

Re: Intend to backport OAK-7132 to 1.8

[Francesco Mari]

+1

On Mon, Jan 22, 2018 at 12:24 PM, Michael Dürig  wrote:
>
> Hi,
>
> I intent to backport OAK-7132 to Oak 1.8. This is a fix to a critical error
> in the TarMK persistence layer that could lead to data loss. The fix was
> evaluated in trunk through an internal (to Adobe) longevity test for 5 day.
>
> Michael

Intent to backport OAK-{7168,7169,7171} to 1.8

[Francesco Mari]

I intend to backport OAK-{7168,7169,7171} to the 1.8 branch. The three
issues fix the behaviour of the debug, datastorecheck, and history
command when it comes to error handling. The fix make these commands
return with a non-zero exit code in case of error and print error
message to standard error instead of standard output.

Intent to backport OAK-7157 to 1.8

[Francesco Mari]

I intend to backport OAK-7157 to the 1.8 branch. The fix implements an
optimisation for cold standby instances. With the fix in place,
standby instances only retained the latest generation, instead of the
last two generations. This allows a cold standby instance to remove
old segments more aggressively and save space on disk.

Intent to backport OAK-7158 to 1.8

[Francesco Mari]

I intend to backport OAK-7158 to the 1.8 branch. The fix is about
disallowing users from changing the number of generations retained by
the FileStore. Setting the number of retained generations to a value
different than its default might cause data loss due to the way
cleanup works.

Re: [VOTE] Release Apache Jackrabbit Oak 1.6.7

[Francesco Mari]

+1

On Tue, Nov 28, 2017 at 10:46 AM, Alex Deparvu  wrote:
> [X] +1 Release this package as Apache Jackrabbit Oak 1.6.7
>
>
> alex
>
> On Tue, Nov 28, 2017 at 10:15 AM, Davide Giannella 
> wrote:
>
>> A candidate for the Jackrabbit Oak 1.6.7 release is available at:
>>
>> https://dist.apache.org/repos/dist/dev/jackrabbit/oak/1.6.7/
>>
>> The release candidate is a zip archive of the sources in:
>>
>>
>> https://svn.apache.org/repos/asf/jackrabbit/oak/tags/jackrabbit-oak-1.6.7/
>>
>> The SHA1 checksum of the archive is
>> ce013aac505d593e74547d71849cdb07197e2ba7.
>>
>> A staged Maven repository is available for review at:
>>
>> https://repository.apache.org/
>>
>> The command for running automated checks against this release candidate is:
>>
>> $ sh check-release.sh oak 1.6.7 ce013aac505d593e74547d71849cdb
>> 07197e2ba7
>>
>> Please vote on releasing this package as Apache Jackrabbit Oak 1.6.7.
>> The vote is open for the next 72 hours and passes if a majority of at
>> least three +1 Jackrabbit PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Jackrabbit Oak 1.6.7
>> [ ] -1 Do not release this package because...
>>
>> D.
>>

Re: identify abandoned oak modules

[Francesco Mari]

I'm in favour of retiring oak-remote. It is not currently used and it
didn't receive much attention in the recent past.

On Tue, Nov 21, 2017 at 3:56 PM, Angela Schreiber
 wrote:
> hi oak devs
>
> looking at the list of modules we have in oak/trunk i get the impression
> that some are not actively worked on or maintained.
> would it make sense or be possible to retire some of the modules that were
> originally started for productive usage and have been abandoned in the
> mean time?
>
> kind regards
> angela
>

Intent to backport OAK-6784

[Francesco Mari]

I would like to backport OAK-6784 to 1.6. The Compact tool backend
swallows the exception thrown during its execution. The issue is about
propagating the exception forward, so that the tool frontend might
handle them properly.

Re: Intent to backport to 1.6 OAK-6931

[Francesco Mari]

+1

On Tue, Nov 14, 2017 at 12:19 PM, Michael Dürig  wrote:
> https://issues.apache.org/jira/browse/OAK-6931
>
> This fixes an issue in the offline compaction tool, which prevents the cache
> size to be set via the command line.
>
> Michael

Re: OAK-6575 - A word of caution

[Francesco Mari]

On Thu, Sep 7, 2017 at 11:05 AM, Ian Boston  wrote:
> On 7 September 2017 at 07:22, Ian Boston  wrote:
>
>> Hi,
>>
>> On 6 September 2017 at 22:43, Michael Dürig  wrote:
>>
>>>
>>>
>>> On 06.09.17 23:08, Michael Dürig wrote:
>>>

 Hi,

 On 05.09.17 14:09, Ian Boston wrote:

> Repeating the comment to on OAK-6575 here for further discussion. 2 new
> Patches exploring both options.
>

 I would actually prefer the original patch (
 https://github.com/ieb/jackrabbit-oak/compare/trunk...ieb:O
 AK-6575?expand=1) in most parts. However I have concerns regarding the
 generality of the new OakConversionService API as mentioned in my previous
 mail. I would be more comfortable if this could be restricted to something
 that resembles more like a "URIProvider", which given a blob returns an 
 URI.

 On the implementation side, why do we need to introduce the adaptable
 machinery? Couldn't we re-use the Whiteboard and OSGiWhiteBoard mechanisms
 instead? I think these could be used to track URIProvider instances
 registered by the various blob stores.


>>> See https://github.com/mduerig/jackrabbit-oak/commit/2709c097b01
>>> a006784b7011135efcbbe3ce1ba88 for a *really* quickly hacked together and
>>> entirely untested POC. But it should get the idea across though.
>>
>>
>>
>> Thank you.
>> That makes sense.
>> I think it only needs the  java/org/apache/jackrabbit/
>> oak/blob/cloud/aws/s3/CloudFrontS3SignedUrlAdapterFactory.java and the
>> API to be inside Oak, everything else can be in Sling.
>> I'll update my patch and do a 2 options for Sling.
>>
>
>
>
>
> https://github.com/ieb/jackrabbit-oak/compare/trunk...ieb:OAK-6575-3?expand=1
>
> and
>
> https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3?expand=1
>
> wdyt ?

I like this a lot. It keeps Oak's side simple and cleanly integrates
Oak's lower-level services in Sling.

> Obviously the second patch needs to be discussed with Sling dev, but is
> should not be too contentious.
>
> Best Regards
> Ian
>
>
>
>>
>> I think that should address others concerns since it drops all signs of
>> any generic object to object conversion from Oak (Francesco), and doesn't
>> require wide scale fragile changes with implied requirements being placed
>> on how intermediate classes are connected and behave (mine).
>>
>> Best Regards
>> Ian
>>
>>
>>>
>>> Michael
>>>
>>
>>

Re: OAK-6575 - A word of caution

[Francesco Mari]

On Wed, Sep 6, 2017 at 1:59 PM, Ian Boston <i...@tfd.co.uk> wrote:
> Hi,
> Thanks for looking at them.
>
> On 6 September 2017 at 12:32, Francesco Mari <mari.france...@gmail.com>
> wrote:
>
>> I personally prefer the second approach.
>
>
> Is that OAK-6575-1 or OAK-6575-2 ?
> I assume OAK-6575-1 since OAK-6575 was my first approach ?

It is OAK-6575-2.

> If you mean OAK-6575-2, then I think someone with more knowledge of Oak
> will need to do the work as I am not at all confident I have covered the
> potential class/method navigation between a OakValue and a DataStore or
> if that navigation is even possible where the exposed datastore might
> actually be a composite datastore with the exposed part having no class
> based connection with the underlying S3 DataStore. (eg S3 DS cache). I am
> definitely not proud of OAK-6575-2, imho it's not elegant or efficient and
> would put up more barriers to future agility rather than remove them.

The rest of the team will surely give an opinion on the original
approach and the OAK-6575-1 and OAK-6575-2 variants. We will go with
whatever the majority prefers.

>> The only thing I'm not sure
>> about is if we want to define OakConversionService with such a
>> wildcard method. Assuming that OakConversionService will be called
>> from code running on top of the JCR API, we could provide instead more
>> specific conversion methods. For example,
>>
>> URI toURI(javax.jcr.Binary binary);
>>
>> What do you think about it? Is it too restrictive? Do we need a
>> wildcard method like currently defined in OakConversionsService?
>>
>
>
> Originally OakConversionsService would not have needed a new version of the
> package for each new conversion Oak supported, greatly simplifying
> dependencies downstream, especially where the source and target classes
> already exist.
>
> If a concrete method is used, the package will need to be versioned
> everytime. I suspect OSGi rules will require a minor version number
> increment each time, which is going to make a downstream developers life
> painful.
>
> In addition if an implementation bundle in Oak decides it wants to
> optionally support a conversion, it wont need to version the Oak API to
> achieve that. With concrete methods, ever change, wherever they are and
> however experimental will require a new version of the Oak API.
>
> This was the reason for going for a wildcard method. It allows extension
> without any downstream disruption, missing dependencies or out of band
> dependencies.

I understand the concerns. I wonder how many conversions we need to
implement, and how often we will introduce new conversions.

> I think the boils down to how much disruption Oak wants to inflict
> downstream to get new capabilities added, or inversely, how open Oak is to
> requests for API changes from downstream ?
>
>
>
>>
>> Moreover, I would leave PrivateURI out of the picture for the moment
>> since it's not clear from the patch how this is supposed to be used.
>> In fact, a comment in S3Backend explicitly states that is not
>> supported at this time.
>>
>
> PrivateURI was discussed on the OAK-6575 thread. It was added to the patch
> to illustrate how each patch would cope with extension of a new type. I
> propose to drop it from the final patch, however, in the second patch the
> disruption is quite large so it might be worth leaving it in there so that
> it can be implemented without more Oak API version changes.
>
> Best Regards
> Ian
>
>
>>
>> Finally, I suspect that in the second patch there was too much of an
>> aggressive rename refactoring. "types" was renamed to "customtypes" in
>> a lot of unrelated places. I would definitely double-check that.
>
>
>> On Tue, Sep 5, 2017 at 2:09 PM, Ian Boston <i...@tfd.co.uk> wrote:
>> > Hi,
>> >
>> > Repeating the comment to on OAK-6575 here for further discussion. 2 new
>> > Patches exploring both options.
>> >
>> > https://github.com/ieb/jackrabbit-oak/compare/trunk..
>> .ieb:OAK-6575-1?expand=1
>> >
>> > This drops the OSGi AdapterManager/AdapterFactory in favour of a non OSGi
>> > static pattern. Implementations of the AdapterFactory self register
>> rather
>> > than rely on OSGi doing the wiring. This is probably an IoC anti pattern,
>> > but does avoid exposing the AdapterFactory/AdapterManager outside Oak.
>> >
>> > https://github.com/ieb/jackrabbit-oak/compare/trunk..
>> .ieb:OAK-6575-2?expand=1
>> >
>> > This drops the AdapterManager concept completely and attempts to get from
>&g

Re: OAK-6575 - A word of caution

[Francesco Mari]

I personally prefer the second approach. The only thing I'm not sure
about is if we want to define OakConversionService with such a
wildcard method. Assuming that OakConversionService will be called
from code running on top of the JCR API, we could provide instead more
specific conversion methods. For example,

URI toURI(javax.jcr.Binary binary);

What do you think about it? Is it too restrictive? Do we need a
wildcard method like currently defined in OakConversionsService?

Moreover, I would leave PrivateURI out of the picture for the moment
since it's not clear from the patch how this is supposed to be used.
In fact, a comment in S3Backend explicitly states that is not
supported at this time.

Finally, I suspect that in the second patch there was too much of an
aggressive rename refactoring. "types" was renamed to "customtypes" in
a lot of unrelated places. I would definitely double-check that.

On Tue, Sep 5, 2017 at 2:09 PM, Ian Boston <i...@tfd.co.uk> wrote:
> Hi,
>
> Repeating the comment to on OAK-6575 here for further discussion. 2 new
> Patches exploring both options.
>
> https://github.com/ieb/jackrabbit-oak/compare/trunk...ieb:OAK-6575-1?expand=1
>
> This drops the OSGi AdapterManager/AdapterFactory in favour of a non OSGi
> static pattern. Implementations of the AdapterFactory self register rather
> than rely on OSGi doing the wiring. This is probably an IoC anti pattern,
> but does avoid exposing the AdapterFactory/AdapterManager outside Oak.
>
> https://github.com/ieb/jackrabbit-oak/compare/trunk...ieb:OAK-6575-2?expand=1
>
> This drops the AdapterManager concept completely and attempts to get from
> Value to URI using mix in interfaces and instanceof. I cant be certain it
> manages to do this as there is a disconnect between Blob, Blobstore and
> DataStore implementations with no guarantee that a BlobStore as seen by the
> Blob implementation actually implements DataStore, or the Blob that is
> exposed in the JCR Value (implemented by OakValue) actually connects to the
> correct DataStore of it it connects to a FileDatastore cache on local disk.
> I could only wire this as far as I did with API changes. I may have broken
> some of the new multi node store and multi datastore code used for 0DT in
> the process. An Oak committer with global knowledge will probably be able
> to do better.
>
>
>
> On 5 September 2017 at 08:19, Ian Boston <i...@tfd.co.uk> wrote:
>
>> Hi,
>>
>> On 5 September 2017 at 07:55, Francesco Mari <mari.france...@gmail.com>
>> wrote:
>>
>>> On Mon, Sep 4, 2017 at 6:18 PM, Ian Boston <i...@tfd.co.uk> wrote:
>>> > Do you mean:
>>> >  keep the OakConversionService but put all the logic to convert from a
>>> > Value to a URI inside that implementation using new Oak SPI/APIs if
>>> > necessary and drop the AdapterManager completely ?
>>>
>>> Yes. I think there is no need to provide a generic adapter-like
>>> implementation to solve this use case.
>>>
>>> > This would mean something the datastore implementation implements which
>>> > oak-core can navigate to would have to implement a mix in interface
>>> with a
>>> > getURI() method. I am not certain what or how without trying to do it.
>>> >
>>> > Would that address your concern here ?
>>>
>>> I think it's worth trying. Thanks for bringing the conversation forward.
>>>
>>
>>
>> I will create 2 new branches.
>> 1 with no adapter manager relying on mixin interfaces and one with a non
>> OSGi adapter manager plugin pattern.
>>
>> Thanks for the input.
>> Best Regards
>> Ian
>>
>>

Re: OAK-6575 - A word of caution

[Francesco Mari]

On Mon, Sep 4, 2017 at 6:18 PM, Ian Boston  wrote:
> Do you mean:
>  keep the OakConversionService but put all the logic to convert from a
> Value to a URI inside that implementation using new Oak SPI/APIs if
> necessary and drop the AdapterManager completely ?

Yes. I think there is no need to provide a generic adapter-like
implementation to solve this use case.

> This would mean something the datastore implementation implements which
> oak-core can navigate to would have to implement a mix in interface with a
> getURI() method. I am not certain what or how without trying to do it.
>
> Would that address your concern here ?

I think it's worth trying. Thanks for bringing the conversation forward.

Re: OAK-6575 - A word of caution

[Francesco Mari]

On Mon, Sep 4, 2017 at 4:41 PM, Ian Boston  wrote:
> If Oak committers feel that the contribution can't be included. Please feel
> free to close OAK-6575 and I will delete the GitHub branch.

Please don't! I'm not here to criticise your work, quite the contrary.
The idea is good, our users seem to need such a feature and we need to
provide an implementation. My personal opinions about the *need* for
this feature are, well, personal. With this thread I only wanted to
remind to the team that we had a similar conversation in the past
about a similar proposal and we didn't really like part of it. As you
stated, you couldn't know it. This is yet another reason why it's
often useful to keep older conversations going, instead of starting
new, seemingly fresh ones.

Re: OAK-6575 - A word of caution

[Francesco Mari]

On Mon, Sep 4, 2017 at 4:57 PM, Ian Boston <i...@tfd.co.uk> wrote:
> Hi,
> IIUC There are 2 patterns:
>
> 1 Emitting a short lived signed URL as per the AWS CloudFront recommended
> method of serving private content.

I have nothing to object to choice of AWS CloudFront.  The past
technological choices were very different. At the time, we were taking
about converting a binary to an S3 bucket ID for users to manipulate.
This had, as you could understand, a bigger impact on Oak from the
point of view of security and data ownership.

I am still of the opinion, though, that if S3 and CloudFront are part
of the technological stack chosen by a user, it's the user's
responsibility to directly interact with it. Oak can transparently
manage binaries in S3 for you, as long as it remains transparent. If
as a user you need to manipulate buckets, or create CloudFront
distributions, or use the AWS infrastructure to replicate S3 buckets
in a different zone, then this is part of your business logic. Oak can
still be used to store metadata and S3 bucket IDs, but the management
of S3 and CloudFront it is up to you. This is just my view on the
problem. If our users desperately want Oak to transparently store data
in S3 but also to opt out when needed, we are going to provide this
functionality. Still, my point of view is that it's wrong.

> 2 An Oak internal AdapterFactory/AdapterManager pattern to avoid Oak API.
> changes.

Sling adapters or a partial implementation of it is too dangerous in
its flexibility. Today we are using this system to solve only one
problem. Tomorrow we are going to end up like in Sling, where
everything might be adapted to everything else and module boundaries
will be more difficult to enforce. Sling adapters require too much
discipline to get right and are too easy to misuse.

Moreover, as soon as you register an AdapterManager in OSGi people are
going to get curious. There is no real control in OSGi unless you use
subsystems, and we are not using them. The most widely used commercial
products based on Oak are not using OSGi subsystems either, and this
is just going to exacerbate the problem.

My take on this is that it's alright to have something like a
OakConversionService, but please don't use a solution based on or
inspired to Sling adapters to implement it. It's easier, safer and
more controllable to implement this solution ad-hoc.

> Would you be willing state your concerns for each one separately ?
>
> Best Regards
> Ian
>
> On 4 September 2017 at 15:43, Francesco Mari <mari.france...@gmail.com>
> wrote:
>
>> I'm in no position to veto the POC and I'm not willing to. I am well
>> aware of the importance of this feature. I expressed my concerns and
>> so did others. As the subject of this thread clearly stated, I only
>> wanted to point out that I had the feeling that we had a "reboot" of
>> the conversation for no good reason, with the unpleasant side effect
>> of proposing once again a pattern that received a lot of criticism in
>> the past.
>>
>> On Mon, Sep 4, 2017 at 4:18 PM, Bertrand Delacretaz
>> <bdelacre...@apache.org> wrote:
>> > On Mon, Sep 4, 2017 at 3:44 PM, Ian Boston <i...@tfd.co.uk> wrote:
>> >> ...I feel
>> >> that Oak is weaker without the ability to offload bulk data streaming to
>> >> infrastructure designed for that purpose
>> >
>> > FWIW as an Oak user I share that feeling, IMO the use cases described
>> > at https://wiki.apache.org/jackrabbit/JCR%20Binary%20Usecase are
>> > becoming more and more important.
>> >
>> > Not being a committer I don't really care about the internals, but
>> > please do not "throw the baby out with the bath water" if the
>> > internals need to change.
>> >
>> > -Bertrand
>>

Re: OAK-6575 - A word of caution

[Francesco Mari]

I'm in no position to veto the POC and I'm not willing to. I am well
aware of the importance of this feature. I expressed my concerns and
so did others. As the subject of this thread clearly stated, I only
wanted to point out that I had the feeling that we had a "reboot" of
the conversation for no good reason, with the unpleasant side effect
of proposing once again a pattern that received a lot of criticism in
the past.

On Mon, Sep 4, 2017 at 4:18 PM, Bertrand Delacretaz
 wrote:
> On Mon, Sep 4, 2017 at 3:44 PM, Ian Boston  wrote:
>> ...I feel
>> that Oak is weaker without the ability to offload bulk data streaming to
>> infrastructure designed for that purpose
>
> FWIW as an Oak user I share that feeling, IMO the use cases described
> at https://wiki.apache.org/jackrabbit/JCR%20Binary%20Usecase are
> becoming more and more important.
>
> Not being a committer I don't really care about the internals, but
> please do not "throw the baby out with the bath water" if the
> internals need to change.
>
> -Bertrand

OAK-6575 - A word of caution

[Francesco Mari]

The POC for OAK-6575 aims at introducing the AdapterManager and
AdapterFactory API in Oak. This is just a step away from introducing a
full-fledged implementation of Sling-like adapters in Oak. In fact,
the OakConversionService is just an Adapter for a single, very
specific conversion.

We had a very similar conversation for OAK-1963 in a thread [1] in
this mailing list. In that thread many people raised that concerns
about adapters. Many of us, me included, see adapters as a glorified
way of breaking data encapsulation and are not in favour of
introducing them.

After some quiet time in the thread and in OAK-1963, a new thread [2]
was created, a new issue [3] was opened and a POC [4] was written. The
content of the POC is little more than a permutation of what was
already discussed in OAK-1963. Why a new thread and issue were needed?
Why couldn't we discuss this implementation in the light of the
OAK-1963? Why the old concerns are not taken into consideration once
again? We already talked about all that.

If the majority of the people here is alright in going forward with
the aforementioned POC, so be it. But, at least, let's avoid closing
threads and issues and create new ones that are just carbon copies.
This might be misinterpreted as a way to clear concerns out of our
way.

[1]: http://markmail.org/thread/enmibgfwedypjnos
[2]: 
https://lists.apache.org/thread.html/2efb1da82b9b93a3a394c3abccbac960175ffc7a13facaac79c9181a@%3Coak-dev.jackrabbit.apache.org%3E
[3]: https://issues.apache.org/jira/browse/OAK-6575
[4]: 
https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:OAK-6575?expand=1

Re: Backporting cold standby chunking to 1.6.x?

[Francesco Mari]

I wouldn't backport unless strictly necessary. In my opinion, this is
not a bug but an improvement.

On Mon, Aug 21, 2017 at 9:03 AM, Andrei Dulceanu
 wrote:
> Hi all,
>
> With [0] and [1] blob chunking in cold standby was addressed in 1.8. I
> think now we have a stable and robust solution which got rid of the 2.14
> GB/blob limitation. As a positive side-effect, the memory footprint needed
> for a successful sync of a big blob reduced considerably. While previously
> 4GB of heap memory were needed for syncing 1GB blob, now only 512MB are
> needed for the same operation.
>
> Considering all the above, I was wondering if it would make sense to
> backport these fixes to 1.6.x. I know that traditionally we only backport
> bug fixes, but depending on how you look at it, the limitation was also
> kind of a bug :). I was only considering 1.6.x as a candidate branch
> because the cold standby code in 1.8 and 1.6.x is 98% the same.
>
> Thanks,
>
> Andrei
>
> [0] https://issues.apache.org/jira/browse/OAK-5902
>
> [1] https://issues.apache.org/jira/browse/OAK-6565

Re: svn commit: r1805209 - in /jackrabbit/oak/trunk/oak-segment-tar: ./ src/main/java/org/apache/jackrabbit/oak/segment/file/ src/main/java/org/apache/jackrabbit/oak/segment/file/tar/ src/test/java/or

[Francesco Mari]

Instead of creating IOMonitor.NULL, you could instantiate a new
IOMonitorAdapter when needed. This has already been done many times in
TarFileTest and TarWriterTest.

On Wed, Aug 16, 2017 at 5:26 PM,   wrote:
> Author: mduerig
> Date: Wed Aug 16 15:26:29 2017
> New Revision: 1805209
>
> URL: http://svn.apache.org/viewvc?rev=1805209=rev
> Log:
> OAK-6555: Implement ITs for rolling upgrade
>
> Added:
> 
> jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/upgrade/
> 
> jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/upgrade/UpgradeIT.java
> jackrabbit/oak/trunk/oak-segment-tar/src/test/upgrade-it-scripts/
> 
> jackrabbit/oak/trunk/oak-segment-tar/src/test/upgrade-it-scripts/create16store.groovy
> Modified:
> jackrabbit/oak/trunk/oak-segment-tar/pom.xml
> 
> jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ManifestChecker.java
> 
> jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/IOMonitor.java
>
> Modified: jackrabbit/oak/trunk/oak-segment-tar/pom.xml
> URL: 
> http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/pom.xml?rev=1805209=1805208=1805209=diff
> ==
> --- jackrabbit/oak/trunk/oak-segment-tar/pom.xml (original)
> +++ jackrabbit/oak/trunk/oak-segment-tar/pom.xml Wed Aug 16 15:26:29 2017
> @@ -87,6 +87,40 @@
>  true
>  
>  
> +
> +org.apache.maven.plugins
> +maven-dependency-plugin
> +
> +
> +pre-integration-test
> +copy
> +
> +
> org.apache.jackrabbit:oak-run:1.6.1
> +
> ${project.build.directory}/upgrade-it
> +true
> +
> +
> +
> +
> +
> +org.apache.maven.plugins
> +maven-resources-plugin
> +
> +
> +pre-integration-test
> +copy-resources
> +
> +
> ${project.build.directory}/upgrade-it
> +
> +
> +
> src/test/upgrade-it-scripts
> +false
> +
> +
> +
> +
> +
> +
>  
>  
>
>
> Modified: 
> jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ManifestChecker.java
> URL: 
> http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ManifestChecker.java?rev=1805209=1805208=1805209=diff
> ==
> --- 
> jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ManifestChecker.java
>  (original)
> +++ 
> jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/ManifestChecker.java
>  Wed Aug 16 15:26:29 2017
> @@ -22,9 +22,9 @@ import static com.google.common.base.Pre
>  import java.io.File;
>  import java.io.IOException;
>
> -class ManifestChecker {
> +public class ManifestChecker {
>
> -static ManifestChecker newManifestChecker(File path, boolean 
> shouldExist, int minStoreVersion, int maxStoreVersion) {
> +public static ManifestChecker newManifestChecker(File path, boolean 
> shouldExist, int minStoreVersion, int maxStoreVersion) {
>  checkArgument(path != null, "path");
>  checkArgument(minStoreVersion > 0, "minStoreVersion");
>  checkArgument(maxStoreVersion > 0, "maxStoreVersion");
> @@ -52,7 +52,7 @@ class ManifestChecker {
>  updateManifest(manifest);
>  }
>
> -void checkManifest() throws IOException, 
> InvalidFileStoreVersionException {
> +public void checkManifest() throws IOException, 
> InvalidFileStoreVersionException {
>  checkManifest(openManifest());
>  }
>
>
> Modified: 
> jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/IOMonitor.java
> URL: 
> http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/IOMonitor.java?rev=1805209=1805208=1805209=diff
> ==
> --- 
> jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/IOMonitor.java
>  (original)
> +++ 
> 

Re: BUILD FAILURE: Jackrabbit Oak - Build # 603 - Still Failing

[Francesco Mari]

Actually the build succeeded, but the job failed due to an exception
thrown by Jenkins.

ERROR: Step ?JIRA: Create issue? aborted due to exception:
java.lang.ClassNotFoundException: com.atlassian.fugue.Effect
at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1374)
at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1327)
at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1080)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Caused: java.lang.NoClassDefFoundError: com/atlassian/fugue/Effect
at 
com.atlassian.httpclient.apache.httpcomponents.DefaultHttpClientFactory.doCreate(DefaultHttpClientFactory.java:68)
at 
com.atlassian.httpclient.apache.httpcomponents.DefaultHttpClientFactory.create(DefaultHttpClientFactory.java:35)
at 
com.atlassian.jira.rest.client.internal.async.AsynchronousHttpClientFactory.createClient(AsynchronousHttpClientFactory.java:63)
at 
com.atlassian.jira.rest.client.internal.async.AsynchronousJiraRestClientFactory.create(AsynchronousJiraRestClientFactory.java:35)
at 
com.atlassian.jira.rest.client.internal.async.AsynchronousJiraRestClientFactory.createWithBasicHttpAuthentication(AsynchronousJiraRestClientFactory.java:42)
at hudson.plugins.jira.JiraSite.createSession(JiraSite.java:280)
at hudson.plugins.jira.JiraSite.getSession(JiraSite.java:255)
at 
hudson.plugins.jira.JiraCreateIssueNotifier.getJiraSession(JiraCreateIssueNotifier.java:285)
at 
hudson.plugins.jira.JiraCreateIssueNotifier.getStatus(JiraCreateIssueNotifier.java:216)
at 
hudson.plugins.jira.JiraCreateIssueNotifier.currentBuildResultSuccess(JiraCreateIssueNotifier.java:387)
at 
hudson.plugins.jira.JiraCreateIssueNotifier.perform(JiraCreateIssueNotifier.java:159)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:676)
at 
hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.post2(MavenModuleSetBuild.java:1072)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:621)
at hudson.model.Run.execute(Run.java:1760)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:542)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:405)


On Mon, Aug 7, 2017 at 11:30 AM, Apache Jenkins Server
 wrote:
> The Apache Jenkins build system has built Jackrabbit Oak (build #603)
>
> Status: Still Failing
>
> Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/603/ 
> to view the results.
>
> Changes:
> [frm] OAK-6518 - Remove unused member variable
>
>
>
> Test results:
> All tests passed

Intent to backport OAK-4883 to 1.4

[Francesco Mari]

OAK-4883 fixes an issue introduced by a previous backport. The core of
the issue is about publishing the JMX bean that allows to run the data
store garbage collection on a standby instance.

Re: Percentile implementation

[Francesco Mari]

2017-07-04 10:52 GMT+02:00 Andrei Dulceanu :
> Now my question is this: do we have a simple percentile implementation in
> Oak (I didn't find one)?

I'm not aware of a percentile implementation in Oak.

> If not, would you recommend writing my own or adapting/extracting an
> existing one in a utility class?

In the past we copied and pasted source code from other projects in
Oak. As long as the license allows it and proper attribution is given,
it shouldn't be a problem. That said, I'm not a big fan of either
rewriting an implementation from scratch or copying and pasting source
code from other projects. Is exposing a percentile really necessary?
If yes, how big of a problem is embedding of commons-math3?

Re: Trunk doesn't compile

[Francesco Mari]

Thanks for taking care of this.

2017-07-04 10:17 GMT+02:00 Chetan Mehrotra <chetan.mehro...@gmail.com>:

> My fault. Looks like code used API from Tika 1.15 which I am yet
> testing. Would fix it now
> Chetan Mehrotra
>
>
> On Tue, Jul 4, 2017 at 1:34 PM, Francesco Mari <mari.france...@gmail.com>
> wrote:
> > When compiling trunk (r1800739) I get the following error in the
> oak-lucene
> > module.
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile
> > (default-compile) on project oak-lucene: Compilation failure
> > [ERROR]
> > /Users/mari/src/svn/oak/trunk/oak-lucene/src/main/java/org/
> apache/jackrabbit/oak/plugins/index/lucene/binary/
> TikaParserConfig.java:[94,34]
> > cannot find symbol
> > [ERROR]   symbol:   method getDocumentBuilder()
> > [ERROR]   location: class org.apache.tika.parser.ParseContext
> >
> > Can someone have a look at it?
>

Trunk doesn't compile

[Francesco Mari]

When compiling trunk (r1800739) I get the following error in the oak-lucene
module.

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile
(default-compile) on project oak-lucene: Compilation failure
[ERROR]
/Users/mari/src/svn/oak/trunk/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/binary/TikaParserConfig.java:[94,34]
cannot find symbol
[ERROR]   symbol:   method getDocumentBuilder()
[ERROR]   location: class org.apache.tika.parser.ParseContext

Can someone have a look at it?

Re: Dangling Java 8 Maven profile!?

[Francesco Mari]

+1

2017-04-25 9:08 GMT+02:00 Michael Dürig :

>
> Hi,
>
> After OAK-5664 moved us to Java 8 I believe we can remove the java8 Maven
> profile as well [1].
>
> Michael
>
> [1] https://github.com/apache/jackrabbit-oak/blob/trunk/oak-pare
> nt/pom.xml#L960
>

Re: Oak 1.0.29 vs 1.4.10 memory mapping.

[Francesco Mari]

Hi,

2017-03-23 16:13 GMT+01:00 Alex Parvulescu :
> Hi,
>
> To add what I have found so far. This seems related to OAK-4274, but I
> think there might be a twist in there somewhere.
> I'm pretty sure that this method is the one introducing the extra full
> mapping of the repository: FileStoreHelper.checkFileStoreVersionOrFail [0].
> Disabling this method takes the 2x mapping away completely.

I think your analysis matches with the 2x increase in memory mappings.
It might be that the memory mappings created while opening the
FileStore in read-only mode are never garbage collected, even if the
FileStore is correctly closed when checkFileStoreVersionOrFail()
returns. To avoid this, we might want to open the FileStore in
checkFileStoreVersionOrFail() with memory mapping disabled.

> The reason I'm saying it is relate to OAK-4274 is because I looked at a
> heap dump to verify what is keeping the references to the readonly store
> and there are no live ones, the refs should be GC'ed, but for some reason
> they are not.
>
> I'm still poking around, did not create an oak issue yet. Still pending is
> to verify if this affects other areas than oak-run.
>
> Feedback is more than welcome!
>
> best,
> alex
>
> [0]
> https://github.com/apache/jackrabbit-oak/blob/1.6/oak-run/src/main/java/org/apache/jackrabbit/oak/plugins/segment/FileStoreHelper.java#L209

Re: Oak 1.0.29 vs 1.4.10 memory mapping.

[Francesco Mari]

You might be hitting OAK-4274, which I discovered quite some time ago.
I'm not aware of a way to resolve this issue at the moment.

2017-03-22 16:47 GMT+01:00 Alex Parvulescu :
> Hi,
>
> To give more background this came about during an investigation into a slow
> offline compaction but it may affect any running FileStore as well (to be
> verified).
> I don't think it's related to oak-run itself, but more with the way we map
> files, and so far it looks like a bug (there is no reasonable explanation
> for mapping each tar file twice).
>
> Took a quick look at the TarReader but there are not many changes in this
> area 1.0 vs. 1.4 branches.
> If no one has better ideas, I'll create an oak issue and investigate this a
> bit further.
>
> thanks,
> alex
>
>
> On Wed, Mar 22, 2017 at 4:28 PM, Ian Boston  wrote:
>
>> Hi,
>> I am looking at Oak-run and I see 2x the mapped memory between 1.0.29 and
>> 1.4.10. It looks like in 1.0.29 each segment file is mapped into memory
>> once, but in 1.4.10 its mapped into memory 2x.
>>
>> Is this expected ?
>>
>> Its not great for page faults.
>> Best Regards
>> Ian
>>

Re: Merge policy for the 1.6 branch

[Francesco Mari]

2017-03-14 11:59 GMT+01:00 Michael Dürig :
>
> Hi,
>
> Following up on Davide's release plan for Oak 1.6 [1] we should define a
> merge policy for the 1.6 branch. I would suggest to be a bit more
> conservative here than we have been in the past and ask for reviews of
> backports. That is, announce candidates on @oak-dev mentioning the issue
> reference, potential risks, mitigations, etc. I don't think we need to block
> the actual backport being performed on the outcome of the review as in the
> worst case changes can always be reverted. The main aim of the announcement
> should be to increase visibility of the backports and ensure they are
> eventually reviewed.
>
> In short, announce your backport on @oak-dev and ask for review. If
> confident enough that the review will pass anyway, go ahead but be prepared
> to revert.
>
> I think this is what we informally did so far already but wanted to state
> this a bit more explicitly.
>
> WDYT?

I'm in favour. I also like the idea of permanently applying this
policy to every branch.

> Michael
>
>
>
> [1]
> https://lists.apache.org/thread.html/e5e71b61de9612d7cac195cbe948e8bdca58ee38ee16e7f124ea742c@%3Coak-dev.jackrabbit.apache.org%3E

Re: svn commit: r1784874 - in /jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document: BlobReferenceIteratorTest.java OrphanedBranchTest.java RevisionGCTest.java

[Francesco Mari]

The test will fail, but the cleanup after the failed test might run
only partially. I had many cleanup issues myself for the segment store
(e.g. files kept open, unclosed connections), and I found that code
similar to this was the main offender. It was just a gentle warning,
it might be that this problem is irrelevant to the document store!

2017-03-01 13:26 GMT+01:00 Marcel Reutegger <mreut...@adobe.com>:
> On 01/03/17 09:55, Francesco Mari wrote:
>>
>> I don't know the implementation of store.dispose() but if that call
>> throws an exception fixture.dispose() will not be invoked. If there is
>> no ordering dependency between store.dispose(), fixture.dispose(), and
>> Revision.resetClockToDefault(), it's more robust to put them in
>> different @After methods.
>
>
> The ordering is important because there are dependencies between the calls.
> store.dispose() may indeed fail with an exception, but that would fail the
> test anyway, right? So, not sure if it is worth fixing all those tests. It's
> not just the three tests that I updated. There are many more that would be
> affected.
>
> Regards
>  Marcel

Re: svn commit: r1784874 - in /jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document: BlobReferenceIteratorTest.java OrphanedBranchTest.java RevisionGCTest.java

[Francesco Mari]

I don't know the implementation of store.dispose() but if that call
throws an exception fixture.dispose() will not be invoked. If there is
no ordering dependency between store.dispose(), fixture.dispose(), and
Revision.resetClockToDefault(), it's more robust to put them in
different @After methods.

2017-03-01 9:38 GMT+01:00  :
> Author: mreutegg
> Date: Wed Mar  1 08:38:32 2017
> New Revision: 1784874
>
> URL: http://svn.apache.org/viewvc?rev=1784874=rev
> Log:
> OAK-5866: Some tests do not dispose fixture
>
> Modified:
> 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/BlobReferenceIteratorTest.java
> 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/OrphanedBranchTest.java
> 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/RevisionGCTest.java
>
> Modified: 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/BlobReferenceIteratorTest.java
> URL: 
> http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/BlobReferenceIteratorTest.java?rev=1784874=1784873=1784874=diff
> ==
> --- 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/BlobReferenceIteratorTest.java
>  (original)
> +++ 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/BlobReferenceIteratorTest.java
>  Wed Mar  1 08:38:32 2017
> @@ -73,6 +73,7 @@ public class BlobReferenceIteratorTest {
>  @After
>  public void tearDown() throws Exception {
>  store.dispose();
> +fixture.dispose();
>  }
>
>  @Test
>
> Modified: 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/OrphanedBranchTest.java
> URL: 
> http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/OrphanedBranchTest.java?rev=1784874=1784873=1784874=diff
> ==
> --- 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/OrphanedBranchTest.java
>  (original)
> +++ 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/OrphanedBranchTest.java
>  Wed Mar  1 08:38:32 2017
> @@ -82,6 +82,7 @@ public class OrphanedBranchTest {
>  @After
>  public void tearDown() throws Exception {
>  store.dispose();
> +fixture.dispose();
>  }
>
>  @Test
>
> Modified: 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/RevisionGCTest.java
> URL: 
> http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/RevisionGCTest.java?rev=1784874=1784873=1784874=diff
> ==
> --- 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/RevisionGCTest.java
>  (original)
> +++ 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/plugins/document/RevisionGCTest.java
>  Wed Mar  1 08:38:32 2017
> @@ -72,6 +72,7 @@ public class RevisionGCTest {
>  if (store != null) {
>  store.dispose();
>  }
> +fixture.dispose();
>  Revision.resetClockToDefault();
>  }
>
>
>

Re: JIRA commit plugin

[Francesco Mari]

2017-02-14 13:49 GMT+01:00 Alex Parvulescu :
> Am I the only one relying on this feature to figure out backports? Of
> course you have the comments on Jira, but almost always there's that
> forgotten commit that fixes a tiny thing which isn't mentioned any more.

A temporary solution would be to use "svn log --search ..." to locate
the commits whose message matches a specific pattern.

[DISCUSS] Which I/O statistics should the FileStore expose?

[Francesco Mari]

Hi all,

The recently introduced IOMonitor allows the FileStore to trigger I/O
events. Callback methods from IOMonitor can be implemented to receive
information about segment reads and writes.

A trivial implementation of IOMonitor is able to track the following raw data.

- The number of segments read and write operations.
- The duration in nanoseconds of every read and write.
- The number of bytes read or written by each operation.

We are about to expose this kind of information from an MBean - for
the sake of discussion, let's call it IOMonitorMBean. I'm currently in
favour of starting small and exposing the following statistics:

- The duration of the latest write (long).
- The duration of the latest read (long).
- The number of write operations (long).
- The number of read operations (long).

I would like your opinion about what's the most useful way to present
this data through an MBean. Should just raw data be exposed? Is it
appropriate for IOMonitorMBean to perform some kind of aggregation,
like sum and average? Should richer data be returned from the MBean,
like tabular data?

Please keep in mind that this data is supposed to be consumed by a
monitoring solution, and not a by human reader.

Re: oak-run, diet and new module

[Francesco Mari]

As much as I like the proposal of slimming down oak-run, I think that
dividing oak-run in oak-operations and oak-development is the wrong
way to go. This kind of division is horizontal, since commands
pertaining to different persistence backends are grouped together
according to their roles. This division will not solve the problem of
feature bloat. These two modules will grow over time in the same way
that oak-run did.

I'm more in favour of a vertical separation of oak-run. I explained
part of this idea in OAK-5437. I think it's more effective to split
oak-run in vertical slices, where each slice pertains to a persistence
layer (segment, mongo, etc.) or a well defined functional area
(indexing, security, etc.). This kind of separation would bring the
CLI code close to the main code they are working with. Changes in the
main code are more easily reflected in the CLI code, and the other way
around. It would also be easier to figure out which individual or
group of individuals is actively maintaining a certain piece of code.

2017-02-10 9:44 GMT+01:00 Angela Schreiber :
> hi davide
>
> could you elaborate a bit on your proposal? from the names (oak-operations
> and oak-development) it's not clear to me what code would go into which
> module... also i am not sure about deleting oak-run. for the sake of
> limiting impact (also when it comes to the backport you mention later on)
> i would rather suggest to move out code that doesn't belong there and keep
> stuff that more naturally fits into 'run': so, only one additional module
> and no deletion.
>
> as far as backporting to all branches is concerned: that's for sure not
> feasible for the benchmarks i have been putting into oak-run when
> introducing new features and improvements.
>
> kind regards
> angela
>
> On 09/02/17 20:28, "Davide Giannella"  wrote:
>
>>hello team,
>>
>>while having a bit of time I resumed the topic grouped in the epic
>>https://issues.apache.org/jira/browse/OAK-5599.
>>
>>Part of the discussion we already had in the past 1 or two years is that
>>oak-run is big and begin to be a challenge during releases and the fact
>>that we could split development functionalities from production tooling
>>would allow us to remove quite a bunch of libraries from the jar
>>deployed on mvn for production tooling and will leave the development
>>one not deployed.
>>
>>Main scratching I have now is: assuming we proceed what about backports?
>>So i thought the following:
>>
>>- main goal: create oak-operations and oak-development modules.
>>Eventaully delete oak-run.
>>- backport these on all the branches. Up to what version? Can we blindly
>>backport all of the stuff?
>>- what are the differences nowadays in oak-run between branches?
>>Repository construction? others?
>>
>>Thoughts?
>>
>>Cheers
>>Davide
>

Re: svn commit: r1779324 - in /jackrabbit/oak/trunk/oak-segment-tar: ./ src/test/java/org/apache/jackrabbit/oak/segment/standby/ src/test/java/org/apache/jackrabbit/oak/segment/test/

[Francesco Mari]

Good suggestion. I opened OAK-5483 to track this.

2017-01-19 5:26 GMT+01:00 Chetan Mehrotra :
> Hi Francesco,
>
> On Wed, Jan 18, 2017 at 7:01 PM,   wrote:
>> +package org.apache.jackrabbit.oak.segment.test;
>> +
>> +import java.net.ServerSocket;
>> +
>> +import org.junit.rules.ExternalResource;
>> +
>> +public class TemporaryPort extends ExternalResource {
>> +
>> +private int port;
>> +
>> +@Override
>> +protected void before() throws Throwable {
>> +try (ServerSocket socket = new ServerSocket(0)) {
>> +port = socket.getLocalPort();
>> +}
>> +}
>> +
>> +public int getPort() {
>> +return port;
>> +}
>> +
>> +}
>
> This looks useful and can be used in other places also like in [1].
> Would be good if we can move it to oak-commons in
> org.apache.jackrabbit.oak.commons.junit package.
>
> Chetan Mehrotra
> [1] 
> https://issues.apache.org/jira/browse/OAK-5441?focusedCommentId=15823491=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15823491

Re: Restrict matrix tasks to specific nodes

[Francesco Mari]

Thanks for your help. I will monitor the build during the following
days and verify that everything is alright.

2016-12-08 20:01 GMT+01:00 Gavin McDonald <ga...@16degrees.com.au>:
>
>> On 8 Dec. 2016, at 9:34 pm, Francesco Mari <mari.france...@gmail.com> wrote:
>>
>> Hi,
>>
>> I recently discovered that some of the tests run by the Apache
>> Jackrabbit Oak Matrix job [1] consistently fail when matrix tasks are
>> scheduled on one of the nodes labelled "beam". I documented the
>> failures in this issue [2].
>>
>> I would like to schedule the matrix tasks to run on specific nodes or,
>> at least, not to run on the faulty ones. What is the best way to
>> achieve this?
>
> I added a ‘label’ expression to your build so that they only build on nodes 
> labelled ‘ubuntu’
>
> Let me know if that helps.
>
> (Beam and others don’t use the ‘ubuntu’ label
>
> See: https://cwiki.apache.org/confluence/display/INFRA/Jenkins+node+labels 
> <https://cwiki.apache.org/confluence/display/INFRA/Jenkins+node+labels> )
>
>
> Gav…
>
>>
>> Thanks,
>>
>> Francesco
>>
>> [1]: 
>> https://builds.apache.org/view/All/job/Apache%20Jackrabbit%20Oak%20matrix/
>> [2]: https://issues.apache.org/jira/browse/OAK-5017
>

Restrict matrix tasks to specific nodes

[Francesco Mari]

Hi,

I recently discovered that some of the tests run by the Apache
Jackrabbit Oak Matrix job [1] consistently fail when matrix tasks are
scheduled on one of the nodes labelled "beam". I documented the
failures in this issue [2].

I would like to schedule the matrix tasks to run on specific nodes or,
at least, not to run on the faulty ones. What is the best way to
achieve this?

Thanks,

Francesco

[1]: https://builds.apache.org/view/All/job/Apache%20Jackrabbit%20Oak%20matrix/
[2]: https://issues.apache.org/jira/browse/OAK-5017

Re: Frequent failures in standby test

[Francesco Mari]

In the meantime, I configured those tests not to run when they are
scheduled under a "beam" node. I will send a separate mail to builds@
and put oak-dev@ in CC.

2016-12-08 11:23 GMT+01:00 Robert Munteanu <romb...@apache.org>:
> Hi,
>
> On Thu, 2016-12-08 at 10:01 +0100, Francesco Mari wrote:
>> I analysed the latest builds to figure out what's going on with some
>> tests related to Cold Standby. It seems that the problem is
>> restricted
>> to some Jenkins nodes - see [1] for more details. Can we configure
>> the
>> Jenkins job not to run on nodes labelled "beam"?
>>
>> [1]: https://issues.apache.org/jira/browse/OAK-
>> 5017?focusedCommentId=15731590=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-15731590
>
> The job is configured to only run on 'ubuntu' nodes, but it seems that
> this restriction is not taken into account. The following Jenkins issue
> might be related
>
>   https://issues.jenkins-ci.org/browse/JENKINS-23459
>
> Perhaps a question better suited to builds@ ? They would be interested
> in proper resource utilisation.
>
> Robert
>
>>
>> 2016-11-25 13:25 GMT+01:00 Francesco Mari <mari.france...@gmail.com>:
>> > Every failure in build 1298 is due to "java.net.BindException:
>> > Address
>> > already in use". According to the build log, the port wasn't bound
>> > by
>> > a server started during a test. The first failing test
>> > (ExternalSharedStoreIT) was also the first test in the build to
>> > start
>> > a server. The issue persisted for some tests (the whole range of
>> > tests
>> > in ExternalSharedStoreIT and the first test in FailoverSslTestIT),
>> > so
>> > I can't deduce any random behaviour. Once the port (presumingly
>> > bound
>> > to an external process) was released, every other integration test
>> > worked without any problem.
>> >
>> > 2016-11-25 5:38 GMT+01:00 Chetan Mehrotra <chetan.mehro...@gmail.co
>> > m>:
>> > > Per https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20mat
>> > > rix/1298/
>> > > the test again failed but mostly on Jdk 1.7. The test on Jdk 1.8
>> > > looks
>> > > like passed.
>> > > Chetan Mehrotra
>> > >
>> > >
>> > > On Tue, Nov 22, 2016 at 12:48 PM, Chetan Mehrotra
>> > > <chetan.mehro...@gmail.com> wrote:
>> > > > They are from oak-segment-tar. See
>> > > > https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matri
>> > > > x/1295/#showFailuresLink
>> > > > Chetan Mehrotra
>> > > >
>> > > >
>> > > > On Tue, Nov 22, 2016 at 12:42 PM, Francesco Mari
>> > > > <mari.france...@gmail.com> wrote:
>> > > > > Are those from oak-tarmk-standby or oak-segment-tar?
>> > > > >
>> > > > > 2016-11-22 6:11 GMT+01:00 Chetan Mehrotra <chetan.mehrotra@gm
>> > > > > ail.com>:
>> > > > > > Hi Team,
>> > > > > >
>> > > > > > Since last 4-6 builds I am seeing a recurring failure of
>> > > > > > few test in
>> > > > > > standby module
>> > > > > >
>> > > > > > * FailoverIPRangeIT
>> > > > > > * ExternalPrivateStoreIT
>> > > > > > * StandbyTestIT
>> > > > > >
>> > > > > > Probably something to be looked into
>> > > > > >
>> > > > > > Chetan Mehrotra
>

Re: Frequent failures in standby test

[Francesco Mari]

I analysed the latest builds to figure out what's going on with some
tests related to Cold Standby. It seems that the problem is restricted
to some Jenkins nodes - see [1] for more details. Can we configure the
Jenkins job not to run on nodes labelled "beam"?

[1]: 
https://issues.apache.org/jira/browse/OAK-5017?focusedCommentId=15731590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15731590

2016-11-25 13:25 GMT+01:00 Francesco Mari <mari.france...@gmail.com>:
> Every failure in build 1298 is due to "java.net.BindException: Address
> already in use". According to the build log, the port wasn't bound by
> a server started during a test. The first failing test
> (ExternalSharedStoreIT) was also the first test in the build to start
> a server. The issue persisted for some tests (the whole range of tests
> in ExternalSharedStoreIT and the first test in FailoverSslTestIT), so
> I can't deduce any random behaviour. Once the port (presumingly bound
> to an external process) was released, every other integration test
> worked without any problem.
>
> 2016-11-25 5:38 GMT+01:00 Chetan Mehrotra <chetan.mehro...@gmail.com>:
>> Per https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1298/
>> the test again failed but mostly on Jdk 1.7. The test on Jdk 1.8 looks
>> like passed.
>> Chetan Mehrotra
>>
>>
>> On Tue, Nov 22, 2016 at 12:48 PM, Chetan Mehrotra
>> <chetan.mehro...@gmail.com> wrote:
>>> They are from oak-segment-tar. See
>>> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1295/#showFailuresLink
>>> Chetan Mehrotra
>>>
>>>
>>> On Tue, Nov 22, 2016 at 12:42 PM, Francesco Mari
>>> <mari.france...@gmail.com> wrote:
>>>> Are those from oak-tarmk-standby or oak-segment-tar?
>>>>
>>>> 2016-11-22 6:11 GMT+01:00 Chetan Mehrotra <chetan.mehro...@gmail.com>:
>>>>> Hi Team,
>>>>>
>>>>> Since last 4-6 builds I am seeing a recurring failure of few test in
>>>>> standby module
>>>>>
>>>>> * FailoverIPRangeIT
>>>>> * ExternalPrivateStoreIT
>>>>> * StandbyTestIT
>>>>>
>>>>> Probably something to be looked into
>>>>>
>>>>> Chetan Mehrotra

Re: svn commit: r1772167 - /jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/query/QueryShapeTool.java

[Francesco Mari]

Would it make sense to make this part of oak-run instead?

2016-12-01 11:59 GMT+01:00  :
> Author: thomasm
> Date: Thu Dec  1 10:59:55 2016
> New Revision: 1772167
>
> URL: http://svn.apache.org/viewvc?rev=1772167=rev
> Log:
> OAK-1884 Query: log file analyzer tool
>
> Added:
> 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/query/QueryShapeTool.java
>
> Added: 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/query/QueryShapeTool.java
> URL: 
> http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/query/QueryShapeTool.java?rev=1772167=auto
> ==
> --- 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/query/QueryShapeTool.java
>  (added)
> +++ 
> jackrabbit/oak/trunk/oak-core/src/test/java/org/apache/jackrabbit/oak/query/QueryShapeTool.java
>  Thu Dec  1 10:59:55 2016
> @@ -0,0 +1,92 @@
> +/*
> + * Licensed to the Apache Software Foundation (ASF) under one or more
> + * contributor license agreements.  See the NOTICE file distributed with
> + * this work for additional information regarding copyright ownership.
> + * The ASF licenses this file to You under the Apache License, Version 2.0
> + * (the "License"); you may not use this file except in compliance with
> + * the License.  You may obtain a copy of the License at
> + *
> + *  http://www.apache.org/licenses/LICENSE-2.0
> + *
> + * Unless required by applicable law or agreed to in writing, software
> + * distributed under the License is distributed on an "AS IS" BASIS,
> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> + * See the License for the specific language governing permissions and
> + * limitations under the License.
> + */
> +package org.apache.jackrabbit.oak.query;
> +
> +import java.io.BufferedReader;
> +import java.io.File;
> +import java.io.FileReader;
> +import java.io.IOException;
> +import java.io.LineNumberReader;
> +import java.util.TreeSet;
> +
> +/**
> + * A tool that converts a list of queries to parameterized queries. This for
> + * example allows to extract unique queries.
> + */
> +public class QueryShapeTool {
> +
> +public static void main(String... args) throws IOException {
> +process(new File(args[0]));
> +}
> +
> +public static void process(File file) throws IOException {
> +processFile(file);
> +}
> +
> +private static void processFile(File file) throws IOException {
> +if (file.isDirectory()) {
> +for (File f : file.listFiles()) {
> +processFile(f);
> +}
> +return;
> +}
> +System.out.println("File " + file);
> +LineNumberReader r = new LineNumberReader(new BufferedReader(
> +new FileReader(file)));
> +try {
> +process(r);
> +} finally {
> +r.close();
> +}
> +System.out.println();
> +}
> +
> +private static void process(LineNumberReader reader) throws IOException {
> +TreeSet sortedUnique = new  TreeSet();
> +while (true) {
> +String line = reader.readLine();
> +if (line == null) {
> +break;
> +}
> +sortedUnique.add(shape(line));
> +}
> +for (String s : sortedUnique) {
> +System.out.println(s);
> +}
> +}
> +
> +private static String shape(String query) {
> +String result = query;
> +// replace double quoted string literals with "$s"
> +result = result.replaceAll("\"[^\"]*\"", "\\$s");
> +// replace single quoted string literals with "$s"
> +result = result.replaceAll("'[^\']*\'", "\\$s");
> +// replace repeated "$s" with a single one (due to escape characters 
> in string literals)
> +result = result.replaceAll("(\\$s)+", "\\$s");
> +
> +// xpath: replace "//" with "/ /" so we can more easily stop there
> +result = result.replaceAll("//", "/ /");
> +// xpath: replace "/element(" with "/ element" for the same reason
> +result = result.replaceAll("/element\\(", "/ element\\(");
> +// xpath: replace "/text(" with "/ text" for the same reason
> +result = result.replaceAll("/text\\(", "/ text\\(");
> +// xpath: replace a path at the beginning of the query with $path
> +result = result.replaceAll("/jcr:root(/([^ /]*))*[ /]", 
> "/jcr:root/\\$path/");
> +return result;
> +}
> +
> +}
> \ No newline at end of file
>
>

Re: Remove oak-remote from reactor to get rid of JSON dependency

[Francesco Mari]

I will revisit RemoteServletIT to isolate and remove every usage of the
JSON library. The test will have to be @Ignore'd, but oak-remote can be
kept in the build without further modifications.

I will rework the test in a second iteration to remove Unirest once and for
all and restore RemoteServletIT.

On Nov 28, 2016 3:53 PM, "Julian Reschke"  wrote:

> On 2016-11-28 15:33, Michael Dürig wrote:
>
>>
>> Hi,
>>
>> As announced earlier today we need to get rid of all dependencies to
>> org.json:json. In Oak only oak-remote in trunk and 1.4 is affected via a
>> test dependency [1]. As I currently don't see any activity on oak-remote
>> I would suggest to remove that module from the reactor, open an issue to
>> add it back and block the latter on an issue to remove the JSON
>> dependency. This would allow us to go forward with our release plan for
>> the rest of Oak and pick up oak-remote at any time without forgetting
>> the JSON dependency issue.
>>
>> Please raise your concerns here otherwise I will go forward as proposed
>> later this week.
>>
>> Michael
>>
>> [1] https://issues.apache.org/jira/browse/OAK-5171
>>
>
> Unless it's trivial to fix (which is unlikely given the fact it's a
> transitive dependency), I agree that removing it from the build seems to be
> the best approach for now.
>
> Best regards, Julian
>
>

Re: Frequent failures in standby test

[Francesco Mari]

Every failure in build 1298 is due to "java.net.BindException: Address
already in use". According to the build log, the port wasn't bound by
a server started during a test. The first failing test
(ExternalSharedStoreIT) was also the first test in the build to start
a server. The issue persisted for some tests (the whole range of tests
in ExternalSharedStoreIT and the first test in FailoverSslTestIT), so
I can't deduce any random behaviour. Once the port (presumingly bound
to an external process) was released, every other integration test
worked without any problem.

2016-11-25 5:38 GMT+01:00 Chetan Mehrotra <chetan.mehro...@gmail.com>:
> Per https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1298/
> the test again failed but mostly on Jdk 1.7. The test on Jdk 1.8 looks
> like passed.
> Chetan Mehrotra
>
>
> On Tue, Nov 22, 2016 at 12:48 PM, Chetan Mehrotra
> <chetan.mehro...@gmail.com> wrote:
>> They are from oak-segment-tar. See
>> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1295/#showFailuresLink
>> Chetan Mehrotra
>>
>>
>> On Tue, Nov 22, 2016 at 12:42 PM, Francesco Mari
>> <mari.france...@gmail.com> wrote:
>>> Are those from oak-tarmk-standby or oak-segment-tar?
>>>
>>> 2016-11-22 6:11 GMT+01:00 Chetan Mehrotra <chetan.mehro...@gmail.com>:
>>>> Hi Team,
>>>>
>>>> Since last 4-6 builds I am seeing a recurring failure of few test in
>>>> standby module
>>>>
>>>> * FailoverIPRangeIT
>>>> * ExternalPrivateStoreIT
>>>> * StandbyTestIT
>>>>
>>>> Probably something to be looked into
>>>>
>>>> Chetan Mehrotra

Re: Build failing due to compilation errors in oak-segment-tar

[Francesco Mari]

Sorry, my bad. I messed up when I applied the patch. Thanks Chetan for
fixing this.

2016-11-23 7:22 GMT+01:00 Andrei Dulceanu :
> @Chetan Mehrotra, yes, this is the file. Thanks for adding it!
>
> On 23/11/2016, 07:18, "Chetan Mehrotra"  wrote:
>
> Added missed file in r1770910
>
> @Francesco/Andrei Can you check if its the intended file. With this
> compilation passes on my setup
> Chetan Mehrotra
>
>
> On Wed, Nov 23, 2016 at 10:42 AM, Chetan Mehrotra
>  wrote:
>> Build is failing locally and in CI [1] due to compilation error in
>> oak-segment-tar. Looks like SegmentGCStatus class is not checked in
>>
>> [ERROR] 
>> /home/chetanm/git/apache/jackrabbit-oak/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStoreGCMonitor.java:[32,51]
>> error: cannot find symbol
>> [ERROR] symbol:   class SegmentGCStatus
>> [ERROR] location: package org.apache.jackrabbit.oak.segment.compaction
>>
>>
>> Chetan Mehrotra
>> [1] 
>> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1296/jdk=JDK%201.8%20(latest),nsfixtures=SEGMENT_MK,profile=unittesting/console
>
>

Re: Frequent failures in standby test

[Francesco Mari]

Are those from oak-tarmk-standby or oak-segment-tar?

2016-11-22 6:11 GMT+01:00 Chetan Mehrotra :
> Hi Team,
>
> Since last 4-6 builds I am seeing a recurring failure of few test in
> standby module
>
> * FailoverIPRangeIT
> * ExternalPrivateStoreIT
> * StandbyTestIT
>
> Probably something to be looked into
>
> Chetan Mehrotra

Re: Deprecate oak-segment and oak-tarmk-standby

[Francesco Mari]

The other solution will be to keep oak-segment and oak-tarmk-standby
as deprecated modules in 1.6.x and remove them in 1.8.

2016-11-01 14:00 GMT+01:00 Francesco Mari <mari.france...@gmail.com>:
> 2016-11-01 12:56 GMT+01:00 Julian Reschke <julian.resc...@gmx.de>:
>> On 2016-11-01 11:44, Francesco Mari wrote:
>>>
>>> Hi,
>>>
>>> 2016-11-01 11:38 GMT+01:00 Robert Munteanu <romb...@apache.org>:
>>>>
>>>> Hi Francesco,
>>>>
>>>> On Tue, 2016-11-01 at 10:16 +0100, Francesco Mari wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> This is a heads up for the deprecation oak-segment and
>>>>> oak-tarmk-standby. The new oak-segment-tar module is a replacement
>>>>> for
>>>>> both of the aforementioned modules. If somebody is concerned about
>>>>> the
>>>>> deprecation, please let me know.
>>>>
>>>>
>>>> A couple of 'outsider' questions:
>>>>
>>>> 1. What does the deprecation mean exactly?
>>>
>>>
>>> It means that we are marking every class in those modules as
>>> @Deprecated. These modules will have a 1.6 release, but will probably
>>> be removed from the reactor after that to exclude them from further
>>> releases.
>>> ...
>>
>>
>> That doesn't sound right to me.
>>
>> Either remove them *before* 1.6 is cut, or keep them in the 1.6 branch later
>> on.
>
> I think that we will need at least a 1.6 release for oak-run and
> oak-upgrade to correctly use the tooling for the migration.
>
>> Best regards, Julian

Re: Deprecate oak-segment and oak-tarmk-standby

[Francesco Mari]

2016-11-01 12:56 GMT+01:00 Julian Reschke <julian.resc...@gmx.de>:
> On 2016-11-01 11:44, Francesco Mari wrote:
>>
>> Hi,
>>
>> 2016-11-01 11:38 GMT+01:00 Robert Munteanu <romb...@apache.org>:
>>>
>>> Hi Francesco,
>>>
>>> On Tue, 2016-11-01 at 10:16 +0100, Francesco Mari wrote:
>>>>
>>>> Hi,
>>>>
>>>> This is a heads up for the deprecation oak-segment and
>>>> oak-tarmk-standby. The new oak-segment-tar module is a replacement
>>>> for
>>>> both of the aforementioned modules. If somebody is concerned about
>>>> the
>>>> deprecation, please let me know.
>>>
>>>
>>> A couple of 'outsider' questions:
>>>
>>> 1. What does the deprecation mean exactly?
>>
>>
>> It means that we are marking every class in those modules as
>> @Deprecated. These modules will have a 1.6 release, but will probably
>> be removed from the reactor after that to exclude them from further
>> releases.
>> ...
>
>
> That doesn't sound right to me.
>
> Either remove them *before* 1.6 is cut, or keep them in the 1.6 branch later
> on.

I think that we will need at least a 1.6 release for oak-run and
oak-upgrade to correctly use the tooling for the migration.

> Best regards, Julian

Re: Deprecate oak-segment and oak-tarmk-standby

[Francesco Mari]

Hi,

2016-11-01 11:38 GMT+01:00 Robert Munteanu <romb...@apache.org>:
> Hi Francesco,
>
> On Tue, 2016-11-01 at 10:16 +0100, Francesco Mari wrote:
>> Hi,
>>
>> This is a heads up for the deprecation oak-segment and
>> oak-tarmk-standby. The new oak-segment-tar module is a replacement
>> for
>> both of the aforementioned modules. If somebody is concerned about
>> the
>> deprecation, please let me know.
>
> A couple of 'outsider' questions:
>
> 1. What does the deprecation mean exactly?

It means that we are marking every class in those modules as
@Deprecated. These modules will have a 1.6 release, but will probably
be removed from the reactor after that to exclude them from further
releases.

> 2. What is the migration strategy for deployments using oak-segment?

Just use oak-segment-tar instead of oak-segment and oak-tarmk-standby.
It is almost a trivial replacement, except for some minor details that
will be covered in the documentation.

> THanks,
>
> Robert

Deprecate oak-segment and oak-tarmk-standby

[Francesco Mari]

Hi,

This is a heads up for the deprecation oak-segment and
oak-tarmk-standby. The new oak-segment-tar module is a replacement for
both of the aforementioned modules. If somebody is concerned about the
deprecation, please let me know.

Re: segment-tar depending on oak-core

[Francesco Mari]

Hi

2016-10-27 19:08 GMT+02:00 Alexander Klimetschek <aklim...@adobe.com>:
> Maybe looking at this step by step would help.

The oak-segment-tar bundle was supposed to be the first step.

>For example, start with the nodestore implementations and extract everything 
>into separate modules that is necessary for this - i.e. an oak-store-api along 
>with the impls. But keep other apis in oak-core in that first step, to limit 
>the effort. (And try not renaming the API packages, as well as keeping them 
>backwards compatible, i.e. no major version bump, if possible).

This didn't happen because of lack of consensus. See my previous
answer to Michael Marth.

>See how that works out and if positive, continue with more.

The reaction to the modularization effort was not positive, so
oak-segment-tar backed up.

>
> Cheers,
> Alex
>
> Am 27. Okt. 2016, 03:48 -0700 schrieb Francesco Mari 
> <mari.france...@gmail.com>:
> Something did happen: the first NodeStore implementation living in its
> own module was oak-segment-tar. We just decided to go back to the old
> model exactly because we didn't reach consensus about modularizing its
> upstream and downstream dependencies.
>
> 2016-10-27 12:22 GMT+02:00 Michael Marth <mma...@adobe.com>:
> fwiw: last year a concrete proposal was made that seemed to have consensus
>
> “Move NodeStore implementations into their own modules"
> http://markmail.org/message/6ylxk4twdi2lzfdz
>
> Agree that nothing happened - but I believe that this move might again find 
> consenus
>
>
>
> On 27/10/16 10:49, "Francesco Mari" <mari.france...@gmail.com> wrote:
>
> We keep having this conversation regularly but nothing ever changes.
> As much as I would like to push the modularization effort forward, I
> recognize that the majority of the team is either not in favour or
> openly against it. I don't want to disrupt the way most of us are used
> to work. Michael Dürig already provided an extensive list of what we
> will be missing if we keep writing software the way we do, so I'm not
> going to repeat it. The most sensible thing to do is, in my humble
> opinion, accept the decision of the majority.
>
> 2016-10-27 11:05 GMT+02:00 Davide Giannella <dav...@apache.org>:
> On 27/10/2016 08:53, Michael Dürig wrote:
>
> +1.
>
> It would also help re. backporting, continuous integration, releasing,
> testing, longevity, code reuse, maintainability, reducing technical
> debt, deploying, stability, etc, etc...
>
> While I can agree on the above, and the fact that now we have
> https://issues.apache.org/jira/browse/OAK-5007 in place, just for the
> sake or argument I would say that if we want to have any part of Oak
> with an independent release cycle we need to
>
> Have proper API packages that abstract things. Specially from oak-core
>
> As soon as we introduce a separate release cycle for a single module we
> have to look at a wider picture. What other modules are affected?
>
> Taking the example of segment-tar we saw that we need
>
> - oak-core-api (name can be changed)
> - independent releases of the oak tools: oak-run, oak-upgrade, ...
> - independent release cycle for parent/pom.xml
> - anything I'm missing?
>
> So if we want to go down that route than we have to do it properly and
> for good. Not half-way.
>
> Davide
>
>

Re: segment-tar depending on oak-core

[Francesco Mari]

Something did happen: the first NodeStore implementation living in its
own module was oak-segment-tar. We just decided to go back to the old
model exactly because we didn't reach consensus about modularizing its
upstream and downstream dependencies.

2016-10-27 12:22 GMT+02:00 Michael Marth <mma...@adobe.com>:
> fwiw: last year a concrete proposal was made that seemed to have consensus
>
> “Move NodeStore implementations into their own modules"
> http://markmail.org/message/6ylxk4twdi2lzfdz
>
> Agree that nothing happened - but I believe that this move might again find 
> consenus
>
>
>
> On 27/10/16 10:49, "Francesco Mari" <mari.france...@gmail.com> wrote:
>
>>We keep having this conversation regularly but nothing ever changes.
>>As much as I would like to push the modularization effort forward, I
>>recognize that the majority of the team is either not in favour or
>>openly against it. I don't want to disrupt the way most of us are used
>>to work. Michael Dürig already provided an extensive list of what we
>>will be missing if we keep writing software the way we do, so I'm not
>>going to repeat it. The most sensible thing to do is, in my humble
>>opinion, accept the decision of the majority.
>>
>>2016-10-27 11:05 GMT+02:00 Davide Giannella <dav...@apache.org>:
>>> On 27/10/2016 08:53, Michael Dürig wrote:
>>>>
>>>> +1.
>>>>
>>>> It would also help re. backporting, continuous integration, releasing,
>>>> testing, longevity, code reuse, maintainability, reducing technical
>>>> debt, deploying, stability, etc, etc...
>>>
>>> While I can agree on the above, and the fact that now we have
>>> https://issues.apache.org/jira/browse/OAK-5007 in place, just for the
>>> sake or argument I would say that if we want to have any part of Oak
>>> with an independent release cycle we need to
>>>
>>> Have proper API packages that abstract things. Specially from oak-core
>>>
>>> As soon as we introduce a separate release cycle for a single module we
>>> have to look at a wider picture. What other modules are affected?
>>>
>>> Taking the example of segment-tar we saw that we need
>>>
>>> - oak-core-api (name can be changed)
>>> - independent releases of the oak tools: oak-run, oak-upgrade, ...
>>> - independent release cycle for parent/pom.xml
>>> - anything I'm missing?
>>>
>>> So if we want to go down that route than we have to do it properly and
>>> for good. Not half-way.
>>>
>>> Davide
>>>
>>>

Re: segment-tar depending on oak-core

[Francesco Mari]

We keep having this conversation regularly but nothing ever changes.
As much as I would like to push the modularization effort forward, I
recognize that the majority of the team is either not in favour or
openly against it. I don't want to disrupt the way most of us are used
to work. Michael Dürig already provided an extensive list of what we
will be missing if we keep writing software the way we do, so I'm not
going to repeat it. The most sensible thing to do is, in my humble
opinion, accept the decision of the majority.

2016-10-27 11:05 GMT+02:00 Davide Giannella :
> On 27/10/2016 08:53, Michael Dürig wrote:
>>
>> +1.
>>
>> It would also help re. backporting, continuous integration, releasing,
>> testing, longevity, code reuse, maintainability, reducing technical
>> debt, deploying, stability, etc, etc...
>
> While I can agree on the above, and the fact that now we have
> https://issues.apache.org/jira/browse/OAK-5007 in place, just for the
> sake or argument I would say that if we want to have any part of Oak
> with an independent release cycle we need to
>
> Have proper API packages that abstract things. Specially from oak-core
>
> As soon as we introduce a separate release cycle for a single module we
> have to look at a wider picture. What other modules are affected?
>
> Taking the example of segment-tar we saw that we need
>
> - oak-core-api (name can be changed)
> - independent releases of the oak tools: oak-run, oak-upgrade, ...
> - independent release cycle for parent/pom.xml
> - anything I'm missing?
>
> So if we want to go down that route than we have to do it properly and
> for good. Not half-way.
>
> Davide
>
>

[RESULT][VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.16

[Francesco Mari]

Hi,

The vote passes as follows:

+1 Michael Dürig
+1 Francesco Mari
+1 Davide Giannella
+1 Alex Parvulescu

Thanks for voting. I'll push the release out.

Re: segment-tar depending on oak-core

[Francesco Mari]

Luckily for us this is not a computer science problem but an easier
software engineering concern.

The release process in Oak is a joke. Releasing every two weeks by
using version numbers as counters just for the sake of it is
embarrassing. I don't even know how many releases of our parent POM we
have, every one of them equal to the other, and this is nonsense.

We shouldn't go backward, but forward. We need to extract APIs into
their own independently released bundles. We should split oak-run in
different CLI utility modules, so that every implementation can take
better care of their own utilities. Oak is not a pet project and we
have to admit that its current level of complexity doesn't allow us to
use oak-core and oak-run as dumping grounds anymore.

2016-10-21 14:08 GMT+02:00 Thomas Mueller :
> Hi,
>
>> could adding an oak-core-api with independent lifecycle solve the
>>situation?
>
> "All problems in computer science can be solved by another level of
> indirection"
>
> I would prefer if we get oak-segment-tar in line with the rest of oak
> (release it at the same time and so on). I understand, there are some
> disadvantages. But I think all alternatives also have disadvantages.
>
> Regards,
> Thomas
>
>
>
>
> On 21/10/16 12:46, "Davide Giannella"  wrote:
>
>>Hello team,
>>
>>while integrating Oak with segment-tar in other products, I'm facing
>>quite a struggle with a sort-of circular dependencies. We have
>>segment-tar that depends on oak-core and then we have tools like oak-run
>>or oak-upgrade which depends on both oak-core and segment-tar.
>>
>>this may not be an issue but in case of changes in the API, like for
>>1.5.12 we have the following situation. 1.5.12 has been released with
>>segment-tar 0.0.14 but this mix doesn't actually work on OSGi
>>environment as of API changes. On the other hand, in order to release
>>0.0.16 we need oak-core 1.5.12 with the changes.
>>
>>Now oak-run and other tools may fail, or at least be in an unknown
>>situation.
>>
>>All of this is my understanding and I may be wrong, so please correct me
>>if I'm wrong. I'm right, could adding an oak-core-api with independent
>>lifecycle solve the situation?
>>
>>Davide
>>
>>
>

Re: [VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.16

[Francesco Mari]

+1

2016-10-20 14:17 GMT+02:00 Michael Dürig <mdue...@apache.org>:
>
>
> On 20.10.16 2:08 , Francesco Mari wrote:
>>
>>   [X] +1 Approve the release
>
>
> Michael

[VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.16

[Francesco Mari]

Hi,

We solved 19 issues in this release:
https://issues.apache.org/jira/browse/OAK/fixforversion/12338072

There are still some outstanding issues:
https://issues.apache.org/jira/browse/OAK/component/12329487

Staging repository:
https://repository.apache.org/content/repositories/orgapachejackrabbit-1199

You can use this UNIX script to download the release and verify the signatures:
http://svn.apache.org/repos/asf/jackrabbit/oak/trunk/check_staged_release.sh

Usage:
sh check_staged_release.sh 1199 /tmp/oak-staging

Please vote to approve this release:

  [ ] +1 Approve the release
  [ ]  0 Don't care
  [ ] -1 Don't release, because ...

This majority vote is open for at least 72 hours.

Oak Segment Tar 0.0.16 release plan

[Francesco Mari]

I'm planning to release oak-segment-tar tomorrow. There are currently
some open issues, but none of them is critical. Currently open issues
can be postponed to the next release. Let me know if I'm missing
something.

Re: [VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.14

[Francesco Mari]

+1

Re: another trunk build failure

[Francesco Mari]

I reverted that commit at r1763270.

2016-10-04 14:09 GMT+02:00 Francesco Mari <mari.france...@gmail.com>:
> Yes, I guess I was too fast at enabling that fixture. The version of
> Segment Tar implementing the Checkpoints interface hasn't been
> released and referenced from the rest of Oak yet.
>
> 2016-10-04 14:03 GMT+02:00 Julian Reschke <julian.resc...@gmx.de>:
>> On 2016-10-04 12:08, Julian Reschke wrote:
>>>
>>>
>>>> [ERROR]
>>>> org.apache.jackrabbit.oak.spi.security.authentication.external.basic:
>>>> Version increase required; detected 1.2.0, suggested 1.2.1
>>
>>
>> The above can be mitigated with -Dbase.skip.
>>
>> With that, I get a test failure at:
>>
>>> Tests in error:
>>>
>>> checkpoints[SegmentTar](org.apache.jackrabbit.oak.spi.state.NodeStoreTest):
>>> org.apache.jackrabbit.oak.segment.SegmentNodeStore.checkpoints()Ljava/lang/Iterable;
>>
>>
>> Maybe related to
>>
>>> r1763253 | frm | 2016-10-04 10:09:16 +0100 (Di, 04. Okt 2016) | 2 Zeilen
>>>
>>> OAK-4856 - Enable SEGMENT_TAR fixture in NodeStoreTest.checkpoints()
>>
>>
>> ?
>>
>> Best regards, Julian

Re: another trunk build failure

[Francesco Mari]

Yes, I guess I was too fast at enabling that fixture. The version of
Segment Tar implementing the Checkpoints interface hasn't been
released and referenced from the rest of Oak yet.

2016-10-04 14:03 GMT+02:00 Julian Reschke :
> On 2016-10-04 12:08, Julian Reschke wrote:
>>
>>
>>> [ERROR]
>>> org.apache.jackrabbit.oak.spi.security.authentication.external.basic:
>>> Version increase required; detected 1.2.0, suggested 1.2.1
>
>
> The above can be mitigated with -Dbase.skip.
>
> With that, I get a test failure at:
>
>> Tests in error:
>>
>> checkpoints[SegmentTar](org.apache.jackrabbit.oak.spi.state.NodeStoreTest):
>> org.apache.jackrabbit.oak.segment.SegmentNodeStore.checkpoints()Ljava/lang/Iterable;
>
>
> Maybe related to
>
>> r1763253 | frm | 2016-10-04 10:09:16 +0100 (Di, 04. Okt 2016) | 2 Zeilen
>>
>> OAK-4856 - Enable SEGMENT_TAR fixture in NodeStoreTest.checkpoints()
>
>
> ?
>
> Best regards, Julian

Re: [VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.12

[Francesco Mari]

+1

2016-09-15 18:29 GMT+02:00 Michael Dürig :
> Hi,
>
> We solved 36 issues in this release:
> https://issues.apache.org/jira/browse/OAK/fixforversion/12338153
>
> There are still some outstanding issues:
> https://issues.apache.org/jira/browse/OAK/component/12329487
>
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachejackrabbit-1186
>
> You can use this UNIX script to download the release and verify the
> signatures:
> http://svn.apache.org/repos/asf/jackrabbit/oak/trunk/check_staged_release.sh
>
> Usage:
> sh check_staged_release.sh 1186 /tmp/oak-staging
>
> Please vote to approve this release:
>
>   [ ] +1 Approve the release
>   [ ]  0 Don't care
>   [ ] -1 Don't release, because ...
>
> This majority vote is open for at least 72 hours.
>
> Michael

[RESULT][VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.10

[Francesco Mari]

Hi,

The vote passes as follows:

+1 Francesco Mari
+1 Alex Parvulescu
+1 Michael Dürig

Thanks for voting. I'll push the release out.

[VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.10

[Francesco Mari]

Hi,

We solved 18 issues in this release:
https://issues.apache.org/jira/browse/OAK/fixforversion/12338041

There are still some outstanding issues:
https://issues.apache.org/jira/browse/OAK/component/12329487

Staging repository:
https://repository.apache.org/content/repositories/orgapachejackrabbit-1174

You can use this UNIX script to download the release and verify the signatures:
http://svn.apache.org/repos/asf/jackrabbit/oak/trunk/check_staged_release.sh

Usage:
sh check_staged_release.sh 1174 /tmp/oak-staging

Please vote to approve this release:

  [ ] +1 Approve the release
  [ ]  0 Don't care
  [ ] -1 Don't release, because ...

This majority vote is open for at least 72 hours.

Re: [VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.10

[Francesco Mari]

+1

2016-08-23 18:07 GMT+02:00 Francesco Mari <mari.france...@gmail.com>:
> Hi,
>
> We solved 18 issues in this release:
> https://issues.apache.org/jira/browse/OAK/fixforversion/12338041
>
> There are still some outstanding issues:
> https://issues.apache.org/jira/browse/OAK/component/12329487
>
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachejackrabbit-1174
>
> You can use this UNIX script to download the release and verify the 
> signatures:
> http://svn.apache.org/repos/asf/jackrabbit/oak/trunk/check_staged_release.sh
>
> Usage:
> sh check_staged_release.sh 1174 /tmp/oak-staging
>
> Please vote to approve this release:
>
>   [ ] +1 Approve the release
>   [ ]  0 Don't care
>   [ ] -1 Don't release, because ...
>
> This majority vote is open for at least 72 hours.

Cold standby for oak-segment-tar

[Francesco Mari]

Hi all,

The cold standby functionality originally implemented for oak-segment
is currently unusable with oak-segment-tar. The cold standby requires
access to the SegmentStore API and its lower level functionalities,
but this API has been changed in oak-segment-tar in an incompatible
way.

I would like to restore the possibility of running the cold standby on
top of oak-segment-tar, but I can only see two ways of achieving this.

1. Assume that the cold standby functionality is intimately coupled
with oak-segment, duplicate its code in oak-segment-tar and make the
appropriate adjustments to make it work. No change will be done in
oak-segment or oak-tarmk-standby.

2. Define an API (or, more appropriately, an SPI) sitting in between
oak-tarmk-standby and the segment stores. This API will have two
implementations in oak-segment and oak-segment-tar and will isolate
oak-tarmk-standby from the differences of the two modules.

I would be in favour for the second option for the obvious benefits of
code reuse. What do you guys think about this problem? Is there any
other option I'm missing?

Re: Oak Segment Tar 0.0.8 release plan

[Francesco Mari]

The main topic seems to be that we need a way to identify the issues
that we want to solve for a stable release of oak-segment-tar. In a
quick offline conversation with Michael, we came up with the following
ideas - to be discussed - to track issues.

1. If an issue is not essential for the stable release of
oak-segment-tar, leave the fix version empty.

2. If an issue is essential for the stable release of oak-segment-tar,
set the fix version to one of the upcoming releases.

3. When a release is performed, if there are unresolved issues for
that release, move those issues to the next release.

Points 2 and 3 guarantee that the issues that we consider important
have visibility on the mailing list. If an issue is unresolved and is
postponed to the next release it will appear on the mailing list.
Point 1 guarantees that this will not happen for unimportant or
unplanned issues, reducing the noise considerably. This approach also
avoids the creation of Jira versions that have no meaning from a
semantic versioning point of view.

Looking forward to your opinions about this.

2016-07-29 15:21 GMT+02:00 Francesco Mari <mari.france...@gmail.com>:
> The question to answer now is what will happen if we release the 1.0.0
> version and we end up having 1.1.13 in Oak 1.6.
>
>
> On Jul 29, 2016 12:16 PM, "Michael Dürig" <mdue...@apache.org> wrote:
>>
>>
>> Hi,
>>
>> I went for adding a "Segment Tar 1.0.0" version. So far this is a place
>> holder for all we want to have released at the same time we release Oak 1.6.
>> That is, it is the "recommended version" to use along with Oak 1.6.
>>
>> Michael
>>
>>
>> On 29.7.16 10:57 , Davide Giannella wrote:
>>>
>>> On 27/07/2016 16:57, Michael Dürig wrote:
>>>>
>>>>
>>>> In a quick chat with Francesco we came up with a couple of ways of
>>>> achieving this. I.e. we want to have a mechanism to differentiate in
>>>> Jira between issues in oak-segment-tar that we want to have fixed once
>>>> Oak 1.6. is out and those that can be deferred:
>>>>
>>>> 1) Use fix version 1.6. This is confusing as 1.6 has no relation to
>>>> oak-segment-tar whatsoever and we risk messing up the release notes
>>>> for Oak 1.6
>>>>
>>>> 2) Introduce a fake version in Jira ("Oak Segment For Oak 1.6"). This
>>>> is a misuse of the version field probably leading to confusion later on.
>>>>
>>>> 3) Use a label. Might work but then labels are s overloaded and
>>>> weak. Also it is difficult to spot and align them in a table in Jira.
>>>>
>>>> 4) Resolve as later. But later is usually never...
>>>>
>>>> 5) Use the assignee field: unassigned issues are tackled post Oak 1.6
>>>> all others are planed to be tackled until Oak 1.6.
>>>>
>>>> My preference would be 5).
>>>
>>>
>>> Well the fixVersion field can have multiple values. So I would say that
>>> for segment we have 2 or 3 targets for solving issues.
>>>
>>> 1) stuff that needs to be solved before oak 1.6
>>> 2) stuff that needs to be solved for a specific major segment version;
>>> let's say 1.0
>>> 3) stuff that needs to be solved by the next segment release; ie
>>> blockers.
>>>
>>> I would suggest then
>>>
>>> - Rename version 1.6 as "Oak 1.6"
>>> - Assign such version to segment versions that *have to* be solved and
>>> included by 1.6
>>> - create a "Segment 1.0" version and give it to what you think should go
>>> in 1.0. IMO Segment 1.0 does not mean Oak 1.6. It may be the same but I
>>> see the two very different.
>>>
>>> Thoughts?
>>>
>>> Davide
>>>
>>>
>

Re: Are dumb segments dumb?

[Francesco Mari]

The recent discovery in OAK-4604 shows that my POC suffers from the
same problem. I fixed it in my latest commit.

2016-07-26 11:47 GMT+02:00 Francesco Mari <mari.france...@gmail.com>:
> With my latest commits on this branch [1] I enabled every previously
> ignored test, fixing them when needed., The only two exceptions are
> RecordUsageAnalyserTest and SegmentSizeTest, that were simply deleted.
> I also added a couple of tests to cover the cases that work slightly
> differently than before.
>
> [1]: https://github.com/francescomari/jackrabbit-oak/tree/dumb
>
> 2016-07-25 17:48 GMT+02:00 Francesco Mari <mari.france...@gmail.com>:
>> It might be a variation in the process I tried. This shouldn't affect
>> much the statistics anyway, given that the population sample is big
>> enough in both cases.
>>
>> 2016-07-25 17:46 GMT+02:00 Michael Dürig <mdue...@apache.org>:
>>>
>>> Interesting numbers. Most of them look as I would have expected. I.e. the
>>> distributions in the dumb case are more regular (smaller std. dev, mean and
>>> median closer to each other), bigger segment sizes, etc.
>>>
>>> What I don't understand is the total number of records. These numbers differ
>>> greatly between current and dumb. Is this a test artefact (i.e. test not
>>> reproducible) or are we missing out on something.
>>>
>>> Michael
>>>
>>>
>>> On 25.7.16 4:01 , Francesco Mari wrote:
>>>>
>>>> I put together some statistics [1] for the process I described above.
>>>> The "dumb" variant requires more segments to store the same amount of
>>>> data, because of the increased size of serialised record IDs.  As you
>>>> can see the amount of records per segment is definitely lower in the
>>>> dumb variant.
>>>>
>>>> On the other hand, ignoring the growth of segment ID reference table
>>>> seems to be a good choice. As shown from the segment size average,
>>>> dumb segments are usually fuller that their counterpart. Moreover, a
>>>> lower standard deviation shows that it's more common to have full dumb
>>>> segments.
>>>>
>>>> In addition, my analysis seems to have found a bug too. There are a
>>>> lot of segments with no segment ID references and only one record,
>>>> which is very likely to be the segment info. The flush thread writes
>>>> every 5 seconds the current segment buffer, provided that the buffer
>>>> is not empty. It turns out that a segment buffer is never empty, since
>>>> it always contains at least one record. As such, we are currently
>>>> leaking almost empty segments every 5 seconds, that waste additional
>>>> space on disk because of the padding required by the TAR format.
>>>>
>>>> [1]:
>>>> https://docs.google.com/spreadsheets/d/1gXhmPsm4rDyHnle4TUh-mtB2HRtRyADXALARRFDh7z4/edit?usp=sharing
>>>>
>>>> 2016-07-25 10:05 GMT+02:00 Michael Dürig <mdue...@apache.org>:
>>>>>
>>>>>
>>>>> Hi Jukka,
>>>>>
>>>>> Thanks for sharing your perspective and the historical background.
>>>>>
>>>>> I agree that repository size shouldn't be a primary concern. However, we
>>>>> have seen many repositories (especially with an external data store)
>>>>> where
>>>>> the content is extremely fine granular. Much more than in an initial
>>>>> content
>>>>> installation of CQ (which I believe was one of the initial setup for
>>>>> collecting statistics). So we should at least understand the impact of
>>>>> the
>>>>> patch in various scenarios.
>>>>>
>>>>> My main concern is the cache footprint of node records. Those are made up
>>>>> of
>>>>> a list of record ids and would thus grow by a factor of 6 with the
>>>>> current
>>>>> patch.
>>>>>
>>>>> Locality is not so much of concern here. I would expect it to actually
>>>>> improve as the patch gets rid of the 255 references limit of segments. A
>>>>> limit which in practical deployments leads to degeneration of segment
>>>>> sizes
>>>>> (I regularly see median sizes below 5k). See OAK-2896 for some background
>>>>> on
>>>>> this.
>>>>> Furthermore we already did a big step forward in improving locality in
>>>>> concurr

Re: Are dumb segments dumb?

[Francesco Mari]

With my latest commits on this branch [1] I enabled every previously
ignored test, fixing them when needed., The only two exceptions are
RecordUsageAnalyserTest and SegmentSizeTest, that were simply deleted.
I also added a couple of tests to cover the cases that work slightly
differently than before.

[1]: https://github.com/francescomari/jackrabbit-oak/tree/dumb

2016-07-25 17:48 GMT+02:00 Francesco Mari <mari.france...@gmail.com>:
> It might be a variation in the process I tried. This shouldn't affect
> much the statistics anyway, given that the population sample is big
> enough in both cases.
>
> 2016-07-25 17:46 GMT+02:00 Michael Dürig <mdue...@apache.org>:
>>
>> Interesting numbers. Most of them look as I would have expected. I.e. the
>> distributions in the dumb case are more regular (smaller std. dev, mean and
>> median closer to each other), bigger segment sizes, etc.
>>
>> What I don't understand is the total number of records. These numbers differ
>> greatly between current and dumb. Is this a test artefact (i.e. test not
>> reproducible) or are we missing out on something.
>>
>> Michael
>>
>>
>> On 25.7.16 4:01 , Francesco Mari wrote:
>>>
>>> I put together some statistics [1] for the process I described above.
>>> The "dumb" variant requires more segments to store the same amount of
>>> data, because of the increased size of serialised record IDs.  As you
>>> can see the amount of records per segment is definitely lower in the
>>> dumb variant.
>>>
>>> On the other hand, ignoring the growth of segment ID reference table
>>> seems to be a good choice. As shown from the segment size average,
>>> dumb segments are usually fuller that their counterpart. Moreover, a
>>> lower standard deviation shows that it's more common to have full dumb
>>> segments.
>>>
>>> In addition, my analysis seems to have found a bug too. There are a
>>> lot of segments with no segment ID references and only one record,
>>> which is very likely to be the segment info. The flush thread writes
>>> every 5 seconds the current segment buffer, provided that the buffer
>>> is not empty. It turns out that a segment buffer is never empty, since
>>> it always contains at least one record. As such, we are currently
>>> leaking almost empty segments every 5 seconds, that waste additional
>>> space on disk because of the padding required by the TAR format.
>>>
>>> [1]:
>>> https://docs.google.com/spreadsheets/d/1gXhmPsm4rDyHnle4TUh-mtB2HRtRyADXALARRFDh7z4/edit?usp=sharing
>>>
>>> 2016-07-25 10:05 GMT+02:00 Michael Dürig <mdue...@apache.org>:
>>>>
>>>>
>>>> Hi Jukka,
>>>>
>>>> Thanks for sharing your perspective and the historical background.
>>>>
>>>> I agree that repository size shouldn't be a primary concern. However, we
>>>> have seen many repositories (especially with an external data store)
>>>> where
>>>> the content is extremely fine granular. Much more than in an initial
>>>> content
>>>> installation of CQ (which I believe was one of the initial setup for
>>>> collecting statistics). So we should at least understand the impact of
>>>> the
>>>> patch in various scenarios.
>>>>
>>>> My main concern is the cache footprint of node records. Those are made up
>>>> of
>>>> a list of record ids and would thus grow by a factor of 6 with the
>>>> current
>>>> patch.
>>>>
>>>> Locality is not so much of concern here. I would expect it to actually
>>>> improve as the patch gets rid of the 255 references limit of segments. A
>>>> limit which in practical deployments leads to degeneration of segment
>>>> sizes
>>>> (I regularly see median sizes below 5k). See OAK-2896 for some background
>>>> on
>>>> this.
>>>> Furthermore we already did a big step forward in improving locality in
>>>> concurrent write scenarios when we introduced the
>>>> SegmentBufferWriterPool.
>>>> In essence: thread affinity for segments.
>>>>
>>>> We should probably be more carefully looking at the micro benchmarks. I
>>>> guess we neglected this part a bit in the past. Unfortunately CI
>>>> infrastructure isn't making this easy for us... OTOH those benchmarks
>>>> only
>>>> tell you so much. Many of the problems we recently faced only surfaced in
>>&g

Re: Are dumb segments dumb?

[Francesco Mari]

It might be a variation in the process I tried. This shouldn't affect
much the statistics anyway, given that the population sample is big
enough in both cases.

2016-07-25 17:46 GMT+02:00 Michael Dürig <mdue...@apache.org>:
>
> Interesting numbers. Most of them look as I would have expected. I.e. the
> distributions in the dumb case are more regular (smaller std. dev, mean and
> median closer to each other), bigger segment sizes, etc.
>
> What I don't understand is the total number of records. These numbers differ
> greatly between current and dumb. Is this a test artefact (i.e. test not
> reproducible) or are we missing out on something.
>
> Michael
>
>
> On 25.7.16 4:01 , Francesco Mari wrote:
>>
>> I put together some statistics [1] for the process I described above.
>> The "dumb" variant requires more segments to store the same amount of
>> data, because of the increased size of serialised record IDs.  As you
>> can see the amount of records per segment is definitely lower in the
>> dumb variant.
>>
>> On the other hand, ignoring the growth of segment ID reference table
>> seems to be a good choice. As shown from the segment size average,
>> dumb segments are usually fuller that their counterpart. Moreover, a
>> lower standard deviation shows that it's more common to have full dumb
>> segments.
>>
>> In addition, my analysis seems to have found a bug too. There are a
>> lot of segments with no segment ID references and only one record,
>> which is very likely to be the segment info. The flush thread writes
>> every 5 seconds the current segment buffer, provided that the buffer
>> is not empty. It turns out that a segment buffer is never empty, since
>> it always contains at least one record. As such, we are currently
>> leaking almost empty segments every 5 seconds, that waste additional
>> space on disk because of the padding required by the TAR format.
>>
>> [1]:
>> https://docs.google.com/spreadsheets/d/1gXhmPsm4rDyHnle4TUh-mtB2HRtRyADXALARRFDh7z4/edit?usp=sharing
>>
>> 2016-07-25 10:05 GMT+02:00 Michael Dürig <mdue...@apache.org>:
>>>
>>>
>>> Hi Jukka,
>>>
>>> Thanks for sharing your perspective and the historical background.
>>>
>>> I agree that repository size shouldn't be a primary concern. However, we
>>> have seen many repositories (especially with an external data store)
>>> where
>>> the content is extremely fine granular. Much more than in an initial
>>> content
>>> installation of CQ (which I believe was one of the initial setup for
>>> collecting statistics). So we should at least understand the impact of
>>> the
>>> patch in various scenarios.
>>>
>>> My main concern is the cache footprint of node records. Those are made up
>>> of
>>> a list of record ids and would thus grow by a factor of 6 with the
>>> current
>>> patch.
>>>
>>> Locality is not so much of concern here. I would expect it to actually
>>> improve as the patch gets rid of the 255 references limit of segments. A
>>> limit which in practical deployments leads to degeneration of segment
>>> sizes
>>> (I regularly see median sizes below 5k). See OAK-2896 for some background
>>> on
>>> this.
>>> Furthermore we already did a big step forward in improving locality in
>>> concurrent write scenarios when we introduced the
>>> SegmentBufferWriterPool.
>>> In essence: thread affinity for segments.
>>>
>>> We should probably be more carefully looking at the micro benchmarks. I
>>> guess we neglected this part a bit in the past. Unfortunately CI
>>> infrastructure isn't making this easy for us... OTOH those benchmarks
>>> only
>>> tell you so much. Many of the problems we recently faced only surfaced in
>>> the large: huge repos, high concurrent load, many days of traffic.
>>>
>>> Michael
>>>
>>>
>>>
>>>
>>>
>>> On 23.7.16 12:34 , Jukka Zitting wrote:
>>>>
>>>>
>>>> Hi,
>>>>
>>>> Cool! I'm pretty sure there are various ways in which the format could
>>>> be
>>>> improved, as the original design was based mostly on intuition, guided
>>>> somewhat by collected stats
>>>> <http://markmail.org/message/kxe3iy2hnodxsghe>
>>>> and
>>>> the micro-benchmarks <https://issues.apache.org/jira/browse/OAK-119>
>>>> used
>>>> to optimize commo

Re: Are dumb segments dumb?

[Francesco Mari]

I opened OAK-4596 to track the segment leak.

2016-07-25 16:01 GMT+02:00 Francesco Mari <mari.france...@gmail.com>:
> I put together some statistics [1] for the process I described above.
> The "dumb" variant requires more segments to store the same amount of
> data, because of the increased size of serialised record IDs.  As you
> can see the amount of records per segment is definitely lower in the
> dumb variant.
>
> On the other hand, ignoring the growth of segment ID reference table
> seems to be a good choice. As shown from the segment size average,
> dumb segments are usually fuller that their counterpart. Moreover, a
> lower standard deviation shows that it's more common to have full dumb
> segments.
>
> In addition, my analysis seems to have found a bug too. There are a
> lot of segments with no segment ID references and only one record,
> which is very likely to be the segment info. The flush thread writes
> every 5 seconds the current segment buffer, provided that the buffer
> is not empty. It turns out that a segment buffer is never empty, since
> it always contains at least one record. As such, we are currently
> leaking almost empty segments every 5 seconds, that waste additional
> space on disk because of the padding required by the TAR format.
>
> [1]: 
> https://docs.google.com/spreadsheets/d/1gXhmPsm4rDyHnle4TUh-mtB2HRtRyADXALARRFDh7z4/edit?usp=sharing
>
> 2016-07-25 10:05 GMT+02:00 Michael Dürig <mdue...@apache.org>:
>>
>> Hi Jukka,
>>
>> Thanks for sharing your perspective and the historical background.
>>
>> I agree that repository size shouldn't be a primary concern. However, we
>> have seen many repositories (especially with an external data store) where
>> the content is extremely fine granular. Much more than in an initial content
>> installation of CQ (which I believe was one of the initial setup for
>> collecting statistics). So we should at least understand the impact of the
>> patch in various scenarios.
>>
>> My main concern is the cache footprint of node records. Those are made up of
>> a list of record ids and would thus grow by a factor of 6 with the current
>> patch.
>>
>> Locality is not so much of concern here. I would expect it to actually
>> improve as the patch gets rid of the 255 references limit of segments. A
>> limit which in practical deployments leads to degeneration of segment sizes
>> (I regularly see median sizes below 5k). See OAK-2896 for some background on
>> this.
>> Furthermore we already did a big step forward in improving locality in
>> concurrent write scenarios when we introduced the SegmentBufferWriterPool.
>> In essence: thread affinity for segments.
>>
>> We should probably be more carefully looking at the micro benchmarks. I
>> guess we neglected this part a bit in the past. Unfortunately CI
>> infrastructure isn't making this easy for us... OTOH those benchmarks only
>> tell you so much. Many of the problems we recently faced only surfaced in
>> the large: huge repos, high concurrent load, many days of traffic.
>>
>> Michael
>>
>>
>>
>>
>>
>> On 23.7.16 12:34 , Jukka Zitting wrote:
>>>
>>> Hi,
>>>
>>> Cool! I'm pretty sure there are various ways in which the format could be
>>> improved, as the original design was based mostly on intuition, guided
>>> somewhat by collected stats <http://markmail.org/message/kxe3iy2hnodxsghe>
>>> and
>>> the micro-benchmarks <https://issues.apache.org/jira/browse/OAK-119> used
>>> to optimize common operations.
>>>
>>> Note though that the total size of the repository was not and probably
>>> shouldn't be a primary metric, since the size of a typical repository is
>>> governed mostly by binaries and string properties (though it's a good idea
>>> to make sure you avoid things like duplicates of large binaries). Instead
>>> the rationale for squeezing things like record ids to as few bytes as
>>> possible is captured in the principles listed in the original design doc
>>> <http://jackrabbit.apache.org/oak/docs/nodestore/segmentmk.html>:
>>>
>>>- Compactness. The formatting of records is optimized for size to
>>> reduce
>>>IO costs and to fit as much content in caches as possible. A node
>>> stored in
>>>SegmentNodeStore typically consumes only a fraction of the size it
>>> would as
>>>a bundle in Jackrabbit Classic.
>>>- Locality. Segments are written so that related records, like a node
>>>    and its immediate

Re: Are dumb segments dumb?

[Francesco Mari]

I put together some statistics [1] for the process I described above.
The "dumb" variant requires more segments to store the same amount of
data, because of the increased size of serialised record IDs.  As you
can see the amount of records per segment is definitely lower in the
dumb variant.

On the other hand, ignoring the growth of segment ID reference table
seems to be a good choice. As shown from the segment size average,
dumb segments are usually fuller that their counterpart. Moreover, a
lower standard deviation shows that it's more common to have full dumb
segments.

In addition, my analysis seems to have found a bug too. There are a
lot of segments with no segment ID references and only one record,
which is very likely to be the segment info. The flush thread writes
every 5 seconds the current segment buffer, provided that the buffer
is not empty. It turns out that a segment buffer is never empty, since
it always contains at least one record. As such, we are currently
leaking almost empty segments every 5 seconds, that waste additional
space on disk because of the padding required by the TAR format.

[1]: 
https://docs.google.com/spreadsheets/d/1gXhmPsm4rDyHnle4TUh-mtB2HRtRyADXALARRFDh7z4/edit?usp=sharing

2016-07-25 10:05 GMT+02:00 Michael Dürig <mdue...@apache.org>:
>
> Hi Jukka,
>
> Thanks for sharing your perspective and the historical background.
>
> I agree that repository size shouldn't be a primary concern. However, we
> have seen many repositories (especially with an external data store) where
> the content is extremely fine granular. Much more than in an initial content
> installation of CQ (which I believe was one of the initial setup for
> collecting statistics). So we should at least understand the impact of the
> patch in various scenarios.
>
> My main concern is the cache footprint of node records. Those are made up of
> a list of record ids and would thus grow by a factor of 6 with the current
> patch.
>
> Locality is not so much of concern here. I would expect it to actually
> improve as the patch gets rid of the 255 references limit of segments. A
> limit which in practical deployments leads to degeneration of segment sizes
> (I regularly see median sizes below 5k). See OAK-2896 for some background on
> this.
> Furthermore we already did a big step forward in improving locality in
> concurrent write scenarios when we introduced the SegmentBufferWriterPool.
> In essence: thread affinity for segments.
>
> We should probably be more carefully looking at the micro benchmarks. I
> guess we neglected this part a bit in the past. Unfortunately CI
> infrastructure isn't making this easy for us... OTOH those benchmarks only
> tell you so much. Many of the problems we recently faced only surfaced in
> the large: huge repos, high concurrent load, many days of traffic.
>
> Michael
>
>
>
>
>
> On 23.7.16 12:34 , Jukka Zitting wrote:
>>
>> Hi,
>>
>> Cool! I'm pretty sure there are various ways in which the format could be
>> improved, as the original design was based mostly on intuition, guided
>> somewhat by collected stats <http://markmail.org/message/kxe3iy2hnodxsghe>
>> and
>> the micro-benchmarks <https://issues.apache.org/jira/browse/OAK-119> used
>> to optimize common operations.
>>
>> Note though that the total size of the repository was not and probably
>> shouldn't be a primary metric, since the size of a typical repository is
>> governed mostly by binaries and string properties (though it's a good idea
>> to make sure you avoid things like duplicates of large binaries). Instead
>> the rationale for squeezing things like record ids to as few bytes as
>> possible is captured in the principles listed in the original design doc
>> <http://jackrabbit.apache.org/oak/docs/nodestore/segmentmk.html>:
>>
>>- Compactness. The formatting of records is optimized for size to
>> reduce
>>IO costs and to fit as much content in caches as possible. A node
>> stored in
>>SegmentNodeStore typically consumes only a fraction of the size it
>> would as
>>a bundle in Jackrabbit Classic.
>>- Locality. Segments are written so that related records, like a node
>>and its immediate children, usually end up stored in the same segment.
>> This
>>makes tree traversals very fast and avoids most cache misses for
>> typical
>>clients that access more than one related node per session.
>>
>> Thus I would recommend keeping an eye also on benchmark results in
>> addition
>> to raw repository size when evaluating possible improvements. Also, the
>> number and size of data segments are good size metrics to look at in
>> additio

[RESULT][VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.6

[Francesco Mari]

Hi,

The vote passes as follows:

+1 Francesco Mari
+1 Michael Dürig
+1 Alex Parvulescu

Thanks for voting. I'll push the release out.

Re: Are dumb segments dumb?

[Francesco Mari]

The impact on repository size needs to be assessed with more specific
tests. In particular, I found RecordUsageAnalyserTest and
SegmentSizeTest unsuitable to this task. It's not a coincidence that
these tests are usually the first to be disabled or blindly updated
every time a small fix changes the size of the records.

Regarding GC, the segment graph could be computed during the mark
phase. Of course, it's handy to have this information pre-computed for
you, but since the record graph is traversed anyway we could think
about dynamically reconstructing the segment graph when needed.

There are still so many questions to answer, but I think that this
simplification exercise can be worth the effort.

2016-07-22 11:34 GMT+02:00 Michael Dürig <mdue...@apache.org>:
>
> Hi,
>
> Neat! I would have expected a greater impact on the size of the segment
> store. But as you say it probably all depends on the binary/content ratio. I
> think we should look at the #references / repository size ratio for
> repositories of different structures and see how such a number differs with
> and without the patch.
>
> I like the patch as it fixes OAK-2896 while at the same time reducing
> complexity a lot.
>
> OTOH we need to figure out how to regain the lost functionality (e.g. gc)
> and asses its impact on repository size.
>
> Michael
>
>
>
> On 22.7.16 11:32 , Francesco Mari wrote:
>>
>> Hi,
>>
>> Yesterday I took some time for a little experiment: how many
>> optimisations can be removed from the current segment format while
>> maintaining the same functionality?
>>
>> I made some work in a branch on GitHub [1]. The code on that branch is
>> similar to the current trunk except for the following changes:
>>
>> 1. Record IDs are always serialised in their entirety. As such, a
>> serialised record ID occupies 18 bytes instead of 3.
>>
>> 2. Because of the previous change, the table of referenced segment IDs
>> is not needed anymore, so I removed it from the segment header. It
>> turns out that this table is indeed needed for the mark phase of
>> compaction, so this feature is broken in that branch.
>>
>> Anyway, since the code is in a runnable state, I generated some
>> content using the current trunk and the dumber version of
>> oak-segment-tar. This is the repository created by the dumb
>> oak-segment-tar:
>>
>> 524744 data0a.tar
>> 524584 data1a.tar
>> 524688 data2a.tar
>> 460896 data3a.tar
>> 8 journal.log
>> 0 repo.lock
>>
>> This is the one created by the current trunk:
>>
>> 524864 data0a.tar
>> 524656 data1a.tar
>> 524792 data2a.tar
>> 297288 data3a.tar
>> 8 journal.log
>> 0 repo.lock
>>
>> The process that generates the content doesn't change between the two
>> executions, and the generated content is coming from a real world
>> scenario. For those familiar with it, the content is generated by an
>> installation of Adobe Experience Manager.
>>
>> It looks like that the size of the repository is not changing so much.
>> Probably the de-optimisation in the small is dwarfed by the binary
>> content in the large. Another effect of my change is that there is no
>> limit on the number of referenced segment IDs per segment, and this
>> might allow segments to pack more records than before.
>>
>> Questions apart, the clear advantage of this change is a great
>> simplification of the code. I guess I can remove some lines more, but
>> what I peeled off is already a considerable amount. Look at the code!
>>
>> Francesco
>>
>> [1]: https://github.com/francescomari/jackrabbit-oak/tree/dumb
>>
>

Are dumb segments dumb?

[Francesco Mari]

Hi,

Yesterday I took some time for a little experiment: how many
optimisations can be removed from the current segment format while
maintaining the same functionality?

I made some work in a branch on GitHub [1]. The code on that branch is
similar to the current trunk except for the following changes:

1. Record IDs are always serialised in their entirety. As such, a
serialised record ID occupies 18 bytes instead of 3.

2. Because of the previous change, the table of referenced segment IDs
is not needed anymore, so I removed it from the segment header. It
turns out that this table is indeed needed for the mark phase of
compaction, so this feature is broken in that branch.

Anyway, since the code is in a runnable state, I generated some
content using the current trunk and the dumber version of
oak-segment-tar. This is the repository created by the dumb
oak-segment-tar:

524744 data0a.tar
524584 data1a.tar
524688 data2a.tar
460896 data3a.tar
8 journal.log
0 repo.lock

This is the one created by the current trunk:

524864 data0a.tar
524656 data1a.tar
524792 data2a.tar
297288 data3a.tar
8 journal.log
0 repo.lock

The process that generates the content doesn't change between the two
executions, and the generated content is coming from a real world
scenario. For those familiar with it, the content is generated by an
installation of Adobe Experience Manager.

It looks like that the size of the repository is not changing so much.
Probably the de-optimisation in the small is dwarfed by the binary
content in the large. Another effect of my change is that there is no
limit on the number of referenced segment IDs per segment, and this
might allow segments to pack more records than before.

Questions apart, the clear advantage of this change is a great
simplification of the code. I guess I can remove some lines more, but
what I peeled off is already a considerable amount. Look at the code!

Francesco

[1]: https://github.com/francescomari/jackrabbit-oak/tree/dumb

Re: [VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.6

[Francesco Mari]

+1

2016-07-19 16:00 GMT+02:00 Francesco Mari <mari.france...@gmail.com>:

> Hi,
>
> We solved 5 issues in this release:
> https://issues.apache.org/jira/browse/OAK/fixforversion/12337966
>
> There are still some outstanding issues:
> https://issues.apache.org/jira/browse/OAK/component/12329487
>
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachejackrabbit-1162
>
> You can use this UNIX script to download the release and verify the
> signatures:
>
> http://svn.apache.org/repos/asf/jackrabbit/oak/trunk/check_staged_release.sh
>
> Usage:
> sh check_staged_release.sh 1162 /tmp/oak-staging
>
> Please vote to approve this release:
>
>   [ ] +1 Approve the release
>   [ ]  0 Don't care
>   [ ] -1 Don't release, because ...
>
> This majority vote is open for at least 72 hours.
>
>

[VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.6

[Francesco Mari]

Hi,

We solved 5 issues in this release:
https://issues.apache.org/jira/browse/OAK/fixforversion/12337966

There are still some outstanding issues:
https://issues.apache.org/jira/browse/OAK/component/12329487

Staging repository:
https://repository.apache.org/content/repositories/orgapachejackrabbit-1162

You can use this UNIX script to download the release and verify the
signatures:
http://svn.apache.org/repos/asf/jackrabbit/oak/trunk/check_staged_release.sh

Usage:
sh check_staged_release.sh 1162 /tmp/oak-staging

Please vote to approve this release:

  [ ] +1 Approve the release
  [ ]  0 Don't care
  [ ] -1 Don't release, because ...

This majority vote is open for at least 72 hours.

[RESULT][VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.4

[Francesco Mari]

Hi,

The vote passes as follows:

+1 Francesco Mari
+1 Alex Parvulescu
+1 Tommaso Teofili

Thanks for voting. I'll push the release out.

Re: svn commit: r1752601 - in /jackrabbit/oak/trunk/oak-segment-tar: pom.xml src/main/java/org/apache/jackrabbit/oak/segment/SegmentWriter.java

[Francesco Mari]

I embedded Commons Math to make testing easier. I opened OAK-4561 to track
the removal of the embedded dependency as soon as possible.

2016-07-14 11:57 GMT+02:00 Chetan Mehrotra :

> On Thu, Jul 14, 2016 at 2:04 PM,   wrote:
> >
> > +
> commons-math3
>
> commons-math is a 2.1 MB jar. Would it be possible to avoid embedding
> it whole and only have some parts embedded/copied. (See [1] for an
> example)
>
> Chetan Mehrotra
> [1] https://issues.apache.org/jira/browse/SLING-2361
>

Re: [VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.4

[Francesco Mari]

+1

2016-07-11 17:10 GMT+02:00 Francesco Mari <mari.france...@gmail.com>:

> Hi,
>
> We solved 3 issues in this release:
> https://issues.apache.org/jira/browse/OAK/fixforversion/12336855
>
> There are still some outstanding issues:
> https://issues.apache.org/jira/browse/OAK/component/12329487
>
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachejackrabbit-1156
>
> You can use this UNIX script to download the release and verify the
> signatures:
>
> http://svn.apache.org/repos/asf/jackrabbit/oak/trunk/check_staged_release.sh
>
> Usage:
> sh check_staged_release.sh 1156 /tmp/oak-staging
>
> Please vote to approve this release:
>
>   [ ] +1 Approve the release
>   [ ]  0 Don't care
>   [ ] -1 Don't release, because ...
>
> This majority vote is open for at least 72 hours.
>
>

[VOTE] Release Apache Jackrabbit Oak Segment Tar version 0.0.4

[Francesco Mari]

Hi,

We solved 3 issues in this release:
https://issues.apache.org/jira/browse/OAK/fixforversion/12336855

There are still some outstanding issues:
https://issues.apache.org/jira/browse/OAK/component/12329487

Staging repository:
https://repository.apache.org/content/repositories/orgapachejackrabbit-1156

You can use this UNIX script to download the release and verify the
signatures:
http://svn.apache.org/repos/asf/jackrabbit/oak/trunk/check_staged_release.sh

Usage:
sh check_staged_release.sh 1156 /tmp/oak-staging

Please vote to approve this release:

  [ ] +1 Approve the release
  [ ]  0 Don't care
  [ ] -1 Don't release, because ...

This majority vote is open for at least 72 hours.

Re: svn commit: r1750601 - in /jackrabbit/oak/trunk: oak-segment-tar/ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/ oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/

[Francesco Mari]

Thanks a lot for your understanding!

2016-06-29 10:43 GMT+02:00 Chetan Mehrotra <chetan.mehro...@gmail.com>:

> On Wed, Jun 29, 2016 at 1:25 PM, Francesco Mari
> <mari.france...@gmail.com> wrote:
> > oak-segment-tar should be releasable at any time. If I had to launch a
> quick patch release this morning, I would have to either revert your commit
> or postpone my release until Oak is released.
>
> Given the current release frequency on trunk (2 week) I do not think
> it should be a big problem and holding of commits break the continuity
> and increases work. But then that might be just an issue for me!
>
> For now I have reverted the changes from oak-segment-tar
>
> Chetan Mehrotra
>

Re: svn commit: r1750601 - in /jackrabbit/oak/trunk: oak-segment-tar/ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/ oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/

[Francesco Mari]

The rationale behind this is that we put a lot of effort in making
oak-segment-tar independent from the release process of Oak and I want to
keep it this way.

oak-segment-tar should be releasable at any time. If I had to launch a
quick patch release this morning, I would have to either revert your commit
or postpone my release until Oak is released.

Please don't let Oak impact the way we decided to manage oak-segment-tar.
On Jun 29, 2016 9:25 AM, "Chetan Mehrotra" <chetan.mehro...@gmail.com>
wrote:

> Hi Francesco,
>
> On Wed, Jun 29, 2016 at 12:49 PM, Francesco Mari
> <mari.france...@gmail.com> wrote:
> > Please do not change the "oak.version" property to a snapshot version. If
> > your change relies on code that is only available in the latest snapshot
> of
> > Oak, please revert this commit and hold it back until a proper release of
> > Oak is performed.
>
> I can do that but want to understand the impact here if we switched to
> SNAPSHOT version?
>
> For e.g. in the past we had done some changes in jackrabbit which is
> need in oak then we had switched to snapshot version of JR2 and later
> reverted to released version once JR2 release is done. That has worked
> fine so far and we did not had to hold the feature work for that. So
> want to understand why it should be different here
>
> Chetan Mehrotra
>

Fwd: svn commit: r1750601 - in /jackrabbit/oak/trunk: oak-segment-tar/ oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/ oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/

[Francesco Mari]

Hi Chetan,

Please do not change the "oak.version" property to a snapshot version. If
your change relies on code that is only available in the latest snapshot of
Oak, please revert this commit and hold it back until a proper release of
Oak is performed.

Thanks,

Francesco

-- Forwarded message --
From: 
Date: 2016-06-29 8:26 GMT+02:00
Subject: svn commit: r1750601 - in /jackrabbit/oak/trunk: oak-segment-tar/
oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/
oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/
oak-segment/src/main/java/org/apache/jackrabbit/oak/plug...
To: oak-comm...@jackrabbit.apache.org


Author: chetanm
Date: Wed Jun 29 06:26:19 2016
New Revision: 1750601

URL: http://svn.apache.org/viewvc?rev=1750601=rev
Log:
OAK-4490 - Expose SegmentNodeStore as a secondary NodeStore

Exposes a barebone SegmentNodeStore. This instance is not listening for
observor etc and not directly exposed as a NodeStore but instance exposed
via NodeStoreProvider

Modified:
jackrabbit/oak/trunk/oak-segment-tar/pom.xml

jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreService.java

jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreServiceTest.java

jackrabbit/oak/trunk/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentNodeStoreService.java

jackrabbit/oak/trunk/oak-segment/src/test/java/org/apache/jackrabbit/oak/plugins/segment/SegmentNodeStoreServiceTest.java

Modified: jackrabbit/oak/trunk/oak-segment-tar/pom.xml
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/pom.xml?rev=1750601=1750600=1750601=diff
==
--- jackrabbit/oak/trunk/oak-segment-tar/pom.xml (original)
+++ jackrabbit/oak/trunk/oak-segment-tar/pom.xml Wed Jun 29 06:26:19 2016
@@ -34,7 +34,7 @@
 Oak Segment Tar

 
-1.5.3
+1.6-SNAPSHOT
 

 

Modified:
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreService.java
URL:
http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreService.java?rev=1750601=1750600=1750601=diff
==
---
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreService.java
(original)
+++
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStoreService.java
Wed Jun 29 06:26:19 2016
@@ -81,6 +81,7 @@ import org.apache.jackrabbit.oak.spi.com
 import org.apache.jackrabbit.oak.spi.gc.GCMonitor;
 import org.apache.jackrabbit.oak.spi.gc.GCMonitorTracker;
 import org.apache.jackrabbit.oak.spi.state.NodeStore;
+import org.apache.jackrabbit.oak.spi.state.NodeStoreProvider;
 import org.apache.jackrabbit.oak.spi.state.ProxyNodeStore;
 import org.apache.jackrabbit.oak.spi.state.RevisionGC;
 import org.apache.jackrabbit.oak.spi.state.RevisionGCMBean;
@@ -196,6 +197,13 @@ public class SegmentNodeStoreService ext
 )
 public static final String STANDBY = "standby";

+@Property(
+boolValue = false,
+label = "Secondary Store Mode",
+description = "Flag indicating that this component will not
register as a NodeStore but just as a SecondaryNodeStoreProvider"
+)
+public static final String SECONDARY_STORE = "secondary";
+
 @Property(boolValue = false,
 label = "Custom BlobStore",
 description = "Boolean value indicating that a custom
BlobStore is to be used. " +
@@ -308,6 +316,11 @@ public class SegmentNodeStoreService ext
 return;
 }

+if (toBoolean(property(SECONDARY_STORE), false)){
+registerSecondaryStore();
+return;
+}
+
 if (registerSegmentNodeStore()) {
 Dictionary props = new Hashtable();
 props.put(Constants.SERVICE_PID,
SegmentNodeStore.class.getName());
@@ -317,6 +330,20 @@ public class SegmentNodeStoreService ext
 }
 }

+private void registerSecondaryStore() {
+segmentNodeStore = SegmentNodeStoreBuilders.builder(store).build();
+Dictionary props = new Hashtable();
+props.put(NodeStoreProvider.ROLE, "secondary");
+storeRegistration =
context.getBundleContext().registerService(NodeStoreProvider.class.getName(),
new NodeStoreProvider() {
+@Override
+public NodeStore getNodeStore() {
+return SegmentNodeStoreService.this;
+}
+},
+props);
+log.info("Registered NodeStoreProvider backed by
SegmentNodeStore");