Re: A federated data store

[Matt Ryan]

I misremembered above when I was describing a possible implementation.  I
was thinking we'd add a method to the delegate, but that would be added to
the DataStore interface, obviously (not BlobStore or
GarbageCollectibleBlobStore).  Likewise, the default implementation would
exist in AbstractDataStore (not AbstractBlobStore).

Sorry about the mix-up.

On Thu, Apr 20, 2017 at 12:20 PM, Matt Ryan  wrote:

> Hi,
>
> I'm looking at the possibility of creating a new kind of data store, let's
> call it a federated data store, and wanted to see what everyone thinks
> about this.
>
> The basic idea is that the federated data store would allow for more than
> one data store to be configured for an Oak instance.  Oak would then be
> able to choose which data store to use based on a number of criteria, like
> file size, JCR path, node type, existence of a node property, a node
> property value, or other items, or a combination of items.  In my thinking
> these are defined in configuration so the federated data store would know
> how to select which data store is used to store which binary.
>
> I think this is a step towards UC14 - Hierarchical BlobStore in [0].  Once
> the federated data store was implemented we should be able to support UC14
> with little work.  I can also foresee other possible capabilities it could
> offer, such as storing blobs for different node types in different data
> stores, or choosing from a few different data stores based on geographic
> location (UC2 in [0]).
>
> In my mind we could add capability to DataStoreBlobStore.writeStream()
> where the decision is made whether to write a stream to the data store
> delegate or put it in-memory.  Instead we could defer the decision directly
> to the delegate, adding a method to the appropriate interface (BlobStore or
> GarbageCollectibleBlobStore) to handle this decision, and default the
> decision in AbstractBlobStore to be based on the record size (which is the
> current behavior, except currently that decision is made in
> DataStoreBlobStore IIUC).  All other existing data stores should then
> behave the same.  But in the case of the federated data store this decision
> would be more involved, selecting the right data store based on
> configuration.
>
> The federated data store would need to exist independent of other data
> stores, so figuring out how to create those data stores without having a
> code dependency would be a challenge to figure out.
>
>
> Please let me know what you think, is my idea about the implementation
> flawed, is there a better way to accomplish this, what concerns are there
> about it, etc.  I'd like to brainstorm with the list something that can
> work in this area and then I'll create a ticket for it.  Or I can create
> the ticket, and we can have the discussion in the ticket.  Let me know
> which is best.
>
>
> [0] - https://wiki.apache.org/jackrabbit/JCR%20Binary%20Usecase
>
>
> - Matt Ryan
>

A federated data store

[Matt Ryan]

Hi,

I'm looking at the possibility of creating a new kind of data store, let's
call it a federated data store, and wanted to see what everyone thinks
about this.

The basic idea is that the federated data store would allow for more than
one data store to be configured for an Oak instance.  Oak would then be
able to choose which data store to use based on a number of criteria, like
file size, JCR path, node type, existence of a node property, a node
property value, or other items, or a combination of items.  In my thinking
these are defined in configuration so the federated data store would know
how to select which data store is used to store which binary.

I think this is a step towards UC14 - Hierarchical BlobStore in [0].  Once
the federated data store was implemented we should be able to support UC14
with little work.  I can also foresee other possible capabilities it could
offer, such as storing blobs for different node types in different data
stores, or choosing from a few different data stores based on geographic
location (UC2 in [0]).

In my mind we could add capability to DataStoreBlobStore.writeStream()
where the decision is made whether to write a stream to the data store
delegate or put it in-memory.  Instead we could defer the decision directly
to the delegate, adding a method to the appropriate interface (BlobStore or
GarbageCollectibleBlobStore) to handle this decision, and default the
decision in AbstractBlobStore to be based on the record size (which is the
current behavior, except currently that decision is made in
DataStoreBlobStore IIUC).  All other existing data stores should then
behave the same.  But in the case of the federated data store this decision
would be more involved, selecting the right data store based on
configuration.

The federated data store would need to exist independent of other data
stores, so figuring out how to create those data stores without having a
code dependency would be a challenge to figure out.


Please let me know what you think, is my idea about the implementation
flawed, is there a better way to accomplish this, what concerns are there
about it, etc.  I'd like to brainstorm with the list something that can
work in this area and then I'll create a ticket for it.  Or I can create
the ticket, and we can have the discussion in the ticket.  Let me know
which is best.


[0] - https://wiki.apache.org/jackrabbit/JCR%20Binary%20Usecase


- Matt Ryan

BUILD FAILURE: Jackrabbit Oak - Build # 208 - Failure

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #208)

Status: Failure

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/208/ to 
view the results.

Changes:
[angela] OAK-6092 Find a new home for ApproximateCounter

[angela] OAK-6114 : Move InitialContent to package org.apache.jackrabbit.oak

[mreutegg] OAK-6109: Revision GC command in oak-run

 

Test results:
All tests passed

OAK-6089 (maven plugin updates) and OAK-5664 (require Java 8), also: move to Maven 3.3.9?

[Julian Reschke]

Hi there,

in preparation of the switch to Java 8, I have updated lots of maven 
plugin dependencies and also explicitly required the use of Maven 3.2.1 
(this is the version we use in CI, and was apparently was also de facto 
required already).


I ran integration tests after each of these changes, but can't promise 
that I didn't break something not covered by the tests.


There are a few recommended plugin updates still to be done:


[INFO]   org.apache.felix:maven-scr-plugin .. 1.21.0 -> 1.24.0
[INFO]   org.springframework.boot:spring-boot-maven-plugin  1.3.8.RELEASE -> 
1.5.2.RELEASE


...but these require more changes, and I'll open separate tickets for these.

With this done, we can switch to Java 8 (as required language level); 
I'll do that later today.


That said, while at it, we could go even further and require Maven 3.3.9 
- I've been building past branches of Jackrabbit and Oak with it, and 
haven't noticed any problems. Feedback on this would be appreciated.


Best regards, Julian

Re: [m12n] Location of InitialContent

[Robert Munteanu]

On Thu, 2017-04-20 at 08:07 +, Angela Schreiber wrote:
> hi
> 
> the original intention of the 'InitialContent' was just to registers
> built-in JCR node types, which explains it's location in
> org.apache.jackrabbit.oak.plugins.nodetype.write
> 
> in the mean time it has a evolved to container for all kind of
> initial
> content required for a JCR repository: mandatory structure, version
> storage, uuid-index and most recently document ns specific
> configuration
> (see also OAK-5656 ).
> 
> to me the location in the
> org.apache.jackrabbit.oak.plugins.nodetype.write
> package no longer makes sense and i would suggest to move it to the
> org.apache.jackrabbit.oak package along with the Oak, OakInitializer,
> OakVersion and other utilities used to create an JCR/Oak repository.
> 
> wdyt?

+1, would fit in well there.

Robert

Re: [m12n] A new home for ApproximateCounter, NodeUtil and TreeUtil?

[Robert Munteanu]

On Wed, 2017-04-19 at 15:53 +, Angela Schreiber wrote:
> hi robert
> 
> i created 2 separate tasks in the OAK-6069 m12n epic to track
> progress
> with ApproximateCounter and NodeUtil|TreeUtil respectively.
> moving the ApproximateCounter should be doable with limited effort,
> we
> just need to get a better understanding if that is/will/shouldbe used
> more
> widely than just for the indexing.

I've commented on OAK-6093 - I think we should start with getting the
naming right.

Robert

> 
> the other one probably needs some more effort.
> angela 
> 
> 
> On 19/04/17 17:45, "Robert Munteanu"  wrote:
> 
> > On Tue, 2017-04-18 at 06:21 +, Angela Schreiber wrote:
> > > Hi Robert
> > > 
> > > While NodeUtil and TreeUtil would naturally fit to plugins.tree,
> > > I am
> > > not
> > > convinced that ApproximateCounter really belongs there. Afaik it
> > > is
> > > only
> > > used for query index strategy and counting. I would rather move
> > > 'ApproximateCounter' to 'plugins.index'.
> > > 
> > > Regarding moving 'NodeUtil' and 'TreeUtil': IMHO we have here 2
> > > utility
> > > classes providing almost the same functionality. I would prefer
> > > to
> > > decide
> > > on the redundancy (and potentially clean it up) before moving it
> > > to a
> > > package that already has semantic versioning enabled (in contrast
> > > to
> > > the
> > > util package where they currently are located).
> > > 
> > > wdyt?
> > 
> > (Sorry, missed this somehow)
> > 
> > Yes, holding on for now sounds good to me.
> > 
> > Thanks,
> > 
> > Robert
> > 
> > > 
> > > Kind regards
> > > Angela
> > > 
> > > 
> > > On 14/04/17 12:47, "Robert Munteanu"  wrote:
> > > 
> > > > I created a final PR for this as I have somewhat mixed
> > > > feelings.
> > > > One
> > > > one had, it finally nukes the util package. On the other hand,
> > > > it
> > > > looks
> > > > like a lot of noise for 3 classes.
> > > > 
> > > >  
> > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > 2Fgi
> > > > thub.co
> > > > m%2Fmreutegg%2Fjackrabbit-
> > > > oak%2Fpull%2F6&data=02%7C01%7C%7C287ecd3d735246c
> > > > cbc8308d48323b62e%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C
> > > > 6362
> > > > 7763678
> > > > 7796080&sdata=HSDWauCHB%2Bb4OaX90CfEWsA7487EP3FvrSicZNgKD8Q%3D&
> > > > rese
> > > > rved=0
> > > > 
> > > > Robert
> > > > 
> > > > On Thu, 2017-04-06 at 14:49 +, Angela Schreiber wrote:
> > > > > Hi Robert
> > > > > 
> > > > > plugins.tree would feel natural to me.
> > > > > regarding the export: not sure about that either... the
> > > > > plugins.tree
> > > > > has
> > > > > some unfortunate dependencies e.g. to oak.core. so probably
> > > > > more
> > > > > work
> > > > > ahead in that area.
> > > > > 
> > > > > kind regards
> > > > > angela
> > > > > 
> > > > > On 06/04/17 16:41, "Robert Munteanu" 
> > > > > wrote:
> > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > Working in the m12n branch [1] I'm trying to get rid of the
> > > > > > o.a.j.oak.util package and the last surviving members are
> > > > > > ApproximateCounter, NodeUtil and TreeUtil.
> > > > > > 
> > > > > > As I see it these classes are essentially helpers built on
> > > > > > top
> > > > > > of
> > > > > > the
> > > > > > Tree and NodeState APIs. Those would make them candidates
> > > > > > on
> > > > > > for
> > > > > > either
> > > > > > oak-store-spi or (if we manage to trim down the
> > > > > > dependencies)
> > > > > > oak-
> > > > > > base.
> > > > > > 
> > > > > > However I am having trouble naming the package which will
> > > > > > hold
> > > > > > them.
> > > > > > They're not part of the spi, so I can't put them in
> > > > > > spi.state .
> > > > > > 
> > > > > > Maybe they belong in oak-core in plugins.tree, but I'm not
> > > > > > sure
> > > > > > if
> > > > > > we
> > > > > > want to keep that as a package which is exported outside
> > > > > > oak-
> > > > > > core.
> > > > > > 
> > > > > > Thoughts?
> > > > > > 
> > > > > > Robert
> > > > > > 
> > > > > > [1]: 
> > > > > > https://na01.safelinks.protection.outlook.com/?url=https%3A
> > > > > > %2F%
> > > > > > 2Fgi
> > > > > > thub.co
> > > > > > m%2Fmreutegg%2Fjackrabbit-
> > > > > > oak%2Ftree%2Fm12n&data=02%7C01%7C%7Cbfc1feb5ff4a
> > > > > > 4866c79c08d47cfafe6d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0
> > > > > > %7C0
> > > > > > %7C6
> > > > > > 3627086
> > > > > > 4841267177&sdata=CWwq4ifTZIU1gW9UEd2STRLm%2B1svSP0kvlkLMksm
> > > > > > WcM%
> > > > > > 3D&r
> > > > > > eserved
> > > > > > =0
> > > > > 
> > > > > 
> > > 
> > > 
> 
> 

BUILD FAILURE: Jackrabbit Oak - Build # 201 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #201)

Status: Still Failing

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/201/ to 
view the results.

Changes:
No changes
 

Test results:
No tests ran.

BUILD FAILURE: Jackrabbit Oak - Build # 200 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #200)

Status: Still Failing

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/200/ to 
view the results.

Changes:
No changes
 

Test results:
No tests ran.

BUILD FAILURE: Jackrabbit Oak - Build # 199 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #199)

Status: Still Failing

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/199/ to 
view the results.

Changes:
No changes
 

Test results:
No tests ran.

BUILD FAILURE: Jackrabbit Oak - Build # 198 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #198)

Status: Still Failing

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/198/ to 
view the results.

Changes:
[reschke] OAK-6098: update maven plugins

maven-assembly-plugin

 

Test results:
No tests ran.

BUILD FAILURE: Jackrabbit Oak - Build # 197 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #197)

Status: Still Failing

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/197/ to 
view the results.

Changes:
[angela] minor improvement to oak-commons-run: add private constructor to 
utilities and remove unused imports

 

Test results:
No tests ran.

Re: [m12n] Location of InitialContent

[Angela Schreiber]

Hi Julian

Yes... having it located where JCR comes into place would be optimal...
but not yet possible I guess. That's why I would like to move to a place
that contains similar utilities in a first place and then look at
everything together again in a second step.

Kind regards
Angela

On 20/04/17 10:58, "Julian Sedding"  wrote:

>Hi Angela
>
>From the features you describe it sounds like it should go into
>org.apache.jackrabbit.oak.jcr (or at least most of it). It looks like
>it is being used in lots of tests in oak-core, however, so this may
>just be wishful thinking...
>
>Regards
>Julian
>
>
>On Thu, Apr 20, 2017 at 10:07 AM, Angela Schreiber 
>wrote:
>> hi
>>
>> the original intention of the 'InitialContent' was just to registers
>> built-in JCR node types, which explains it's location in
>> org.apache.jackrabbit.oak.plugins.nodetype.write
>>
>> in the mean time it has a evolved to container for all kind of initial
>> content required for a JCR repository: mandatory structure, version
>> storage, uuid-index and most recently document ns specific configuration
>> (see also OAK-5656 ).
>>
>> to me the location in the
>>org.apache.jackrabbit.oak.plugins.nodetype.write
>> package no longer makes sense and i would suggest to move it to the
>> org.apache.jackrabbit.oak package along with the Oak, OakInitializer,
>> OakVersion and other utilities used to create an JCR/Oak repository.
>>
>> wdyt?
>>
>> kind regards
>> angela
>>
>>

BUILD FAILURE: Jackrabbit Oak - Build # 196 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #196)

Status: Still Failing

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/196/ to 
view the results.

Changes:
[mduerig] OAK-6106: Excessive memory usage by the caches segment references
Test case

 

Test results:
No tests ran.

BUILD FAILURE: Jackrabbit Oak - Build # 195 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #195)

Status: Still Failing

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/195/ to 
view the results.

Changes:
[alexparvulescu] OAK-3342 move benchmarks in oak-benchmark module
 - fixed benchmarks scripts

 

Test results:
No tests ran.

Re: [m12n] Location of InitialContent

[Julian Sedding]

Hi Angela

>From the features you describe it sounds like it should go into
org.apache.jackrabbit.oak.jcr (or at least most of it). It looks like
it is being used in lots of tests in oak-core, however, so this may
just be wishful thinking...

Regards
Julian


On Thu, Apr 20, 2017 at 10:07 AM, Angela Schreiber  wrote:
> hi
>
> the original intention of the 'InitialContent' was just to registers
> built-in JCR node types, which explains it's location in
> org.apache.jackrabbit.oak.plugins.nodetype.write
>
> in the mean time it has a evolved to container for all kind of initial
> content required for a JCR repository: mandatory structure, version
> storage, uuid-index and most recently document ns specific configuration
> (see also OAK-5656 ).
>
> to me the location in the org.apache.jackrabbit.oak.plugins.nodetype.write
> package no longer makes sense and i would suggest to move it to the
> org.apache.jackrabbit.oak package along with the Oak, OakInitializer,
> OakVersion and other utilities used to create an JCR/Oak repository.
>
> wdyt?
>
> kind regards
> angela
>
>

BUILD FAILURE: Jackrabbit Oak - Build # 194 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #194)

Status: Still Failing

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/194/ to 
view the results.

Changes:
[reschke] OAK-6098: update maven plugins

maven-surefire-plugin

 

Test results:
No tests ran.

[m12n] Location of InitialContent

[Angela Schreiber]

hi

the original intention of the 'InitialContent' was just to registers
built-in JCR node types, which explains it's location in
org.apache.jackrabbit.oak.plugins.nodetype.write

in the mean time it has a evolved to container for all kind of initial
content required for a JCR repository: mandatory structure, version
storage, uuid-index and most recently document ns specific configuration
(see also OAK-5656 ).

to me the location in the org.apache.jackrabbit.oak.plugins.nodetype.write
package no longer makes sense and i would suggest to move it to the
org.apache.jackrabbit.oak package along with the Oak, OakInitializer,
OakVersion and other utilities used to create an JCR/Oak repository.

wdyt?

kind regards
angela

BUILD FAILURE: Jackrabbit Oak - Build # 193 - Failure

[Apache Jenkins Server]

The Apache Jenkins build system has built Jackrabbit Oak (build #193)

Status: Failure

Check console output at https://builds.apache.org/job/Jackrabbit%20Oak/193/ to 
view the results.

Changes:
[reschke] OAK-6098: update maven plugins

apache-rat-plugin

 

Test results:
No tests ran.