Module to host oak-run console scripts

[Chetan Mehrotra]

Hi Team,

We have few oak-run console groovy scripts used often to
analyze/troubleshoot customer setups. Currently these are scattered
across various github gist.

I propose that we

1. Create a new module oak-run-scripts (or oak-console-scripts) and
host such generic scripts there. 2. This module would not be part of
build
3. Scripts present here would be generic i.e. not tied to any product

Thoughts?

Chetan Mehrotra

Re: possible to graft a remote non-JCR source to a path?

[Kaiser Shahid]

hi david, long-time AEM developer here. i wouldn't recommend grafting at 
oak level even if it was possible. the resource API of sling is loose 
and amenable to all sorts of datatypes. and since your main point of 
interaction is through sling/AEM, that would be the easiest to build out 
and maintain.


-k

On 4/12/17 8:31 AM, Beaudet, David wrote:

Thanks Roy, yes that's the other suggestion Adobe had identified and we're 
looking at it as well.  In your opinion is integration of DAM Assets at the JCR 
level with Oak just a bad idea even if it is possible?

From: Roy Teeuwen [mailto:r...@teeuwen.be]
Sent: Wednesday, April 12, 2017 11:22 AM
To: oak-dev@jackrabbit.apache.org
Subject: Re: possible to graft a remote non-JCR source to a path?

Hey David,

Seeing as you are using AEM, thus Sling, have you looked at the 
ResourceProvider interface? With this you can bind resources coming from 
another place than the JCR repo to the content tree.
You can look at these examples, the first one fetches content from a couchbase, 
the second one from a mongodb:

  - 
https://github.com/apache/sling/tree/trunk/contrib/nosql/couchbase-resourceprovider
  - 
https://github.com/apache/sling/tree/trunk/contrib/nosql/mongodb-resourceprovider
  - 
https://github.com/Adobe-Consulting-Services/acs-aem-samples/blob/master/bundle/src/main/java/com/adobe/acs/samples/resourceproviders/impl/SampleResourceProvider.java

Greets,
Roy
On 12 Apr 2017, at 16:34, Beaudet, David 
> wrote:


Greetings,

We are researching the best way to expose digital assets that are managed in a 
non-JCR based enterprise DAM system via Adobe's native DAM UI and are wondering 
if it's possible to, for example, bind a path like /content/dam/edam/... to a 
remote data source.  Adobe had released an alpha or beta version of a package 
for JackRabbit that enabled a remote RDBMS source dataset to be grafted into 
the JCR tree at a configured path. Content was mapped from the database and 
exposed via the JCR APIs.  Is it still possible to implement such a feature 
with Oak and if so, can someone point me to the interfaces that would have to 
be implemented or a how-to guide if one exists?

Thanks,

Dave Beaudet
National Gallery of Art

Re: [m12] Effort to Improve Modularisation of Oak

[Angela Schreiber]

Hi Michael

I try to describe the changes proposed by the PoC in
https://issues.apache.org/jira/browse/OAK-6073?focusedCommentId=15965623
ge=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment
-15965623.
Additionally added some step-by-step instruction on how we proceeded.

What is still missing is additional extra summary of impact it has on
'oak-core' and the cleanup of the 'oak.util' package. will add that
tomorrow.

More comments and answers to your questions inline:

On 12/04/17 13:56, "Michael Dürig"  wrote:
>
>Thanks for driving this and coming up with a PoC. I like the direction
>this is taking. The module boundaries make sense to me and having
>certain dependencies de-tangled will certainly be helpful going forward.
>Could you share a bit of your experience doing this refactoring?

First of all it wasn't just me but a group of oak developers discussing
and working on this :-)
And I am sure they have more things to add here, looking at the whole
exercise from a different angle.

When we first looked at it on Tuesday last week, I thought that we would
end the exercise with a "tried hard but failed" summary. So, I am quite
pleased that actually ended up with a working PoC.

>What were the main difficulties?

Up to now we were looking for the hanging fruits. I guess the main
difficulties lay ahead of us with plugins.tree and some of the query/index
classes where exported stuff refers to private classes.
> 

Looking back I thing the biggest issues are

- putting everything in oak-core was obviously convenient but it turned
out to be impossible to protect against boundary violations
- packages sometimes contain classes that not really belong together, e.g.
  - org.apache.jackrabbit.oak.spi.lifecycle containing OakInitializer
  - oak.apache.jackrabbit.oak.spi.whiteboard containing classes that
should be located with the corresponding feature (e.g. user mgt, index)
- impl specific methods that are not defined by an API contract such as
e.g. ValueImpl.getBlob, ValueImpl.getOakString... this was actually the
only place where we added a new interface and modified existing code
- somehow i got the impression that we didn't manage to make consistent
decision wrt package naming
  - what should go into a 'spi' package?
  - what should go into 'plugins'-something? and how is that different
from spi? (and what is e.g. the diff between spi.blob and plugins.blob?)
  - when do we create a new package space oak.somethingnew and how are
those packages intended to be used.

Moving forward I think it would help a lot if we had a common
understanding here and come up with some description what is used for what.
Maybe we also need to take a closer look when adding new stuff to oak-core
while moving forward.

>Quick wins? 

Well... for me the biggest win is the fact that 'oak-blob-azure',
'oak-blob-cloud' and 'oak-segment-tar' no longer would depend on oak-core.
Working on modularisation also is quite helpful to spot
'dumpster'-packages like e.g. oak.util
Also, working on the modularisation helped to identify unused imports, the
kind of things that come in naturally when trying to clean up.

>Is there anything that could be
>controversial?

We had a discussion in the group on whether it really makes sense to have
'oak-blob' (existing) and 'oak-blob-plugins' (the blob.plugins part of
oak-core)... I don't have a strong feeling here and would leave that
primarily to those working on the blob handling... at the end we kept it
separate to see the effective dependencies to and from those 2 modules
first and then decide on whether we should merge it. after all merging
should be fairly easy compared to the split.

>
>Looking at the list of modules, its size and the names, did you consider
>switching to a hierarchical module structure?

No, we didn't discuss that.

>Or could this make sense later on?

I don't have any strong preference here. We had some discussion on how we
should align the svn structure in general and what would be the best when
we want to start releasing modules individually.

>Otherwise can we come up with a naming scheme that implies
>grouping (e.g. node store implementations, blob store implementations,
>etc.)

Sure, makes a lot of sense to me. :-)

>
>Re. oak-base and oak-commons, these are probably separated to avoid
>circular dependencies. Is there anyway to otherwise clarify the
>difference between the two? I.e. if I implement a new class, which
>module it should go into? Would oak-base be something like oak-core-spi
>or even oak-spi? This would nicely dual the oak-store-spi module.

Exactly... and actually I like the name 'oak-core-spi' a lot better. In
OAK-6073 I stated that IMO that module might be a tmp solution as it
currently contains a somewhat loose collection of packages that were in
'oak-core' and didn't really fit into 'oak-commons' from my point of view.
After all I wanted to avoid converting 'oak-commons' into a second
'oak-core' :-). 
That module is 

RE: possible to graft a remote non-JCR source to a path?

[Beaudet, David]

Thanks Roy, yes that's the other suggestion Adobe had identified and we're 
looking at it as well.  In your opinion is integration of DAM Assets at the JCR 
level with Oak just a bad idea even if it is possible?

From: Roy Teeuwen [mailto:r...@teeuwen.be]
Sent: Wednesday, April 12, 2017 11:22 AM
To: oak-dev@jackrabbit.apache.org
Subject: Re: possible to graft a remote non-JCR source to a path?

Hey David,

Seeing as you are using AEM, thus Sling, have you looked at the 
ResourceProvider interface? With this you can bind resources coming from 
another place than the JCR repo to the content tree.
You can look at these examples, the first one fetches content from a couchbase, 
the second one from a mongodb:

 - 
https://github.com/apache/sling/tree/trunk/contrib/nosql/couchbase-resourceprovider
 - 
https://github.com/apache/sling/tree/trunk/contrib/nosql/mongodb-resourceprovider
 - 
https://github.com/Adobe-Consulting-Services/acs-aem-samples/blob/master/bundle/src/main/java/com/adobe/acs/samples/resourceproviders/impl/SampleResourceProvider.java

Greets,
Roy
On 12 Apr 2017, at 16:34, Beaudet, David 
> wrote:


Greetings,

We are researching the best way to expose digital assets that are managed in a 
non-JCR based enterprise DAM system via Adobe's native DAM UI and are wondering 
if it's possible to, for example, bind a path like /content/dam/edam/... to a 
remote data source.  Adobe had released an alpha or beta version of a package 
for JackRabbit that enabled a remote RDBMS source dataset to be grafted into 
the JCR tree at a configured path. Content was mapped from the database and 
exposed via the JCR APIs.  Is it still possible to implement such a feature 
with Oak and if so, can someone point me to the interfaces that would have to 
be implemented or a how-to guide if one exists?

Thanks,

Dave Beaudet
National Gallery of Art

Re: possible to graft a remote non-JCR source to a path?

[Roy Teeuwen]

Hey David,

Seeing as you are using AEM, thus Sling, have you looked at the 
ResourceProvider interface? With this you can bind resources coming from 
another place than the JCR repo to the content tree.
You can look at these examples, the first one fetches content from a couchbase, 
the second one from a mongodb:

 - 
https://github.com/apache/sling/tree/trunk/contrib/nosql/couchbase-resourceprovider
 

 - 
https://github.com/apache/sling/tree/trunk/contrib/nosql/mongodb-resourceprovider
 

 - 
https://github.com/Adobe-Consulting-Services/acs-aem-samples/blob/master/bundle/src/main/java/com/adobe/acs/samples/resourceproviders/impl/SampleResourceProvider.java
 


Greets,
Roy
> On 12 Apr 2017, at 16:34, Beaudet, David  wrote:
> 
> 
> Greetings,
> 
> We are researching the best way to expose digital assets that are managed in 
> a non-JCR based enterprise DAM system via Adobe's native DAM UI and are 
> wondering if it's possible to, for example, bind a path like 
> /content/dam/edam/... to a remote data source.  Adobe had released an alpha 
> or beta version of a package for JackRabbit that enabled a remote RDBMS 
> source dataset to be grafted into the JCR tree at a configured path. Content 
> was mapped from the database and exposed via the JCR APIs.  Is it still 
> possible to implement such a feature with Oak and if so, can someone point me 
> to the interfaces that would have to be implemented or a how-to guide if one 
> exists?
> 
> Thanks,
> 
> Dave Beaudet
> National Gallery of Art



signature.asc
Description: Message signed with OpenPGP

possible to graft a remote non-JCR source to a path?

[Beaudet, David]

Greetings,

We are researching the best way to expose digital assets that are managed in a 
non-JCR based enterprise DAM system via Adobe's native DAM UI and are wondering 
if it's possible to, for example, bind a path like /content/dam/edam/... to a 
remote data source.  Adobe had released an alpha or beta version of a package 
for JackRabbit that enabled a remote RDBMS source dataset to be grafted into 
the JCR tree at a configured path. Content was mapped from the database and 
exposed via the JCR APIs.  Is it still possible to implement such a feature 
with Oak and if so, can someone point me to the interfaces that would have to 
be implemented or a how-to guide if one exists?

Thanks,

Dave Beaudet
National Gallery of Art

Re: Backport of OAK-4933 to 1.6

[Michael Dürig]

On 12.04.17 09:17, Marcel Reutegger wrote:

Hi,

On 28/03/17 09:09, Michael Dürig wrote:

As Marcel mentions on the issue a better approach would be to release
this independently. If this is blocked by dependencies we should make an
effort to sort this out, as now is the time in the release cycle for
doing so.

So for now -1 from my side to back porting this until

a) we have a clear picture of the alternatives


A group of Oak committers discussed alternatives offline last week. I'd
like to summarize this discussion here on the dev list again. Feel free
to correct me or add to below if you think I missed important conclusions.

In general we recognized the need for this module to work with 1.6.
After all the whole modularization effort is about enabling independent
releases and make it possible to release new features outside of our so
far yearly minor version Oak release cycle. Having to wait for nearly a
year in order to use a rather independent new module with a stable Oak
release seems disproportionate.

An alternative was presented in OAK-4933. Release the new module
independently and then use it in 1.6. The broad agreement in the
discussion last week was, this is not desirable either. We'd end up in a
situation where the 1.6 branch is a mix of multi-module and independent
release modules. The modularization in progress for trunk, with the goal
to have cleaner dependencies also for the new azure module, will make it
more difficult to use the new module in
the 1.6 branch. We definitively don't want to restructure the 1.6 branch
as well.

The proposal therefore is to backport OAK-4933 despite the concerns
mentioned earlier.


Makes sense, thanks for the in-depth analysis! Given there is good 
progress with the modularisation [1] and that I'm confident that we'll 
keep up with this I'm taking back my -1 from earlier. Consider me +1.




b) in the case of backporting, understand how we would ensure quality.
This is new code that was so far never exposed to the level of testing
people would expect from 1.6 code.


I don't have a good answer to this other than thorough code review and
good test coverage. Anything else we can do?


Can we in addition put some sort of a disclaimer into the 1.6 branch for 
this module stating its origin? This would help setting user's 
expectations right.


Michael

[1] 
https://lists.apache.org/thread.html/73f6916cc920e36c116ba0311bdf602021598c31f1cf0e5e840e379c@%3Coak-dev.jackrabbit.apache.org%3E

Re: [m12] Effort to Improve Modularisation of Oak

[Michael Dürig]

Hi Angela,

Thanks for driving this and coming up with a PoC. I like the direction 
this is taking. The module boundaries make sense to me and having 
certain dependencies de-tangled will certainly be helpful going forward.
Could you share a bit of your experience doing this refactoring? What 
were the main difficulties? Quick wins? Is there anything that could be 
controversial?


Looking at the list of modules, its size and the names, did you consider 
switching to a hierarchical module structure? Or could this make sense 
later on? Otherwise can we come up with a naming scheme that implies 
grouping (e.g. node store implementations, blob store implementations, 
etc.)


Re. oak-base and oak-commons, these are probably separated to avoid 
circular dependencies. Is there anyway to otherwise clarify the 
difference between the two? I.e. if I implement a new class, which 
module it should go into? Would oak-base be something like oak-core-spi 
or even oak-spi? This would nicely dual the oak-store-spi module.


Is there plans to move document/rdb stores to separate modules or is 
this beyond the current scope?


Michael

On 12.04.17 11:21, Angela Schreiber wrote:

Hi

As mentioned my Marcel this morning [0] we had some offline discussions
related to the oak-blob-azure module and how we could independently
release it. While we didn't see a satisfying solution for the 1.6 branch,
we concluded that we should pick up the modularisation discussion for
address this in the near future.

Consequently a group of oak devs started to work on a PoC on how to
improve modularisation of Oak (in particular oak-core). As we managed to
get rid of the dependency of oak-blob-azure (and oak-segment-tar for that
matter) from oak-core with a reasonable effort, we would like move forward
with this in oak-trunk.

For that matter I created a new epic "Modularisation of Oak" (OAK-6069 at
[1]) and added/linked a initial bunch of issues spotted during the
workshop and earlier. For the 'oak-blob-azure' topic I create a dedicated
task OAK-6073 [2], where I will also add some detailed summary of the
initial effort. The latter can also be looked at on a github fork at [3].

Kind regards
Angela

[0]
http://markmail.org/message/neoiyv5qsffo424e?q=azure+list:org%2Eapache%2Eja
ckrabbit%2Eoak-dev+from:%22Marcel+Reutegger%22=1
[1] https://issues.apache.org/jira/browse/OAK-6069
[2] https://issues.apache.org/jira/browse/OAK-6073
[3] https://github.com/mreutegg/jackrabbit-oak/tree/m12n.

Re: [VOTE] Release Apache Jackrabbit Oak 1.2.25

[Alex Parvulescu]

[X] +1 Release this package as Apache Jackrabbit Oak 1.2.25

alex

On Tue, Apr 11, 2017 at 7:51 AM, Amit Jain  wrote:

> A candidate for the Jackrabbit Oak 1.2.25 release is available at:
>
> https://dist.apache.org/repos/dist/dev/jackrabbit/oak/1.2.25/
>
> The release candidate is a zip archive of the sources in:
>
>
> https://svn.apache.org/repos/asf/jackrabbit/oak/tags/
> jackrabbit-oak-1.2.25/
>
> The SHA1 checksum of the archive is
> 46aebfe648d7e0bd600054c54bced2270c09d641.
>
> A staged Maven repository is available for review at:
>
> https://repository.apache.org/
>
> The command for running automated checks against this release candidate is:
>
> $ sh check-release.sh oak 1.2.25
> 46aebfe648d7e0bd600054c54bced2270c09d641
>
> Please vote on releasing this package as Apache Jackrabbit Oak 1.2.25.
> The vote is open for the next 72 hours and passes if a majority of at
> least three +1 Jackrabbit PMC votes are cast.
>
> [ ] +1 Release this package as Apache Jackrabbit Oak 1.2.25
> [ ] -1 Do not release this package because...
>
> Thanks
> Amit
>

[m12] Effort to Improve Modularisation of Oak

[Angela Schreiber]

Hi 

As mentioned my Marcel this morning [0] we had some offline discussions
related to the oak-blob-azure module and how we could independently
release it. While we didn't see a satisfying solution for the 1.6 branch,
we concluded that we should pick up the modularisation discussion for
address this in the near future.

Consequently a group of oak devs started to work on a PoC on how to
improve modularisation of Oak (in particular oak-core). As we managed to
get rid of the dependency of oak-blob-azure (and oak-segment-tar for that
matter) from oak-core with a reasonable effort, we would like move forward
with this in oak-trunk.

For that matter I created a new epic "Modularisation of Oak" (OAK-6069 at
[1]) and added/linked a initial bunch of issues spotted during the
workshop and earlier. For the 'oak-blob-azure' topic I create a dedicated
task OAK-6073 [2], where I will also add some detailed summary of the
initial effort. The latter can also be looked at on a github fork at [3].

Kind regards
Angela

[0] 
http://markmail.org/message/neoiyv5qsffo424e?q=azure+list:org%2Eapache%2Eja
ckrabbit%2Eoak-dev+from:%22Marcel+Reutegger%22=1
[1] https://issues.apache.org/jira/browse/OAK-6069
[2] https://issues.apache.org/jira/browse/OAK-6073
[3] https://github.com/mreutegg/jackrabbit-oak/tree/m12n.

Re: Backport of OAK-4933 to 1.6

[Angela Schreiber]

Hi Marcel

Thanks for the summary. From my understanding the proposal additionally
included the wish that we move forward with the modularisation and aims
for independent releases for the oak-blob-azure module in trunk.

See OAK-6069 [0] and subtask OAK-6073 [1] for the corresponding JIRA issue
including references to an initial PoC.

Kind regards
Angela

[1] https://issues.apache.org/jira/browse/OAK-6069

[1] https://issues.apache.org/jira/browse/OAK-6073

On 12/04/17 09:17, "Marcel Reutegger"  wrote:

>Hi,
>
>On 28/03/17 09:09, Michael Dürig wrote:
>> As Marcel mentions on the issue a better approach would be to release
>> this independently. If this is blocked by dependencies we should make an
>> effort to sort this out, as now is the time in the release cycle for
>> doing so.
>>
>> So for now -1 from my side to back porting this until
>>
>> a) we have a clear picture of the alternatives
>
>A group of Oak committers discussed alternatives offline last week. I'd
>like to summarize this discussion here on the dev list again. Feel free
>to correct me or add to below if you think I missed important conclusions.
>
>In general we recognized the need for this module to work with 1.6.
>After all the whole modularization effort is about enabling independent
>releases and make it possible to release new features outside of our so
>far yearly minor version Oak release cycle. Having to wait for nearly a
>year in order to use a rather independent new module with a stable Oak
>release seems disproportionate.
>
>An alternative was presented in OAK-4933. Release the new module
>independently and then use it in 1.6. The broad agreement in the
>discussion last week was, this is not desirable either. We'd end up in a
>situation where the 1.6 branch is a mix of multi-module and independent
>release modules. The modularization in progress for trunk, with the goal
>to have cleaner dependencies also for the new azure module, will make it
>more difficult to use the new module in
>the 1.6 branch. We definitively don't want to restructure the 1.6 branch
>as well.
>
>The proposal therefore is to backport OAK-4933 despite the concerns
>mentioned earlier.
>
>> b) in the case of backporting, understand how we would ensure quality.
>> This is new code that was so far never exposed to the level of testing
>> people would expect from 1.6 code.
>
>I don't have a good answer to this other than thorough code review and
>good test coverage. Anything else we can do?
>
>Regards
>  Marcel

Re: Backport of OAK-4933 to 1.6

[Marcel Reutegger]

Hi,

On 28/03/17 09:09, Michael Dürig wrote:

As Marcel mentions on the issue a better approach would be to release
this independently. If this is blocked by dependencies we should make an
effort to sort this out, as now is the time in the release cycle for
doing so.

So for now -1 from my side to back porting this until

a) we have a clear picture of the alternatives


A group of Oak committers discussed alternatives offline last week. I'd 
like to summarize this discussion here on the dev list again. Feel free 
to correct me or add to below if you think I missed important conclusions.


In general we recognized the need for this module to work with 1.6. 
After all the whole modularization effort is about enabling independent 
releases and make it possible to release new features outside of our so 
far yearly minor version Oak release cycle. Having to wait for nearly a 
year in order to use a rather independent new module with a stable Oak 
release seems disproportionate.


An alternative was presented in OAK-4933. Release the new module 
independently and then use it in 1.6. The broad agreement in the 
discussion last week was, this is not desirable either. We'd end up in a 
situation where the 1.6 branch is a mix of multi-module and independent 
release modules. The modularization in progress for trunk, with the goal 
to have cleaner dependencies also for the new azure module, will make it 
more difficult to use the new module in
the 1.6 branch. We definitively don't want to restructure the 1.6 branch 
as well.


The proposal therefore is to backport OAK-4933 despite the concerns 
mentioned earlier.



b) in the case of backporting, understand how we would ensure quality.
This is new code that was so far never exposed to the level of testing
people would expect from 1.6 code.


I don't have a good answer to this other than thorough code review and 
good test coverage. Anything else we can do?


Regards
 Marcel