Re: Possibility of making nt:resource unreferenceable

[Thomas Mueller]

Hi,
>
>Currently I am under the impression that we have no knowledge of what
>*might* break, with varying opinions on the matter. Maybe we should to
>find out what *does* break.

I don't think it's possible to easily find out. Customer code might expect
the current behavior, and might silently break (without exception, but
with wrong behavior).

>
>As a remedy for implementations that rely on the current referencable
>nature, we could provide tooling that automatically adds the
>"mix:referencable" mixin to existing nt:resource nodes and recommend
>adapting the code to add the mixin as well.

That might work, but in some cases it might also result in problems (if
the code expects this not to be the case).

Regards,
Thomas

Re: Possibility of making nt:resource unreferenceable

[Julian Sedding]

On Wed, Oct 12, 2016 at 11:24 AM, Bertrand Delacretaz
 wrote:
> On Wed, Oct 12, 2016 at 11:18 AM, Julian Sedding  wrote:
>> ...As a remedy for implementations that rely on the current referencable
>> nature, we could provide tooling that automatically adds the
>> "mix:referencable" mixin to existing nt:resource nodes...
>
> Good idea, I suppose this can be done with a commit hook in a non-intrusive 
> way?

For JR2 content being upgraded to Oak (or during an Oak to Oak
"sidegrade"), i.e. in the oak-upgrade module, it would be easy to add
this functionality via a commit hook.

For an existing Oak repository the same functionality could be
implemented on the JCR API and a full repo traversal, I suppose. If we
can get past the node-type validation. Alternatively we could come up
with an extension SPI/API that allows plugging in an implementation
for specific non-trivial node-type updates. This would even allow for
two alternative implementations: one that adds mix:referencable and
another that removes the jcr:uuid property - so JCR users could choose
which strategy they prefer.

Regards
Julian


>
> -Bertrand

Re: Possibility of making nt:resource unreferenceable

[Bertrand Delacretaz]

On Wed, Oct 12, 2016 at 11:18 AM, Julian Sedding  wrote:
> ...As a remedy for implementations that rely on the current referencable
> nature, we could provide tooling that automatically adds the
> "mix:referencable" mixin to existing nt:resource nodes...

Good idea, I suppose this can be done with a commit hook in a non-intrusive way?

-Bertrand

Re: Possibility of making nt:resource unreferenceable

[Julian Sedding]

I'm with Julian R. on this (as I understand him). We should change the
node-type nt:resource to match the JCR 2.0 spec and deal with the
consequences.

Currently I am under the impression that we have no knowledge of what
*might* break, with varying opinions on the matter. Maybe we should to
find out what *does* break.

As a remedy for implementations that rely on the current referencable
nature, we could provide tooling that automatically adds the
"mix:referencable" mixin to existing nt:resource nodes and recommend
adapting the code to add the mixin as well.

Regards
Julian


On Wed, Oct 12, 2016 at 11:04 AM, Carsten Ziegeler  wrote:
> The latest proposal was not about making nt:resource unreferenceable,
> but silently changing the resource type for a nt:resource child node of
> a nt:file node to Oak:Resource.
>
> I just found three other places in Sling where nt:file nodes are created
> by hand. So with any other mechanism we have to change a lot of places
> in Sling alone. Not to mention all downstream users.
>
> Carsten
>
> Thomas Mueller wrote
>> Hi,
>>
>> I agree with Julian, I think making nt:resource unreferenceable would
>> (hardcoding some "magic" in Oak) would lead to hard-to-find bugs and
>> problems.
>>
>>> So whatever solution we pick, there is a risk that existing code fails.
>>
>> Yes. But I think if we create a new nodetype, at least it would be easier
>> for users to understand the problem.
>>
>> Also, the "upgrade path" with a new nodetype is smoother. This can be done
>> incrementally, even thought it might mean more total work. But making
>> nt:resource unreferenceable would be a hard break, and I think risk of
>> bigger problems is higher.
>>
>> Regards,
>> Thomas
>>
>>
>>
>> On 07/10/16 12:05, "Julian Reschke"  wrote:
>>
>>> On 2016-10-07 10:56, Carsten Ziegeler wrote:
 Julian Reschke wrote
> On 2016-10-07 08:04, Carsten Ziegeler wrote:
>> ...
>> The easiest solution that comes to my mind is:
>>
>> Whenever a nt:resource child node of a nt:file node is created, it is
>> silently changed to oak:resource.
>>
>> Carsten
>> ...
>
> Observation: that might break code that actually wants a referenceable
> node: it would create the node, check for the presence of
> mix:referenceable, and then decide not to add it because it's already
> there.
>

 Well, there might be code that assumes that a file uploaded through
 webdav is using a resource child node that is referenceable.
 Or a file posted through the Sling POST servlet has this. Now, you could
 argue if that code did not create the file, it should check node types,
 but how likely is that if the code has history?

 So whatever solution we pick, there is a risk that existing code fails.
 ...
>>>
>>> That is true..
>>>
>>> However, my preference would be to only break code which is
>>> non-conforming right now. Code should not rely on nt:resource being
>>> referenceable (see
>>> >> ml#3.7.11.5%20nt:resource>).
>>>
>>> So my preference would be to make that change and see what breaks (and
>>> get that fixed).
>>>
 ...
>>>
>>>
>>> Best regards, Julian
>>
>>
>
>
>
>
> --
> Carsten Ziegeler
> Adobe Research Switzerland
> cziege...@apache.org
>

Re: Possibility of making nt:resource unreferenceable

[Carsten Ziegeler]

The latest proposal was not about making nt:resource unreferenceable,
but silently changing the resource type for a nt:resource child node of
a nt:file node to Oak:Resource.

I just found three other places in Sling where nt:file nodes are created
by hand. So with any other mechanism we have to change a lot of places
in Sling alone. Not to mention all downstream users.

Carsten

Thomas Mueller wrote
> Hi,
> 
> I agree with Julian, I think making nt:resource unreferenceable would
> (hardcoding some "magic" in Oak) would lead to hard-to-find bugs and
> problems.
> 
>> So whatever solution we pick, there is a risk that existing code fails.
> 
> Yes. But I think if we create a new nodetype, at least it would be easier
> for users to understand the problem.
> 
> Also, the "upgrade path" with a new nodetype is smoother. This can be done
> incrementally, even thought it might mean more total work. But making
> nt:resource unreferenceable would be a hard break, and I think risk of
> bigger problems is higher.
> 
> Regards,
> Thomas
> 
> 
> 
> On 07/10/16 12:05, "Julian Reschke"  wrote:
> 
>> On 2016-10-07 10:56, Carsten Ziegeler wrote:
>>> Julian Reschke wrote
 On 2016-10-07 08:04, Carsten Ziegeler wrote:
> ...
> The easiest solution that comes to my mind is:
>
> Whenever a nt:resource child node of a nt:file node is created, it is
> silently changed to oak:resource.
>
> Carsten
> ...

 Observation: that might break code that actually wants a referenceable
 node: it would create the node, check for the presence of
 mix:referenceable, and then decide not to add it because it's already
 there.

>>>
>>> Well, there might be code that assumes that a file uploaded through
>>> webdav is using a resource child node that is referenceable.
>>> Or a file posted through the Sling POST servlet has this. Now, you could
>>> argue if that code did not create the file, it should check node types,
>>> but how likely is that if the code has history?
>>>
>>> So whatever solution we pick, there is a risk that existing code fails.
>>> ...
>>
>> That is true..
>>
>> However, my preference would be to only break code which is
>> non-conforming right now. Code should not rely on nt:resource being
>> referenceable (see
>> > ml#3.7.11.5%20nt:resource>).
>>
>> So my preference would be to make that change and see what breaks (and
>> get that fixed).
>>
>>> ...
>>
>>
>> Best regards, Julian
> 
> 


 

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: Possibility of making nt:resource unreferenceable

[Thomas Mueller]

Hi,

I agree with Julian, I think making nt:resource unreferenceable would
(hardcoding some "magic" in Oak) would lead to hard-to-find bugs and
problems.

> So whatever solution we pick, there is a risk that existing code fails.

Yes. But I think if we create a new nodetype, at least it would be easier
for users to understand the problem.

Also, the "upgrade path" with a new nodetype is smoother. This can be done
incrementally, even thought it might mean more total work. But making
nt:resource unreferenceable would be a hard break, and I think risk of
bigger problems is higher.

Regards,
Thomas



On 07/10/16 12:05, "Julian Reschke"  wrote:

>On 2016-10-07 10:56, Carsten Ziegeler wrote:
>> Julian Reschke wrote
>>> On 2016-10-07 08:04, Carsten Ziegeler wrote:
 ...
 The easiest solution that comes to my mind is:

 Whenever a nt:resource child node of a nt:file node is created, it is
 silently changed to oak:resource.

 Carsten
 ...
>>>
>>> Observation: that might break code that actually wants a referenceable
>>> node: it would create the node, check for the presence of
>>> mix:referenceable, and then decide not to add it because it's already
>>> there.
>>>
>>
>> Well, there might be code that assumes that a file uploaded through
>> webdav is using a resource child node that is referenceable.
>> Or a file posted through the Sling POST servlet has this. Now, you could
>> argue if that code did not create the file, it should check node types,
>> but how likely is that if the code has history?
>>
>> So whatever solution we pick, there is a risk that existing code fails.
>> ...
>
>That is true..
>
>However, my preference would be to only break code which is
>non-conforming right now. Code should not rely on nt:resource being
>referenceable (see
>ml#3.7.11.5%20nt:resource>).
>
>So my preference would be to make that change and see what breaks (and
>get that fixed).
>
> > ...
>
>
>Best regards, Julian

Re: Possibility of making nt:resource unreferenceable

[Julian Reschke]

On 2016-10-07 10:56, Carsten Ziegeler wrote:

Julian Reschke wrote

On 2016-10-07 08:04, Carsten Ziegeler wrote:

...
The easiest solution that comes to my mind is:

Whenever a nt:resource child node of a nt:file node is created, it is
silently changed to oak:resource.

Carsten
...


Observation: that might break code that actually wants a referenceable
node: it would create the node, check for the presence of
mix:referenceable, and then decide not to add it because it's already
there.



Well, there might be code that assumes that a file uploaded through
webdav is using a resource child node that is referenceable.
Or a file posted through the Sling POST servlet has this. Now, you could
argue if that code did not create the file, it should check node types,
but how likely is that if the code has history?

So whatever solution we pick, there is a risk that existing code fails.
...


That is true..

However, my preference would be to only break code which is 
non-conforming right now. Code should not rely on nt:resource being 
referenceable (see 
).


So my preference would be to make that change and see what breaks (and 
get that fixed).


> ...


Best regards, Julian

Re: Possibility of making nt:resource unreferenceable

[Carsten Ziegeler]

Julian Reschke wrote
> On 2016-10-07 08:04, Carsten Ziegeler wrote:
>> ...
>> The easiest solution that comes to my mind is:
>>
>> Whenever a nt:resource child node of a nt:file node is created, it is
>> silently changed to oak:resource.
>>
>> Carsten
>> ...
> 
> Observation: that might break code that actually wants a referenceable
> node: it would create the node, check for the presence of
> mix:referenceable, and then decide not to add it because it's already
> there.
> 

Well, there might be code that assumes that a file uploaded through
webdav is using a resource child node that is referenceable.
Or a file posted through the Sling POST servlet has this. Now, you could
argue if that code did not create the file, it should check node types,
but how likely is that if the code has history?

So whatever solution we pick, there is a risk that existing code fails.

Having a single place / configuration where the magic is done, helps in
disabling the magic if required. Spreading it across modules is calling
for more trouble.

And I'm still wondering how likely it is that some code out there
assumes that the resource child node of a file is referenceable and is
actually using this.

Carsten

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: Possibility of making nt:resource unreferenceable

[Julian Reschke]

On 2016-10-07 08:04, Carsten Ziegeler wrote:

...
The easiest solution that comes to my mind is:

Whenever a nt:resource child node of a nt:file node is created, it is
silently changed to oak:resource.

Carsten
...


Observation: that might break code that actually wants a referenceable 
node: it would create the node, check for the presence of 
mix:referenceable, and then decide not to add it because it's already there.


Best regards, Julian

Re: Possibility of making nt:resource unreferenceable

[Chetan Mehrotra]

On Fri, Oct 7, 2016 at 11:34 AM, Carsten Ziegeler  wrote:
> Whenever a nt:resource child node of a nt:file node is created, it is
> silently changed to oak:resource.

I like this!

This can be done via an Editor which does this transformation upon
addition of new node. Something which can be easily enabled/disabled
if need arises. With this we would not have make change in many places
like JcrUtil.putFile, WebDav, Vault, Sling Post Servlet, any custom
code creating nt:file say using JcrUtil.putFile.

Chetan Mehrotra

Re: Possibility of making nt:resource unreferenceable

[Carsten Ziegeler]

What about a totally different approach. My main problems with some of
the suggestions are:

a) we have to fix a lot of places (post servlet, webdav, other modules
creating files), creating duplicate code. If this is made configurable,
then its even more of a nightmare.
b) oak:resource is not a node type defined in the spec

I understand that changing nt:resource might be a problem - although
this might be more a theoretical one.

Obviously the preferred solution is to do it in a single place, which
leads to changing it in Oak or having some way to do it via Oak -
instead of changing several places above Oak in the application layer.

The easiest solution that comes to my mind is:

Whenever a nt:resource child node of a nt:file node is created, it is
silently changed to oak:resource.

Carsten

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziege...@apache.org

Re: Possibility of making nt:resource unreferenceable

[Julian Reschke]

On 2016-10-05 11:20, Marcel Reutegger wrote:

Hi,

On 04/10/16 16:02, Julian Reschke wrote:

At least in our DocumentStoreImplementations (Mongo and RDB), making the
UUID something indexed by the storage (so either Mongo or the relational
database) should be relatively cheap (after all, the UUID never changes
once assigned, right?).


The jcr:uuid of a node mapped into a document *can* change. Consider the
case where a referenceable node with a given path is created, then
removed and re-created again with a different UUID.


Indeed. Thanks for the explanation.

Best regards, Julian

Re: Possibility of making nt:resource unreferenceable

[Marcel Reutegger]

Hi,

On 04/10/16 16:02, Julian Reschke wrote:

At least in our DocumentStoreImplementations (Mongo and RDB), making the
UUID something indexed by the storage (so either Mongo or the relational
database) should be relatively cheap (after all, the UUID never changes
once assigned, right?).


The jcr:uuid of a node mapped into a document *can* change. Consider the
case where a referenceable node with a given path is created, then
removed and re-created again with a different UUID.

Regards
 Marcel

Re: Possibility of making nt:resource unreferenceable

[Vikas Saurabh]

> At least in our DocumentStoreImplementations (Mongo and RDB), making the
> UUID something indexed by the storage (so either Mongo or the relational
> database) should be relatively cheap (after all, the UUID never changes once
> assigned, right?). That would eliminate the indexing overhead in Oak itself
> completely.

That'd just shift the storage from collection/table to
index/rdb_index. At least mongo tries to keep its indices in memory,
so the memory overhead would probably remain similar (well, in terms
of order of magnitude at least).

I think avoiding un-necessary uuid and optimizing uuid index are 2
separate problems. Chetan and Thomas, afaics, are discussing the
former.

Thanks,
Vikas

Re: Possibility of making nt:resource unreferenceable

[Julian Reschke]

On 2016-10-04 15:58, Thomas Mueller wrote:

Hi,


I still do not understand, why we do not delegate the UUID
generation/lookup to the persistence instead of insisting it to be just
like any other property index...


The UUID _generation_ is not a problem. (I it a bit slow, but scales well
- no issue there). The UUID _lookup_ is not a problem either. The problem
is indexing, because this is a unique index (wasted storage space, wasted
CPU and latency). The issue is, in most cases there is no need for this
UUID. So why index it. Sure, we could improve indexing (improve unique
property indexes). But it would be easier to not index those nodes at all.
...



At least in our DocumentStoreImplementations (Mongo and RDB), making the 
UUID something indexed by the storage (so either Mongo or the relational 
database) should be relatively cheap (after all, the UUID never changes 
once assigned, right?). That would eliminate the indexing overhead in 
Oak itself completely.


Best regards, Julian

Re: Possibility of making nt:resource unreferenceable

[Thomas Mueller]

Hi,

>I still do not understand, why we do not delegate the UUID
>generation/lookup to the persistence instead of insisting it to be just
>like any other property index...

The UUID _generation_ is not a problem. (I it a bit slow, but scales well
- no issue there). The UUID _lookup_ is not a problem either. The problem
is indexing, because this is a unique index (wasted storage space, wasted
CPU and latency). The issue is, in most cases there is no need for this
UUID. So why index it. Sure, we could improve indexing (improve unique
property indexes). But it would be easier to not index those nodes at all.

I would not change nt:resource because of backward compatibility issues.
Instead I would recommend not to use nt:resource. See SLING-6090.

Regards,
Thomas

Re: Possibility of making nt:resource unreferenceable

[Francesco Mari]

Hi,

2016-10-04 15:21 GMT+02:00 Julian Reschke :
> On 2016-10-04 08:43, Chetan Mehrotra wrote:
>>
>> <>
>>
>> Hi Team,
>>
>> Sometime back we discussed the requirement for oak:Resource as a non
>> referenceable replacement for nt:resource (OAk-4567). This topic was
>> also discussed on DL [1] and at that time it was decided that changing
>> the defaults (making nt:resource non referenceable ) is not possible
>> and hence applications should switch to other nodetypes while creating
>> nt:file instance.
>>
>> Towards that end I started discussion on Sling side as part of
>> SLING-6090. See [2] for discussion thread. However team there is of
>> the view that this would require changes in many places and wants us
>> to think again about changing the defaults.
>>
>> So question here is
>>
>> ===
>> Can we change the defaults for nt:resource nodetype to be non
>> referenceable. This has also been proposed in JCR 2.0. JR2 and Oak
>> though use the nodetype definition from JCR 1.0
>> ===
>>
>> To reiterate I am just aiming for a solution here which enables a user
>> to use a more optimum nodetype and get best performance out of
>> underlying repository.
>>
>> Hopefully we can converge on some agreement here :)
>> ...
>
>
> (I may have said this before...) -- if the issue we want to solve is that
> referenceable nodes in Oak are more expensive than they used to be in
> Jackrabbit, maybe the right thing to do is to make them less expensive?
>
> I still do not understand, why we do not delegate the UUID generation/lookup
> to the persistence instead of insisting it to be just like any other
> property index...

If you are going through hell, keep going.

Winston S. Churchill

> Best regards, Julian

Re: Possibility of making nt:resource unreferenceable

[Julian Reschke]

On 2016-10-04 08:43, Chetan Mehrotra wrote:

<>

Hi Team,

Sometime back we discussed the requirement for oak:Resource as a non
referenceable replacement for nt:resource (OAk-4567). This topic was
also discussed on DL [1] and at that time it was decided that changing
the defaults (making nt:resource non referenceable ) is not possible
and hence applications should switch to other nodetypes while creating
nt:file instance.

Towards that end I started discussion on Sling side as part of
SLING-6090. See [2] for discussion thread. However team there is of
the view that this would require changes in many places and wants us
to think again about changing the defaults.

So question here is

===
Can we change the defaults for nt:resource nodetype to be non
referenceable. This has also been proposed in JCR 2.0. JR2 and Oak
though use the nodetype definition from JCR 1.0
===

To reiterate I am just aiming for a solution here which enables a user
to use a more optimum nodetype and get best performance out of
underlying repository.

Hopefully we can converge on some agreement here :)
...


(I may have said this before...) -- if the issue we want to solve is 
that referenceable nodes in Oak are more expensive than they used to be 
in Jackrabbit, maybe the right thing to do is to make them less expensive?


I still do not understand, why we do not delegate the UUID 
generation/lookup to the persistence instead of insisting it to be just 
like any other property index...


Best regards, Julian

Possibility of making nt:resource unreferenceable

[Chetan Mehrotra]

<>

Hi Team,

Sometime back we discussed the requirement for oak:Resource as a non
referenceable replacement for nt:resource (OAk-4567). This topic was
also discussed on DL [1] and at that time it was decided that changing
the defaults (making nt:resource non referenceable ) is not possible
and hence applications should switch to other nodetypes while creating
nt:file instance.

Towards that end I started discussion on Sling side as part of
SLING-6090. See [2] for discussion thread. However team there is of
the view that this would require changes in many places and wants us
to think again about changing the defaults.

So question here is

===
Can we change the defaults for nt:resource nodetype to be non
referenceable. This has also been proposed in JCR 2.0. JR2 and Oak
though use the nodetype definition from JCR 1.0
===

To reiterate I am just aiming for a solution here which enables a user
to use a more optimum nodetype and get best performance out of
underlying repository.

Hopefully we can converge on some agreement here :)

Chetan Mehrotra
[1] http://markmail.org/thread/uj2ht4jwdrck7eja
[2] http://markmail.org/thread/77xvjxtx42euhss4
[3] https://java.net/jira/browse/JSR_283-428