[jira] [Commented] (OAK-8552) Minimize network calls required when creating a direct download URI

[Amit Jain (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918283#comment-16918283
 ] 

Amit Jain commented on OAK-8552:


[~alexander.klimetschek] 
Yes nothing needs to change now, I was just trying to highlight that there's 
dynamism involved with lastModified while other properties are stable over the 
lifetime of the blob even without de-deduplication.
{quote}With direct binary access we also no longer support de-duplication (too 
costly), so that aspect can be ignored for binaries uploaded that way or if a 
corresponding configuration is set
{quote}
But there's still a way that the binaries can be uploaded through JCR. So, we 
would in an application still have the mix of both I guess.
{quote}But there is still copying/versioning of nodes which leads to multiple 
references to the same blob - does this lead to a last modified update of the 
blob?
{quote}
No copy of nodes would not update the timestamp of the blob.

Also, there's the case of a 'Shared' DataStore where nodes might have been 
replicated from another instance through some form of replication and share the 
DataStore.

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Fix For: 1.18.0
>
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Closed] (OAK-8560) Update jackson-databind dependency to 2.9.9.3

[Nitin Gupta (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nitin Gupta closed OAK-8560.


> Update jackson-databind dependency to 2.9.9.3
> -
>
> Key: OAK-8560
> URL: https://issues.apache.org/jira/browse/OAK-8560
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: parent
>Affects Versions: 1.16.0, 1.8.15, 1.10.4
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Blocker
>  Labels: candidate_oak_1_6
> Fix For: 1.18.0, 1.8.16, 1.10.5
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8580) Stack trace logging for all binary downloading of cloud blob stores

[Matt Ryan (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918237#comment-16918237
 ] 

Matt Ryan commented on OAK-8580:


I created a PR for review 
[here|https://github.com/apache/jackrabbit-oak/pull/147] - 
[~alexander.klimetschek] if you have a chance to take a look and see if this 
matches what you had in mind.

> Stack trace logging for all binary downloading of cloud blob stores
> ---
>
> Key: OAK-8580
> URL: https://issues.apache.org/jira/browse/OAK-8580
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.16.0, 1.8.16
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8580.patch
>
>
> We want to be able to trace application code that invokes 
> Binary.getInputStream() when a cloud blob store is present.
> The Azure blob store has one place where it supports trace logging 
> [here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L1202-L1207],
>  but this misses another place blobs are fetched in 
> [read()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L218].
> Furthermore, it would be nice to have a separate logger for that in order to 
> create a log file that _only_ includes these stack traces, and no other debug 
> logs of the AzureBackend class.
> And same for the S3 BlobStore implementation.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8298) [Direct Binary Access] Blobs that are directly uploaded are not tracked by BlobIdTracker

[Matt Ryan (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918174#comment-16918174
 ] 

Matt Ryan commented on OAK-8298:


Fix for 1.10.5 in revision 
[r1866047|https://svn.apache.org/viewvc?view=revision&revision=1866047].

> [Direct Binary Access] Blobs that are directly uploaded are not tracked by 
> BlobIdTracker
> 
>
> Key: OAK-8298
> URL: https://issues.apache.org/jira/browse/OAK-8298
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: blob-cloud, blob-cloud-azure, blob-plugins
>Affects Versions: 1.16.0, 1.10.4
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Fix For: 1.18.0, 1.10.5
>
>
> Blobs that are uploaded to the content repository are supposed to be tracked 
> by the {{BlobIdTracker}} once the blob is saved.  This is done for blobs 
> uploaded the traditional way in {{DataStoreBlobStore.writeBlob()}} [0].
> For blobs uploaded directly, they do not go through this method and so the 
> blob ID is never added to the {{BlobIdTracker}}.  This has impact on DSGC as 
> the {{MarkSweepGarbageCollector}} relies on the {{BlobIdTracker}} to provide 
> an accurate accounting of the blob IDs in the blob store to determine which 
> ones to retain and which to delete.
> This should be a pretty easy fix.  All direct uploads pass through 
> {{DataStoreBlobStore.completeBlobUpload()}} [1], and we have at that point 
> access to the {{DataRecord}} created by the direct upload.  We can get the 
> blob ID from the {{DataRecord}} and add it to the {{BlobIdTracker}} there.
> [0] - 
> [https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/DataStoreBlobStore.java#L241]
> [1] - 
> [https://github.com/apache/jackrabbit-oak/blob/46cde5ee49622c94bc95648edf84cf4c00ae1d58/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/DataStoreBlobStore.java#L723]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (OAK-8298) [Direct Binary Access] Blobs that are directly uploaded are not tracked by BlobIdTracker

[Matt Ryan (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Ryan resolved OAK-8298.

Resolution: Fixed

> [Direct Binary Access] Blobs that are directly uploaded are not tracked by 
> BlobIdTracker
> 
>
> Key: OAK-8298
> URL: https://issues.apache.org/jira/browse/OAK-8298
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: blob-cloud, blob-cloud-azure, blob-plugins
>Affects Versions: 1.16.0, 1.10.4
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Fix For: 1.18.0, 1.10.5
>
>
> Blobs that are uploaded to the content repository are supposed to be tracked 
> by the {{BlobIdTracker}} once the blob is saved.  This is done for blobs 
> uploaded the traditional way in {{DataStoreBlobStore.writeBlob()}} [0].
> For blobs uploaded directly, they do not go through this method and so the 
> blob ID is never added to the {{BlobIdTracker}}.  This has impact on DSGC as 
> the {{MarkSweepGarbageCollector}} relies on the {{BlobIdTracker}} to provide 
> an accurate accounting of the blob IDs in the blob store to determine which 
> ones to retain and which to delete.
> This should be a pretty easy fix.  All direct uploads pass through 
> {{DataStoreBlobStore.completeBlobUpload()}} [1], and we have at that point 
> access to the {{DataRecord}} created by the direct upload.  We can get the 
> blob ID from the {{DataRecord}} and add it to the {{BlobIdTracker}} there.
> [0] - 
> [https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/DataStoreBlobStore.java#L241]
> [1] - 
> [https://github.com/apache/jackrabbit-oak/blob/46cde5ee49622c94bc95648edf84cf4c00ae1d58/oak-blob-plugins/src/main/java/org/apache/jackrabbit/oak/plugins/blob/datastore/DataStoreBlobStore.java#L723]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8581) Build Jackrabbit Oak #2346 failed

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918170#comment-16918170
 ] 

Hudson commented on OAK-8581:
-

Previously failing build now is OK.
 Passed run: [Jackrabbit Oak 
#2349|https://builds.apache.org/job/Jackrabbit%20Oak/2349/] [console 
log|https://builds.apache.org/job/Jackrabbit%20Oak/2349/console]

> Build Jackrabbit Oak #2346 failed
> -
>
> Key: OAK-8581
> URL: https://issues.apache.org/jira/browse/OAK-8581
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: continuous integration
>Reporter: Hudson
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #2346 has failed.
> First failed run: [Jackrabbit Oak 
> #2346|https://builds.apache.org/job/Jackrabbit%20Oak/2346/] [console 
> log|https://builds.apache.org/job/Jackrabbit%20Oak/2346/console]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (OAK-8580) Stack trace logging for all binary downloading of cloud blob stores

[Matt Ryan (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Ryan updated OAK-8580:
---
Affects Version/s: 1.8.16

> Stack trace logging for all binary downloading of cloud blob stores
> ---
>
> Key: OAK-8580
> URL: https://issues.apache.org/jira/browse/OAK-8580
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.16.0, 1.8.16
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8580.patch
>
>
> We want to be able to trace application code that invokes 
> Binary.getInputStream() when a cloud blob store is present.
> The Azure blob store has one place where it supports trace logging 
> [here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L1202-L1207],
>  but this misses another place blobs are fetched in 
> [read()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L218].
> Furthermore, it would be nice to have a separate logger for that in order to 
> create a log file that _only_ includes these stack traces, and no other debug 
> logs of the AzureBackend class.
> And same for the S3 BlobStore implementation.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (OAK-8580) Stack trace logging for all binary downloading of cloud blob stores

[Matt Ryan (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Ryan updated OAK-8580:
---
Affects Version/s: (was: 1.8.16)

> Stack trace logging for all binary downloading of cloud blob stores
> ---
>
> Key: OAK-8580
> URL: https://issues.apache.org/jira/browse/OAK-8580
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.16.0
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8580.patch
>
>
> We want to be able to trace application code that invokes 
> Binary.getInputStream() when a cloud blob store is present.
> The Azure blob store has one place where it supports trace logging 
> [here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L1202-L1207],
>  but this misses another place blobs are fetched in 
> [read()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L218].
> Furthermore, it would be nice to have a separate logger for that in order to 
> create a log file that _only_ includes these stack traces, and no other debug 
> logs of the AzureBackend class.
> And same for the S3 BlobStore implementation.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8581) Build Jackrabbit Oak #2346 failed

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918082#comment-16918082
 ] 

Hudson commented on OAK-8581:
-

Previously failing build now is OK.
 Passed run: [Jackrabbit Oak 
#2348|https://builds.apache.org/job/Jackrabbit%20Oak/2348/] [console 
log|https://builds.apache.org/job/Jackrabbit%20Oak/2348/console]

> Build Jackrabbit Oak #2346 failed
> -
>
> Key: OAK-8581
> URL: https://issues.apache.org/jira/browse/OAK-8581
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: continuous integration
>Reporter: Hudson
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #2346 has failed.
> First failed run: [Jackrabbit Oak 
> #2346|https://builds.apache.org/job/Jackrabbit%20Oak/2346/] [console 
> log|https://builds.apache.org/job/Jackrabbit%20Oak/2346/console]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (OAK-8552) Minimize network calls required when creating a direct download URI

[Matt Ryan (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Ryan resolved OAK-8552.

Fix Version/s: 1.18.0
   Resolution: Fixed

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Fix For: 1.18.0
>
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8552) Minimize network calls required when creating a direct download URI

[Matt Ryan (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918033#comment-16918033
 ] 

Matt Ryan commented on OAK-8552:


Speeding up signed URI generation is fixed with 
[r1866044|https://svn.apache.org/viewvc?view=revision&revision=1866044].

I believe that is sufficient to resolve this issue, considering that the other 
aspect (improving the check to determine if a binary was inlined) was resolved 
via OAK-8578 IIUC.  [~amitjain] please reopen if you disagree.

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8580) Stack trace logging for all binary downloading of cloud blob stores

[Matt Ryan (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918024#comment-16918024
 ] 

Matt Ryan commented on OAK-8580:


[~alexander.klimetschek] - I can go ahead with the addition of the stream 
loggers in {{AzureDataStore}} and {{S3DataStore}}.  I think if we want the JCR 
path in the logs that merits a separate issue with further discussion.  The 
fact that we reference blobs by blob ID indicates an intentional layer of 
abstraction between the node path and the blob itself, allowing to treat each 
as separate.  I'm not sure we want to undo that by exposing the JCR path to the 
data store even if it is just for the purpose of logging it.  We'd need some 
further discussion with the larger group - which is why I suggest a separate 
issue.

> Stack trace logging for all binary downloading of cloud blob stores
> ---
>
> Key: OAK-8580
> URL: https://issues.apache.org/jira/browse/OAK-8580
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.16.0, 1.8.16
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8580.patch
>
>
> We want to be able to trace application code that invokes 
> Binary.getInputStream() when a cloud blob store is present.
> The Azure blob store has one place where it supports trace logging 
> [here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L1202-L1207],
>  but this misses another place blobs are fetched in 
> [read()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L218].
> Furthermore, it would be nice to have a separate logger for that in order to 
> create a log file that _only_ includes these stack traces, and no other debug 
> logs of the AzureBackend class.
> And same for the S3 BlobStore implementation.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (OAK-8580) Stack trace logging for all binary downloading of cloud blob stores

[Matt Ryan (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Ryan updated OAK-8580:
---
Issue Type: Improvement  (was: Bug)

> Stack trace logging for all binary downloading of cloud blob stores
> ---
>
> Key: OAK-8580
> URL: https://issues.apache.org/jira/browse/OAK-8580
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.16.0, 1.8.16
>Reporter: Alexander Klimetschek
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8580.patch
>
>
> We want to be able to trace application code that invokes 
> Binary.getInputStream() when a cloud blob store is present.
> The Azure blob store has one place where it supports trace logging 
> [here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L1202-L1207],
>  but this misses another place blobs are fetched in 
> [read()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-cloud-azure/src/main/java/org/apache/jackrabbit/oak/blob/cloud/azure/blobstorage/AzureBlobStoreBackend.java#L218].
> Furthermore, it would be nice to have a separate logger for that in order to 
> create a log file that _only_ includes these stack traces, and no other debug 
> logs of the AzureBackend class.
> And same for the S3 BlobStore implementation.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (OAK-4508) LoggingDocumentStore: better support for multiple instances

[Jira]

 [ 
https://issues.apache.org/jira/browse/OAK-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

José Andrés Cordero Benítez updated OAK-4508:
-
Attachment: prefix-change-LoggingDocumentStore-OAK4508.patch

> LoggingDocumentStore: better support for multiple instances
> ---
>
> Key: OAK-4508
> URL: https://issues.apache.org/jira/browse/OAK-4508
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: documentmk
>Reporter: Julian Reschke
>Priority: Minor
> Attachments: prefix-change-LoggingDocumentStore-OAK4508.patch
>
>
> It would be cool if the logging could be configured to use a specific prefix 
> instead of "ds." - this would make it more useful when debugging code that 
> involves two different ds instances in the same VM.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (OAK-4508) LoggingDocumentStore: better support for multiple instances

[Jira]

 [ 
https://issues.apache.org/jira/browse/OAK-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

José Andrés Cordero Benítez updated OAK-4508:
-
Attachment: (was: prefix-change-LoggingDocumentStore-OAK4508.patch)

> LoggingDocumentStore: better support for multiple instances
> ---
>
> Key: OAK-4508
> URL: https://issues.apache.org/jira/browse/OAK-4508
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: documentmk
>Reporter: Julian Reschke
>Priority: Minor
> Attachments: prefix-change-LoggingDocumentStore-OAK4508.patch
>
>
> It would be cool if the logging could be configured to use a specific prefix 
> instead of "ds." - this would make it more useful when debugging code that 
> involves two different ds instances in the same VM.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8581) Build Jackrabbit Oak #2346 failed

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917953#comment-16917953
 ] 

Hudson commented on OAK-8581:
-

Build is still failing.
Failed run: [Jackrabbit Oak 
#2347|https://builds.apache.org/job/Jackrabbit%20Oak/2347/] [console 
log|https://builds.apache.org/job/Jackrabbit%20Oak/2347/console]

> Build Jackrabbit Oak #2346 failed
> -
>
> Key: OAK-8581
> URL: https://issues.apache.org/jira/browse/OAK-8581
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: continuous integration
>Reporter: Hudson
>Priority: Major
>
> No description is provided
> The build Jackrabbit Oak #2346 has failed.
> First failed run: [Jackrabbit Oak 
> #2346|https://builds.apache.org/job/Jackrabbit%20Oak/2346/] [console 
> log|https://builds.apache.org/job/Jackrabbit%20Oak/2346/console]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (OAK-8583) getNodeByIdentifier may fail with RuntimeException

[Julian Reschke (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917805#comment-16917805
 ] 

Julian Reschke edited comment on OAK-8583 at 8/28/19 5:23 PM:
--

trunk: [r1866041|http://svn.apache.org/r1866041] 
[r1866039|http://svn.apache.org/r1866039]


was (Author: reschke):
trunk: [r1866039|http://svn.apache.org/r1866039]

> getNodeByIdentifier may fail with RuntimeException
> --
>
> Key: OAK-8583
> URL: https://issues.apache.org/jira/browse/OAK-8583
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: jcr
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_10
> Fix For: 1.20
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (OAK-8583) getNodeByIdentifier may fail with RuntimeException

[Julian Reschke (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke resolved OAK-8583.
-
Fix Version/s: 1.20
   Resolution: Fixed

> getNodeByIdentifier may fail with RuntimeException
> --
>
> Key: OAK-8583
> URL: https://issues.apache.org/jira/browse/OAK-8583
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: jcr
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_10
> Fix For: 1.20
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (OAK-8552) Minimize network calls required when creating a direct download URI

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917841#comment-16917841
 ] 

Alexander Klimetschek edited comment on OAK-8552 at 8/28/19 2:54 PM:
-

[~amitjain] I don’t think [~ianeboston] was suggesting to use the 
jcr:lastModified JCR property for that. All of these could be internal fields 
not exposed on the JCR level.

With direct binary access we also no longer support de-duplication (too 
costly), so that aspect can be ignored for binaries uploaded that way or if a 
corresponding configuration is set. But there is still copying/versioning of 
nodes which leads to multiple references to the same blob - does this lead to a 
last modified update of the blob? (Just curious because I wasn’t aware of that)

Nonetheless, for the issue at hand I don’t think we need to change anything 
stored in the NodeStore. The size/length is already stored in the NS as part of 
the internal blob id (OAK-8314).


was (Author: alexander.klimetschek):
[~amitjain] I don’t think [~ianeboston] was suggesting to use the 
jcr:lastModified JCR property for that. All of these could be internal fields 
not exposed on the JCR level.

With direct binary access we also no longer support de-duplication (too 
costly), so that aspect can be ignored for binaries uploaded that way or if a 
corresponding configuration is set. But there is still copying/versioning of 
nodes which leads to multiple references to the same blob - does this lead to a 
last modified update of the blob? (Just curious because I wasn’t aware of that)

Nonetheless, for the issue at hand I don’t think we need to change anything 
stored in the NodeStore. The size/length is already stored in the NS as part of 
the internal blob id.

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8552) Minimize network calls required when creating a direct download URI

[Alexander Klimetschek (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917841#comment-16917841
 ] 

Alexander Klimetschek commented on OAK-8552:


[~amitjain] I don’t think [~ianeboston] was suggesting to use the 
jcr:lastModified JCR property for that. All of these could be internal fields 
not exposed on the JCR level.

With direct binary access we also no longer support de-duplication (too 
costly), so that aspect can be ignored for binaries uploaded that way or if a 
corresponding configuration is set. But there is still copying/versioning of 
nodes which leads to multiple references to the same blob - does this lead to a 
last modified update of the blob? (Just curious because I wasn’t aware of that)

Nonetheless, for the issue at hand I don’t think we need to change anything 
stored in the NodeStore. The size/length is already stored in the NS as part of 
the internal blob id.

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (OAK-8583) getNodeByIdentifier may fail with RuntimeException

[Julian Reschke (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-8583:

Affects Version/s: (was: 1.20)

> getNodeByIdentifier may fail with RuntimeException
> --
>
> Key: OAK-8583
> URL: https://issues.apache.org/jira/browse/OAK-8583
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: jcr
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_10
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (OAK-8583) getNodeByIdentifier may fail with RuntimeException

[Julian Reschke (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-8583:

Labels: candidate_oak_1_10  (was: )

> getNodeByIdentifier may fail with RuntimeException
> --
>
> Key: OAK-8583
> URL: https://issues.apache.org/jira/browse/OAK-8583
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: jcr
>Affects Versions: 1.20
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_10
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8583) getNodeByIdentifier may fail with RuntimeException

[Julian Reschke (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917805#comment-16917805
 ] 

Julian Reschke commented on OAK-8583:
-

trunk: [r1866039|http://svn.apache.org/r1866039]

> getNodeByIdentifier may fail with RuntimeException
> --
>
> Key: OAK-8583
> URL: https://issues.apache.org/jira/browse/OAK-8583
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: jcr
>Affects Versions: 1.20
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8583) getNodeByIdentifier may fail with RuntimeException

[Julian Reschke (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917795#comment-16917795
 ] 

Julian Reschke commented on OAK-8583:
-

...however, the JCR API in general is not supposed to fail with unchecked 
exceptions.

> getNodeByIdentifier may fail with RuntimeException
> --
>
> Key: OAK-8583
> URL: https://issues.apache.org/jira/browse/OAK-8583
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: jcr
>Affects Versions: 1.20
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (OAK-8583) getNodeByIdentifier may fail with RuntimeException

[Julian Reschke (Jira)]

Julian Reschke created OAK-8583:
---

 Summary: getNodeByIdentifier may fail with RuntimeException
 Key: OAK-8583
 URL: https://issues.apache.org/jira/browse/OAK-8583
 Project: Jackrabbit Oak
  Issue Type: Task
  Components: jcr
Affects Versions: 1.20
Reporter: Julian Reschke
Assignee: Julian Reschke






--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-4508) LoggingDocumentStore: better support for multiple instances

[Jira]

[ 
https://issues.apache.org/jira/browse/OAK-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917786#comment-16917786
 ] 

José Andrés Cordero Benítez commented on OAK-4508:
--

I have created a patch to add support to change the prefix. It can be changed 
using a new constructor, or using the "withPrefix" builder.

It includes a test to check the prefix is actually in the logs.

> LoggingDocumentStore: better support for multiple instances
> ---
>
> Key: OAK-4508
> URL: https://issues.apache.org/jira/browse/OAK-4508
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: documentmk
>Reporter: Julian Reschke
>Priority: Minor
> Attachments: prefix-change-LoggingDocumentStore-OAK4508.patch
>
>
> It would be cool if the logging could be configured to use a specific prefix 
> instead of "ds." - this would make it more useful when debugging code that 
> involves two different ds instances in the same VM.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (OAK-4508) LoggingDocumentStore: better support for multiple instances

[Jira]

 [ 
https://issues.apache.org/jira/browse/OAK-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

José Andrés Cordero Benítez updated OAK-4508:
-
Attachment: prefix-change-LoggingDocumentStore-OAK4508.patch

> LoggingDocumentStore: better support for multiple instances
> ---
>
> Key: OAK-4508
> URL: https://issues.apache.org/jira/browse/OAK-4508
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: documentmk
>Reporter: Julian Reschke
>Priority: Minor
> Attachments: prefix-change-LoggingDocumentStore-OAK4508.patch
>
>
> It would be cool if the logging could be configured to use a specific prefix 
> instead of "ds." - this would make it more useful when debugging code that 
> involves two different ds instances in the same VM.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8408) UserImporter must not trigger creation of rep:pwd node unless included in xml (initial-pw-change)

[Julian Reschke (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917762#comment-16917762
 ] 

Julian Reschke commented on OAK-8408:
-

trunk: [r1866037|http://svn.apache.org/r1866037] (1.16.0) 
[r1862074|http://svn.apache.org/r1862074]

> UserImporter must not trigger creation of rep:pwd node unless included in xml 
> (initial-pw-change)
> -
>
> Key: OAK-8408
> URL: https://issues.apache.org/jira/browse/OAK-8408
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: core, security
>Reporter: angela
>Assignee: angela
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: OAK-8408-tests.patch, OAK-8408.patch
>
>
> when xml-importing an existing user (i.e. {{Tree}} doesn't have status NEW 
> upon import) calling {{UserManagerImpl.setPassword}} will force the creation 
> of the {{rep:pwd}} node and {{rep:passwordLastModified}} property contained 
> therein _if_ theinitial-password-change feature is enabled.
> imo the {{rep:pwd}} (and any properties contained therein) must not be 
> auto-created by should only be imported if contained in the XML. 
> proposed fix: {{UserManagerImpl.setPassword}} already contains special 
> treatment for the password hashing triggered upon xml import -> renaming that 
> flag and respect it for the handling of the pw last modified.
> [~stillalex], wdyt?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Issue Comment Deleted] (OAK-8408) UserImporter must not trigger creation of rep:pwd node unless included in xml (initial-pw-change)

[Julian Reschke (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-8408:

Comment: was deleted

(was: trunk: (1.16.0) [r1862074|http://svn.apache.org/r1862074])

> UserImporter must not trigger creation of rep:pwd node unless included in xml 
> (initial-pw-change)
> -
>
> Key: OAK-8408
> URL: https://issues.apache.org/jira/browse/OAK-8408
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: core, security
>Reporter: angela
>Assignee: angela
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: OAK-8408-tests.patch, OAK-8408.patch
>
>
> when xml-importing an existing user (i.e. {{Tree}} doesn't have status NEW 
> upon import) calling {{UserManagerImpl.setPassword}} will force the creation 
> of the {{rep:pwd}} node and {{rep:passwordLastModified}} property contained 
> therein _if_ theinitial-password-change feature is enabled.
> imo the {{rep:pwd}} (and any properties contained therein) must not be 
> auto-created by should only be imported if contained in the XML. 
> proposed fix: {{UserManagerImpl.setPassword}} already contains special 
> treatment for the password hashing triggered upon xml import -> renaming that 
> flag and respect it for the handling of the pw last modified.
> [~stillalex], wdyt?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8408) UserImporter must not trigger creation of rep:pwd node unless included in xml (initial-pw-change)

[Julian Reschke (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917756#comment-16917756
 ] 

Julian Reschke commented on OAK-8408:
-

Eclipse flags this line: 

 as error. Will fix.

 

> UserImporter must not trigger creation of rep:pwd node unless included in xml 
> (initial-pw-change)
> -
>
> Key: OAK-8408
> URL: https://issues.apache.org/jira/browse/OAK-8408
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: core, security
>Reporter: angela
>Assignee: angela
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: OAK-8408-tests.patch, OAK-8408.patch
>
>
> when xml-importing an existing user (i.e. {{Tree}} doesn't have status NEW 
> upon import) calling {{UserManagerImpl.setPassword}} will force the creation 
> of the {{rep:pwd}} node and {{rep:passwordLastModified}} property contained 
> therein _if_ theinitial-password-change feature is enabled.
> imo the {{rep:pwd}} (and any properties contained therein) must not be 
> auto-created by should only be imported if contained in the XML. 
> proposed fix: {{UserManagerImpl.setPassword}} already contains special 
> treatment for the password hashing triggered upon xml import -> renaming that 
> flag and respect it for the handling of the pw last modified.
> [~stillalex], wdyt?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8408) UserImporter must not trigger creation of rep:pwd node unless included in xml (initial-pw-change)

[Julian Reschke (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917754#comment-16917754
 ] 

Julian Reschke commented on OAK-8408:
-

trunk: (1.16.0) [r1862074|http://svn.apache.org/r1862074]

> UserImporter must not trigger creation of rep:pwd node unless included in xml 
> (initial-pw-change)
> -
>
> Key: OAK-8408
> URL: https://issues.apache.org/jira/browse/OAK-8408
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: core, security
>Reporter: angela
>Assignee: angela
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: OAK-8408-tests.patch, OAK-8408.patch
>
>
> when xml-importing an existing user (i.e. {{Tree}} doesn't have status NEW 
> upon import) calling {{UserManagerImpl.setPassword}} will force the creation 
> of the {{rep:pwd}} node and {{rep:passwordLastModified}} property contained 
> therein _if_ theinitial-password-change feature is enabled.
> imo the {{rep:pwd}} (and any properties contained therein) must not be 
> auto-created by should only be imported if contained in the XML. 
> proposed fix: {{UserManagerImpl.setPassword}} already contains special 
> treatment for the password hashing triggered upon xml import -> renaming that 
> flag and respect it for the handling of the pw last modified.
> [~stillalex], wdyt?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (OAK-8582) Failing test in MongoDB 4.2.0: BasicDocumentStoreTest.testLongId

[Jira]

José Andrés Cordero Benítez created OAK-8582:


 Summary: Failing test in MongoDB 4.2.0: 
BasicDocumentStoreTest.testLongId
 Key: OAK-8582
 URL: https://issues.apache.org/jira/browse/OAK-8582
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: mongomk
Reporter: José Andrés Cordero Benítez


{{This is the output of the test. Apparently some changes in MongoDB 4.2.0 are 
allowing to correctly create the document with such long id, which makes the 
test to fail as it expects an error in that case.}}

{{Works fine with MongoDB 4.0.12.}}
{quote}{{[ERROR] Tests run: 111, Failures: 1, Errors: 0, Skipped: 2, Time 
elapsed: 23.079 s <<< FAILURE! - in 
org.apache.jackrabbit.oak.plugins.document.BasicDocumentStoreTest}}
{{[ERROR] testLongId[MongoFixture: 
MongoDB](org.apache.jackrabbit.oak.plugins.document.BasicDocumentStoreTest) 
Time elapsed: 0.019 s <<< FAILURE!}}
{{java.lang.AssertionError: create() with ultra-long id needs to fail}}
 \{{ at 
org.apache.jackrabbit.oak.plugins.document.BasicDocumentStoreTest.testLongId(BasicDocumentStoreTest.java:414)}}
{quote}
I think this is related with the Index Key Limit, which has been removed in 
MongoDB 4.2.0:

[https://docs.mongodb.com/manual/reference/limits/#Index-Key-Limit]

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8579) Composite Node Store: Allow creating an index in the read-only repo first

[Vikas Saurabh (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917572#comment-16917572
 ] 

Vikas Saurabh commented on OAK-8579:


[~tmueller], I was thinking about this issue after we discussed "fix validator 
approach". I wonder if another way could (should??) be for diff to not report 
"magically appearing node to not be reported in {{childNodeAdded}}". I do see 
that this is more hacky - but what I'm really trying to evaluate "should this 
new type of node incarnation be treated differently than simple child node 
addition". wdyt?

> Composite Node Store: Allow creating an index in the read-only repo first
> -
>
> Key: OAK-8579
> URL: https://issues.apache.org/jira/browse/OAK-8579
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: composite, core, indexing, lucene
>Reporter: Thomas Mueller
>Priority: Major
>
> Currently, it is not allowed to first create a new index in the read-only 
> repository, and then in the read-write repository. Trying to do so will fail 
> with "OakConstraint0001: /oak:index/.../:oak:mount-readOnlyV1-index-data[[]]: 
> The primary type null does not exist"
> See OAK-7917: oak-lucene/src/test/java/org/apache/jackrabbit/oak/composite - 
> CompositeNodeStoreLuceneIndexTest.java 
> tryAddIndexInReadWriteWithIndexExistinginReadOnly line 112. 
> It would be better to allow this use case, to reduce the possibility of 
> problems.
> We should specially test with lucene indexes, but also with property indexes. 
> (If that's more complicated, we can concentrate on the lucene case first.)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (OAK-8579) Composite Node Store: Allow creating an index in the read-only repo first

[Thomas Mueller (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917558#comment-16917558
 ] 

Thomas Mueller edited comment on OAK-8579 at 8/28/19 8:56 AM:
--

> inadvertently ignoring/skipping some needed check here

[~nitigup] the changes only affect hidden nodes 
(NodeStateUtils.isHidden(name)). With the JCR API it is not possible to create 
or access hidden nodes: test.addNode(":x") fails with the exception 
"javax.jcr.PathNotFoundException: :x", much before you have the option to save 
the change. So I believe it is not a risk.

> we can reproduce this and see why this check is only coming into picture when 
> index is getting created in read only part of the repo 

This we know. But it is good to explicitly describe the problem: assume you 
create an index exists in the read-only part of the repo, using the "composite 
seed" mode, then the some hidden nodes are created by indexing. But only in the 
so called "read-only" repository. It is not actually read-only in this mode. 
But it is stored in a different segment store. If you investigate, for example 
with oak-run explore, how this repository looks like, you would see something 
like this

{noformat}
/oak:index/abc
/oak:index/abc/:oak:mount-readOnlyV1-index-data
{noformat}

It is not yet created in the read-write repository. So when starting the 
composite store that opens the read-only repo in read-only mode, and the 
read-write repo in read-write mode, the nodes above are not visible.

Now, assume you create an index in the read-write repo with this name:

{noformat}
/oak:index/abc
{noformat}

It is not yet indexing at that time, just storing the node itself. (Possibly 
you could even create a node of any other type, for example nt:base, so not an 
index definition node - I didn't test). What happens is: the hidden node 
/oak:index/abc/:oak:mount-readOnlyV1-index-data will now become visible, due to 
how the composite node store works and is configured. For Oak, it will looks 
like the node /oak:index/abc/:oak:mount-readOnlyV1-index-data was just added by 
the user (which is impossible to do using the JCR API, but the Oak verifiers 
don't know that).

So the verifiers currently check if this node is a correct JCR node. It's not: 
the primary type is not set, so is "null".

To resolve this issue, I changed the verifiers to ignore hidden nodes.


was (Author: tmueller):
> inadvertently ignoring/skipping some needed check here

[~nitigup] the changes only affect hidden nodes 
(NodeStateUtils.isHidden(name)). With the JCR API it is not possible to create 
or access hidden nodes. So I believe it is not a risk.

> we can reproduce this and see why this check is only coming into picture when 
> index is getting created in read only part of the repo 

This we know. But it is good to explicitly describe the problem: assume you 
create an index exists in the read-only part of the repo, using the "composite 
seed" mode, then the some hidden nodes are created by indexing. But only in the 
so called "read-only" repository. It is not actually read-only in this mode. 
But it is stored in a different segment store. If you investigate, for example 
with oak-run explore, how this repository looks like, you would see something 
like this

{noformat}
/oak:index/abc
/oak:index/abc/:oak:mount-readOnlyV1-index-data
{noformat}

It is not yet created in the read-write repository. So when starting the 
composite store that opens the read-only repo in read-only mode, and the 
read-write repo in read-write mode, the nodes above are not visible.

Now, assume you create an index in the read-write repo with this name:

{noformat}
/oak:index/abc
{noformat}

It is not yet indexing at that time, just storing the node itself. (Possibly 
you could even create a node of any other type, for example nt:base, so not an 
index definition node - I didn't test). What happens is: the hidden node 
/oak:index/abc/:oak:mount-readOnlyV1-index-data will now become visible, due to 
how the composite node store works and is configured. For Oak, it will looks 
like the node /oak:index/abc/:oak:mount-readOnlyV1-index-data was just added by 
the user (which is impossible to do using the JCR API, but the Oak verifiers 
don't know that).

So the verifiers currently check if this node is a correct JCR node. It's not: 
the primary type is not set, so is "null".

To resolve this issue, I changed the verifiers to ignore hidden nodes.

> Composite Node Store: Allow creating an index in the read-only repo first
> -
>
> Key: OAK-8579
> URL: https://issues.apache.org/jira/browse/OAK-8579
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: composite, core, indexing, lucene
>Reporter: Thomas Mueller
>Prio

[jira] [Commented] (OAK-8579) Composite Node Store: Allow creating an index in the read-only repo first

[Thomas Mueller (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917558#comment-16917558
 ] 

Thomas Mueller commented on OAK-8579:
-

> inadvertently ignoring/skipping some needed check here

[~nitigup] the changes only affect hidden nodes 
(NodeStateUtils.isHidden(name)). With the JCR API it is not possible to create 
or access hidden nodes. So I believe it is not a risk.

> we can reproduce this and see why this check is only coming into picture when 
> index is getting created in read only part of the repo 

This we know. But it is good to explicitly describe the problem: assume you 
create an index exists in the read-only part of the repo, using the "composite 
seed" mode, then the some hidden nodes are created by indexing. But only in the 
so called "read-only" repository. It is not actually read-only in this mode. 
But it is stored in a different segment store. If you investigate, for example 
with oak-run explore, how this repository looks like, you would see something 
like this

{noformat}
/oak:index/abc
/oak:index/abc/:oak:mount-readOnlyV1-index-data
{noformat}

It is not yet created in the read-write repository. So when starting the 
composite store that opens the read-only repo in read-only mode, and the 
read-write repo in read-write mode, the nodes above are not visible.

Now, assume you create an index in the read-write repo with this name:

{noformat}
/oak:index/abc
{noformat}

It is not yet indexing at that time, just storing the node itself. (Possibly 
you could even create a node of any other type, for example nt:base, so not an 
index definition node - I didn't test). What happens is: the hidden node 
/oak:index/abc/:oak:mount-readOnlyV1-index-data will now become visible, due to 
how the composite node store works and is configured. For Oak, it will looks 
like the node /oak:index/abc/:oak:mount-readOnlyV1-index-data was just added by 
the user (which is impossible to do using the JCR API, but the Oak verifiers 
don't know that).

So the verifiers currently check if this node is a correct JCR node. It's not: 
the primary type is not set, so is "null".

To resolve this issue, I changed the verifiers to ignore hidden nodes.

> Composite Node Store: Allow creating an index in the read-only repo first
> -
>
> Key: OAK-8579
> URL: https://issues.apache.org/jira/browse/OAK-8579
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: composite, core, indexing, lucene
>Reporter: Thomas Mueller
>Priority: Major
>
> Currently, it is not allowed to first create a new index in the read-only 
> repository, and then in the read-write repository. Trying to do so will fail 
> with "OakConstraint0001: /oak:index/.../:oak:mount-readOnlyV1-index-data[[]]: 
> The primary type null does not exist"
> See OAK-7917: oak-lucene/src/test/java/org/apache/jackrabbit/oak/composite - 
> CompositeNodeStoreLuceneIndexTest.java 
> tryAddIndexInReadWriteWithIndexExistinginReadOnly line 112. 
> It would be better to allow this use case, to reduce the possibility of 
> problems.
> We should specially test with lucene indexes, but also with property indexes. 
> (If that's more complicated, we can concentrate on the lucene case first.)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (OAK-8581) Build Jackrabbit Oak #2346 failed

[Hudson (Jira)]

Hudson created OAK-8581:
---

 Summary: Build Jackrabbit Oak #2346 failed
 Key: OAK-8581
 URL: https://issues.apache.org/jira/browse/OAK-8581
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: continuous integration
Reporter: Hudson


No description is provided

The build Jackrabbit Oak #2346 has failed.
First failed run: [Jackrabbit Oak 
#2346|https://builds.apache.org/job/Jackrabbit%20Oak/2346/] [console 
log|https://builds.apache.org/job/Jackrabbit%20Oak/2346/console]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8552) Minimize network calls required when creating a direct download URI

[Amit Jain (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917525#comment-16917525
 ] 

Amit Jain commented on OAK-8552:


[~ianeboston]

 
{quote}the Oak NodeStore (SegmentNodeStore or DocumentNodeStore) should be the 
record of authority for blob existence, length, lastModified
{quote}
The node's jcr:lastModified does not reflect the blob's lastModified timestamp 
and only signifies when the node was last modified (IIUC the spec also says 
that [1]). I am also not sure if it can reflect the blobs lastModified without 
taking a severe hit. The reason being since blobs are de-duplicated, when an 
already existing blob is uploaded to Jcr, its lastModified stamp is updated in 
the DataStore and the blob is not uploaded again to the DataStore. This update 
to the blob's lastModified is a requirement for DGC.

This updated lastModified for the blob cannot be updated for all nodes from 
where already referenced retrospectively without a performance hit (and maybe a 
design change, DataStore is the lowest layer and has no information of the 
NodeStore and de-duplication with SHA hash is an implementation detail not 
known to the NodeStore). 

[1] - [https://docs.adobe.com/docs/en/spec/jcr/2.0/3_Repository_Model.html]

> Minimize network calls required when creating a direct download URI
> ---
>
> Key: OAK-8552
> URL: https://issues.apache.org/jira/browse/OAK-8552
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
> Attachments: OAK-8552_ApiChange.patch
>
>
> We need to isolate and try to optimize network calls required to create a 
> direct download URI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (OAK-8578) Introduce API to check whether blob inlined in Id

[Amit Jain (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917497#comment-16917497
 ] 

Amit Jain edited comment on OAK-8578 at 8/28/19 7:43 AM:
-

*Benchmark results - S3*

*Setup*
 Files uploaded - 100
 File size - 20 KB

S3DataStore config _cacheSize=0_

*Inlined check*
{noformat}
 
# GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 273 278 313 329 1253 189
{noformat}
S3DataStore config _presignedHttpDownloadURIExpirySeconds=0 (URI not retrieved 
from backend)
{noformat}
 # GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 0 0 0 0 1050 228101194
{noformat}
*getReference*
{noformat}
 # GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 549 597 628 834 1674 88
{noformat}
S3DataStore config presignedHttpDownloadURIExpirySeconds=0 (URI not retrieved 
from backend)
{noformat}
 # GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 272 281 313 349 1149 186
{noformat}
The above shows improvements after making changes to use #isInlined method


was (Author: amitjain):
*Benchmark results*

*Setup*
 Files uploaded - 100
 File size - 20 KB

S3DataStore config _cacheSize=0_

*Inlined check*
{noformat}
 
# GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 273 278 313 329 1253 189
{noformat}
S3DataStore config _presignedHttpDownloadURIExpirySeconds=0 (URI not retrieved 
from backend)
{noformat}
 # GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 0 0 0 0 1050 228101194
{noformat}
*getReference*
{noformat}
 # GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 549 597 628 834 1674 88
{noformat}
S3DataStore config presignedHttpDownloadURIExpirySeconds=0 (URI not retrieved 
from backend)
{noformat}
 # GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 272 281 313 349 1149 186
{noformat}
The above shows improvements after making changes to use #isInlined method

> Introduce API to check whether blob inlined in Id
> -
>
> Key: OAK-8578
> URL: https://issues.apache.org/jira/browse/OAK-8578
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Amit Jain
>Assignee: Amit Jain
>Priority: Major
> Fix For: 1.18.0
>
>
> New API in blob to check whether blob inlined in Id. This will replace calls 
> to getReference which was used as a proxy to check whether blob inlined and 
> avoid remote calls to the backend (potentially expensive ones over cloud).



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8551) Minimize network calls in cloud data stores (performance optimization)

[Amit Jain (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917508#comment-16917508
 ] 

Amit Jain commented on OAK-8551:


{quote}Signed urls should be issued based on information in the NodeStore alone 
with no network API calls.
{quote}
For cloud DataStores there will be *a* network call to get the URI from the 
cloud provider. After reverting OAK-7998 in place that n/w call is down to only 
that.

I implemented the change to isInlined from getReference in OAK-8578. The 
comment 
https://issues.apache.org/jira/browse/OAK-8578?focusedCommentId=16917497&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16917497
 shows some improvements but numbers still bogged down by the actual create uri 
call to the backend. 

> Minimize network calls in cloud data stores (performance optimization)
> --
>
> Key: OAK-8551
> URL: https://issues.apache.org/jira/browse/OAK-8551
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob-cloud, blob-cloud-azure
>Affects Versions: 1.16.0, 1.10.4
>Reporter: Matt Ryan
>Assignee: Matt Ryan
>Priority: Major
>
> Oak cloud data stores (e.g. {{AzureDataStore}}, {{S3DataStore}}) are by 
> definition more susceptible to performance degradation due to network issues. 
>  While we can't do much about the performance of uploading or downloading a 
> blob, there are other places within the implementations where we are making 
> network calls to the storage service which might be avoidable or minimized.
> One example is the {{exists()}} call to check whether a blob with a 
> particular identifier exists in the blob storage.  In some places 
> {{exists()}} is being called where instead we could simply attempt the 
> network access and handle failures elegantly, avoiding making an extra 
> network call.  In other places perhaps a cache could be used to minimize 
> round trips.
> Another example is the higher-level {{getReference()}} call in 
> {{DataStoreBlobStore}}.  This asks the implementation for a {{DataRecord}} 
> and then gets the reference from that, but in truth the data store backend 
> can already obtain a reference for an identifier on its own.  Asking for the 
> {{DataRecord}} however requires a network request to get the blob metadata 
> for the record.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8579) Composite Node Store: Allow creating an index in the read-only repo first

[Robert Munteanu (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917505#comment-16917505
 ] 

Robert Munteanu commented on OAK-8579:
--

[~tmueller] - sorry, I am too disconnected from these changes to feel confident 
that I can review them.

> Composite Node Store: Allow creating an index in the read-only repo first
> -
>
> Key: OAK-8579
> URL: https://issues.apache.org/jira/browse/OAK-8579
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: composite, core, indexing, lucene
>Reporter: Thomas Mueller
>Priority: Major
>
> Currently, it is not allowed to first create a new index in the read-only 
> repository, and then in the read-write repository. Trying to do so will fail 
> with "OakConstraint0001: /oak:index/.../:oak:mount-readOnlyV1-index-data[[]]: 
> The primary type null does not exist"
> See OAK-7917: oak-lucene/src/test/java/org/apache/jackrabbit/oak/composite - 
> CompositeNodeStoreLuceneIndexTest.java 
> tryAddIndexInReadWriteWithIndexExistinginReadOnly line 112. 
> It would be better to allow this use case, to reduce the possibility of 
> problems.
> We should specially test with lucene indexes, but also with property indexes. 
> (If that's more complicated, we can concentrate on the lucene case first.)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (OAK-8578) Introduce API to check whether blob inlined in Id

[Amit Jain (Jira)]

 [ 
https://issues.apache.org/jira/browse/OAK-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Jain resolved OAK-8578.

Resolution: Fixed

*Benchmark results*

*Setup*
 Files uploaded - 100
 File size - 20 KB

S3DataStore config _cacheSize=0_

*Inlined check*
{noformat}
 
# GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 273 278 313 329 1253 189
{noformat}
S3DataStore config _presignedHttpDownloadURIExpirySeconds=0 (URI not retrieved 
from backend)
{noformat}
 # GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 0 0 0 0 1050 228101194
{noformat}
*getReference*
{noformat}
 # GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 549 597 628 834 1674 88
{noformat}
S3DataStore config presignedHttpDownloadURIExpirySeconds=0 (URI not retrieved 
from backend)
{noformat}
 # GetURITest C min 10% 50% 90% max N
 Oak-Segment-Tar-DS 1 272 281 313 349 1149 186
{noformat}
The above shows improvements after making changes to use #isInlined method

> Introduce API to check whether blob inlined in Id
> -
>
> Key: OAK-8578
> URL: https://issues.apache.org/jira/browse/OAK-8578
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Amit Jain
>Assignee: Amit Jain
>Priority: Major
> Fix For: 1.18.0
>
>
> New API in blob to check whether blob inlined in Id. This will replace calls 
> to getReference which was used as a proxy to check whether blob inlined and 
> avoid remote calls to the backend (potentially expensive ones over cloud).



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8578) Introduce API to check whether blob inlined in Id

[Amit Jain (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917493#comment-16917493
 ] 

Amit Jain commented on OAK-8578:


Committed changes on trunk with
 * [1866022|http://svn.apache.org/viewvc?rev=1866022&view=rev]
 * [1866023|http://svn.apache.org/viewvc?rev=1866023&view=rev]

> Introduce API to check whether blob inlined in Id
> -
>
> Key: OAK-8578
> URL: https://issues.apache.org/jira/browse/OAK-8578
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Amit Jain
>Assignee: Amit Jain
>Priority: Major
> Fix For: 1.18.0
>
>
> New API in blob to check whether blob inlined in Id. This will replace calls 
> to getReference which was used as a proxy to check whether blob inlined and 
> avoid remote calls to the backend (potentially expensive ones over cloud).



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (OAK-8578) Introduce API to check whether blob inlined in Id

[Julian Reschke (Jira)]

[ 
https://issues.apache.org/jira/browse/OAK-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917494#comment-16917494
 ] 

Julian Reschke commented on OAK-8578:
-

trunk: [r1866023|http://svn.apache.org/r1866023] 
[r1866022|http://svn.apache.org/r1866022] 
[r1865974|http://svn.apache.org/r1865974] 
[r1865962|http://svn.apache.org/r1865962]

> Introduce API to check whether blob inlined in Id
> -
>
> Key: OAK-8578
> URL: https://issues.apache.org/jira/browse/OAK-8578
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob-cloud, blob-cloud-azure
>Reporter: Amit Jain
>Assignee: Amit Jain
>Priority: Major
> Fix For: 1.18.0
>
>
> New API in blob to check whether blob inlined in Id. This will replace calls 
> to getReference which was used as a proxy to check whether blob inlined and 
> avoid remote calls to the backend (potentially expensive ones over cloud).



--
This message was sent by Atlassian Jira
(v8.3.2#803003)