[jira] [Created] (OAK-7996) Ability to disable automatic text extraction via configuration
Matt Ryan created OAK-7996: -- Summary: Ability to disable automatic text extraction via configuration Key: OAK-7996 URL: https://issues.apache.org/jira/browse/OAK-7996 Project: Jackrabbit Oak Issue Type: Improvement Reporter: Matt Ryan Assignee: Matt Ryan This issue is to discuss allowing a user to disable automatic text extraction of binary data via a configuration file. Currently you can save a tika.config file inside an index definition, which overrides the default Tika configuration for that index. You can use this approach to disable automatic text extraction. I'd like to be able to do this at a global level - not per-index - via a configuration file instead. Then inside the document maker code somewhere, we would check to see whether the candidate for text extraction has been disabled by configuration. The value in this approach is that two instances can be identical in terms of index definitions, only differing in local configuration. Separate index definitions don't have to be maintained. And if you want to change which files you extract text, you don't have to refresh an index to make it happen. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7702) [Direct Binary Access] Add CDN support, use S3 Transfer Acceleration only for uploads
[ https://issues.apache.org/jira/browse/OAK-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Klimetschek updated OAK-7702: --- Description: Azure Blob Store and S3 support CDNs in front of private containers/buckets, which also work with presigned URLs ([S3 docs|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html], [cloudfront presigning java|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/CFPrivateDistJavaDevelopment.html], [Azure docs|https://docs.microsoft.com/en-us/azure/cdn/cdn-sas-storage-support]). The binary access feature should support these for download URLs via a configuration on the DataStore. Currently, the S3 datastore has a config for [transfer acceleration|https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html] that if enabled, will make both upload & download URLs use the acceleration option. However, this feature only makes sense for uploads, especially if CDN is an option for downloads. was: Azure Blob Store and S3 support CDNs in front of private containers/buckets, which also work with presigned URLs ([S3 docs|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html], [Azure docs|https://docs.microsoft.com/en-us/azure/cdn/cdn-sas-storage-support]). The binary access feature should support these for download URLs via a configuration on the DataStore. Currently, the S3 datastore has a config for [transfer acceleration|https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html] that if enabled, will make both upload & download URLs use the acceleration option. However, this feature only makes sense for uploads, especially if CDN is an option for downloads. > [Direct Binary Access] Add CDN support, use S3 Transfer Acceleration only for > uploads > - > > Key: OAK-7702 > URL: https://issues.apache.org/jira/browse/OAK-7702 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob-cloud, blob-cloud-azure >Reporter: Alexander Klimetschek >Priority: Major > > Azure Blob Store and S3 support CDNs in front of private containers/buckets, > which also work with presigned URLs ([S3 > docs|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html], > [cloudfront presigning > java|https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/CFPrivateDistJavaDevelopment.html], > [Azure > docs|https://docs.microsoft.com/en-us/azure/cdn/cdn-sas-storage-support]). > The binary access feature should support these for download URLs via a > configuration on the DataStore. > Currently, the S3 datastore has a config for [transfer > acceleration|https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html] > that if enabled, will make both upload & download URLs use the acceleration > option. However, this feature only makes sense for uploads, especially if CDN > is an option for downloads. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7989) Build Jackrabbit Oak #1891 failed
[ https://issues.apache.org/jira/browse/OAK-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746524#comment-16746524 ] Hudson commented on OAK-7989: - Previously failing build now is OK. Passed run: [Jackrabbit Oak #1899|https://builds.apache.org/job/Jackrabbit%20Oak/1899/] [console log|https://builds.apache.org/job/Jackrabbit%20Oak/1899/console] > Build Jackrabbit Oak #1891 failed > - > > Key: OAK-7989 > URL: https://issues.apache.org/jira/browse/OAK-7989 > Project: Jackrabbit Oak > Issue Type: Bug > Components: continuous integration >Reporter: Hudson >Priority: Major > > No description is provided > The build Jackrabbit Oak #1891 has failed. > First failed run: [Jackrabbit Oak > #1891|https://builds.apache.org/job/Jackrabbit%20Oak/1891/] [console > log|https://builds.apache.org/job/Jackrabbit%20Oak/1891/console] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6749) Segment-Tar standby sync fails with "in-memory" blobs present in the source repo
[
https://issues.apache.org/jira/browse/OAK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746449#comment-16746449
]
Csaba Varga commented on OAK-6749:
--
Thank you for the fix! The bug doesn't affect me anymore (I worked around it
shortly after reporting it), but I still appreciate that it's getting fixed.
> Segment-Tar standby sync fails with "in-memory" blobs present in the source
> repo
>
>
> Key: OAK-6749
> URL: https://issues.apache.org/jira/browse/OAK-6749
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob, tarmk-standby
>Affects Versions: 1.6.2
>Reporter: Csaba Varga
>Assignee: Francesco Mari
>Priority: Major
> Fix For: 1.10.1, 1.10
>
> Attachments: OAK-6749-01.patch, OAK-6749-02.patch
>
>
> We have run into some issue when trying to transition from an active/active
> Mongo NodeStore cluster to a single Segment-Tar server with cold standby. The
> issue itself manifests when the standby server tries to pull changes from the
> primary after the first round of online revision GC.
> Let me summarize the way we ended up with the current state, and my
> hypothesis about what happened, based on my debugging so far:
> # We started with a Mongo NodeStore and an external FileDataStore as the blob
> store. The FileDataStore was set up with minRecordLength=4096. The Mongo
> store stores blobs below minRecordLength as special "in-memory" blobIDs where
> the data itself is baked into the ID string in hex.
> # We have executed a sidegrade of the Mongo store into a Segment-Tar store.
> Our datastore is over 1TB in size, so copying the binaries wasn't an option.
> The new repository is simply reusing the existing datastore. The "in-memory"
> blobIDs still look like external blobIDs to the sidegrade process, so they
> were copied into the Segment-Tar repository as-is, instead of being converted
> into the efficient in-line format.
> # The server started up without issues on the new Segment-Tar store. The
> migrated "in-memory" blob IDs seem to work fine, if a bit sub-optimal.
> # At this point, we have created a cold standby instance by copying the files
> of the stopped primary instance and making the necessary config changes on
> both servers.
> # Everything worked fine until the primary server started its first round of
> online revision GC. After that process completed, the standby node started
> throwing exceptions about missing segments, and eventually stopped
> altogether. In the meantime, the following warning showed up in the primary
> log:
> {code:java}
> 29.09.2017 06:12:08.088 *WARN* [nioEventLoopGroup-3-10]
> org.apache.jackrabbit.oak.segment.standby.server.ExceptionHandler Exception
> caught on the server
> io.netty.handler.codec.TooLongFrameException: frame length (8208) exceeds the
> allowed maximum (8192)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:146)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:142)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:99)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:75)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
> at
>
[jira] [Commented] (OAK-7989) Build Jackrabbit Oak #1891 failed
[ https://issues.apache.org/jira/browse/OAK-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746420#comment-16746420 ] Hudson commented on OAK-7989: - Previously failing build now is OK. Passed run: [Jackrabbit Oak #1898|https://builds.apache.org/job/Jackrabbit%20Oak/1898/] [console log|https://builds.apache.org/job/Jackrabbit%20Oak/1898/console] > Build Jackrabbit Oak #1891 failed > - > > Key: OAK-7989 > URL: https://issues.apache.org/jira/browse/OAK-7989 > Project: Jackrabbit Oak > Issue Type: Bug > Components: continuous integration >Reporter: Hudson >Priority: Major > > No description is provided > The build Jackrabbit Oak #1891 has failed. > First failed run: [Jackrabbit Oak > #1891|https://builds.apache.org/job/Jackrabbit%20Oak/1891/] [console > log|https://builds.apache.org/job/Jackrabbit%20Oak/1891/console] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7904) Exporting query duration per index metrics with Sling Metrics / DropWizard
[
https://issues.apache.org/jira/browse/OAK-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746402#comment-16746402
]
Thomas Mueller commented on OAK-7904:
-
* The index name is sometimes not useful ("property", which is the type... or
sometimes even null). To not break compatibility, it might be easiest to add a
new method to QueryIndex and / or IndexPlan.
> Exporting query duration per index metrics with Sling Metrics / DropWizard
> --
>
> Key: OAK-7904
> URL: https://issues.apache.org/jira/browse/OAK-7904
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: indexing, query
>Reporter: Paul Chibulcuteanu
>Assignee: Paul Chibulcuteanu
>Priority: Major
> Fix For: 1.9.10, 1.12
>
> Attachments: OAK-7904.0.patch, OAK-7904.1.patch
>
>
> Purpose of this task is to evaluate & create metric which calculates the
> average duration of query for each index.
> This metric can be later used to evaluate which index(s) need to be optimised.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7904) Exporting query duration per index metrics with Sling Metrics / DropWizard
[
https://issues.apache.org/jira/browse/OAK-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746398#comment-16746398
]
Thomas Mueller commented on OAK-7904:
-
A few comments:
* The patch only measures the execute() method, not the next() method. next()
is called during iteration.
* We already measure timing data for queries, but not for indexes, in the
_caller_ of SelectorImpl.execute() and next(). That is:
QueryImpl.RowIterator.fetchNext(). See
{noformat}
long nanos = System.nanoTime();
...
nanos = System.nanoTime() - nanos;
stats.read(rowIndex - oldIndex, rowIndex, nanos);
{noformat}
* As this is called a lot, it's relatively important that it's fast. So I would
for example avoid String concatenation if possible, in "QUERY_DURATION_" +
indexName. Instead, the String could be cached, or maybe even the timer could
be re-used (not sure) and just times could be increased as done in
stats.read(...)
* TimeUnit.NANOSECONDS.toMillis(context.stop()); is not really needed, it's
sufficient to call context.stop()
> Exporting query duration per index metrics with Sling Metrics / DropWizard
> --
>
> Key: OAK-7904
> URL: https://issues.apache.org/jira/browse/OAK-7904
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: indexing, query
>Reporter: Paul Chibulcuteanu
>Assignee: Paul Chibulcuteanu
>Priority: Major
> Fix For: 1.9.10, 1.12
>
> Attachments: OAK-7904.0.patch, OAK-7904.1.patch
>
>
> Purpose of this task is to evaluate & create metric which calculates the
> average duration of query for each index.
> This metric can be later used to evaluate which index(s) need to be optimised.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OAK-7993) CompositeAuthorizationConfiguration.getRestrictionProvider() should filter duplications
[
https://issues.apache.org/jira/browse/OAK-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angela resolved OAK-7993.
-
Resolution: Fixed
Committed revision 1851625.
> CompositeAuthorizationConfiguration.getRestrictionProvider() should filter
> duplications
> ---
>
> Key: OAK-7993
> URL: https://issues.apache.org/jira/browse/OAK-7993
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, security
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Fix For: 1.12, 1.11.0
>
> Attachments: OAK-7993.patch
>
>
> The current implementation of
> {{CompositeAuthorizationConfiguration.getRestrictionProvider()}} creates a
> list populated with all non-empty providers returned by the aggregated
> authorization modules. This may lead to duplication of providers in case
> multiple modules reference the same provider.
> cc: [~stillalex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7991) Composite node store: tests with queries
[ https://issues.apache.org/jira/browse/OAK-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746346#comment-16746346 ] Thomas Mueller commented on OAK-7991: - New patch ...-b > Composite node store: tests with queries > > > Key: OAK-7991 > URL: https://issues.apache.org/jira/browse/OAK-7991 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: composite, indexing >Reporter: Thomas Mueller >Priority: Major > Attachments: OAK-7991-a.patch, OAK-7991-b.patch > > > Right now there seem to be no tests that run queries and use the composite > node store. This is dangerous, as a lot of code is not tested. Also, it > prevents refactoring and adding new features. > We have seen quite many problems in this area lately, for example: > * creating indexes with a composite nodestore didn't work > * changing the Lucene index to lazily load the index files doesn't work -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7995) Composite node store: reference index not working as expected
[ https://issues.apache.org/jira/browse/OAK-7995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746344#comment-16746344 ] Thomas Mueller commented on OAK-7995: - Test case see OAK-7991 (OAK-7991-b.patch) > Composite node store: reference index not working as expected > - > > Key: OAK-7995 > URL: https://issues.apache.org/jira/browse/OAK-7995 > Project: Jackrabbit Oak > Issue Type: Bug > Components: composite, indexing >Reporter: Thomas Mueller >Priority: Major > > Queries in the global store don't find indexed nodes in the read-only store. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OAK-7992) Composite node store: support the reference index
[ https://issues.apache.org/jira/browse/OAK-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller resolved OAK-7992. - Resolution: Fixed Actually, it already implemented... but buggy (see OAK-7995) > Composite node store: support the reference index > - > > Key: OAK-7992 > URL: https://issues.apache.org/jira/browse/OAK-7992 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: composite, indexing >Reporter: Thomas Mueller >Priority: Major > > The reference index currently doesn't support the composite node store. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7991) Composite node store: tests with queries
[ https://issues.apache.org/jira/browse/OAK-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller updated OAK-7991: Attachment: OAK-7991-b.patch > Composite node store: tests with queries > > > Key: OAK-7991 > URL: https://issues.apache.org/jira/browse/OAK-7991 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: composite, indexing >Reporter: Thomas Mueller >Priority: Major > Attachments: OAK-7991-a.patch, OAK-7991-b.patch > > > Right now there seem to be no tests that run queries and use the composite > node store. This is dangerous, as a lot of code is not tested. Also, it > prevents refactoring and adding new features. > We have seen quite many problems in this area lately, for example: > * creating indexes with a composite nodestore didn't work > * changing the Lucene index to lazily load the index files doesn't work -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-6749) Segment-Tar standby sync fails with "in-memory" blobs present in the source repo
[
https://issues.apache.org/jira/browse/OAK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francesco Mari updated OAK-6749:
Fix Version/s: 1.10
> Segment-Tar standby sync fails with "in-memory" blobs present in the source
> repo
>
>
> Key: OAK-6749
> URL: https://issues.apache.org/jira/browse/OAK-6749
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob, tarmk-standby
>Affects Versions: 1.6.2
>Reporter: Csaba Varga
>Assignee: Francesco Mari
>Priority: Major
> Fix For: 1.10.1, 1.10
>
> Attachments: OAK-6749-01.patch, OAK-6749-02.patch
>
>
> We have run into some issue when trying to transition from an active/active
> Mongo NodeStore cluster to a single Segment-Tar server with cold standby. The
> issue itself manifests when the standby server tries to pull changes from the
> primary after the first round of online revision GC.
> Let me summarize the way we ended up with the current state, and my
> hypothesis about what happened, based on my debugging so far:
> # We started with a Mongo NodeStore and an external FileDataStore as the blob
> store. The FileDataStore was set up with minRecordLength=4096. The Mongo
> store stores blobs below minRecordLength as special "in-memory" blobIDs where
> the data itself is baked into the ID string in hex.
> # We have executed a sidegrade of the Mongo store into a Segment-Tar store.
> Our datastore is over 1TB in size, so copying the binaries wasn't an option.
> The new repository is simply reusing the existing datastore. The "in-memory"
> blobIDs still look like external blobIDs to the sidegrade process, so they
> were copied into the Segment-Tar repository as-is, instead of being converted
> into the efficient in-line format.
> # The server started up without issues on the new Segment-Tar store. The
> migrated "in-memory" blob IDs seem to work fine, if a bit sub-optimal.
> # At this point, we have created a cold standby instance by copying the files
> of the stopped primary instance and making the necessary config changes on
> both servers.
> # Everything worked fine until the primary server started its first round of
> online revision GC. After that process completed, the standby node started
> throwing exceptions about missing segments, and eventually stopped
> altogether. In the meantime, the following warning showed up in the primary
> log:
> {code:java}
> 29.09.2017 06:12:08.088 *WARN* [nioEventLoopGroup-3-10]
> org.apache.jackrabbit.oak.segment.standby.server.ExceptionHandler Exception
> caught on the server
> io.netty.handler.codec.TooLongFrameException: frame length (8208) exceeds the
> allowed maximum (8192)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:146)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:142)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:99)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:75)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at
>
[jira] [Updated] (OAK-6749) Segment-Tar standby sync fails with "in-memory" blobs present in the source repo
[
https://issues.apache.org/jira/browse/OAK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francesco Mari updated OAK-6749:
Fix Version/s: 1.10.1
> Segment-Tar standby sync fails with "in-memory" blobs present in the source
> repo
>
>
> Key: OAK-6749
> URL: https://issues.apache.org/jira/browse/OAK-6749
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob, tarmk-standby
>Affects Versions: 1.6.2
>Reporter: Csaba Varga
>Assignee: Francesco Mari
>Priority: Major
> Fix For: 1.10.1
>
> Attachments: OAK-6749-01.patch, OAK-6749-02.patch
>
>
> We have run into some issue when trying to transition from an active/active
> Mongo NodeStore cluster to a single Segment-Tar server with cold standby. The
> issue itself manifests when the standby server tries to pull changes from the
> primary after the first round of online revision GC.
> Let me summarize the way we ended up with the current state, and my
> hypothesis about what happened, based on my debugging so far:
> # We started with a Mongo NodeStore and an external FileDataStore as the blob
> store. The FileDataStore was set up with minRecordLength=4096. The Mongo
> store stores blobs below minRecordLength as special "in-memory" blobIDs where
> the data itself is baked into the ID string in hex.
> # We have executed a sidegrade of the Mongo store into a Segment-Tar store.
> Our datastore is over 1TB in size, so copying the binaries wasn't an option.
> The new repository is simply reusing the existing datastore. The "in-memory"
> blobIDs still look like external blobIDs to the sidegrade process, so they
> were copied into the Segment-Tar repository as-is, instead of being converted
> into the efficient in-line format.
> # The server started up without issues on the new Segment-Tar store. The
> migrated "in-memory" blob IDs seem to work fine, if a bit sub-optimal.
> # At this point, we have created a cold standby instance by copying the files
> of the stopped primary instance and making the necessary config changes on
> both servers.
> # Everything worked fine until the primary server started its first round of
> online revision GC. After that process completed, the standby node started
> throwing exceptions about missing segments, and eventually stopped
> altogether. In the meantime, the following warning showed up in the primary
> log:
> {code:java}
> 29.09.2017 06:12:08.088 *WARN* [nioEventLoopGroup-3-10]
> org.apache.jackrabbit.oak.segment.standby.server.ExceptionHandler Exception
> caught on the server
> io.netty.handler.codec.TooLongFrameException: frame length (8208) exceeds the
> allowed maximum (8192)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:146)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:142)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:99)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:75)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at
>
[jira] [Commented] (OAK-6749) Segment-Tar standby sync fails with "in-memory" blobs present in the source repo
[
https://issues.apache.org/jira/browse/OAK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746313#comment-16746313
]
Francesco Mari commented on OAK-6749:
-
I fixed the issue at r1851619. I'm going to backport the relevant commits up to
1.6.
> Segment-Tar standby sync fails with "in-memory" blobs present in the source
> repo
>
>
> Key: OAK-6749
> URL: https://issues.apache.org/jira/browse/OAK-6749
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob, tarmk-standby
>Affects Versions: 1.6.2
>Reporter: Csaba Varga
>Assignee: Francesco Mari
>Priority: Major
> Attachments: OAK-6749-01.patch, OAK-6749-02.patch
>
>
> We have run into some issue when trying to transition from an active/active
> Mongo NodeStore cluster to a single Segment-Tar server with cold standby. The
> issue itself manifests when the standby server tries to pull changes from the
> primary after the first round of online revision GC.
> Let me summarize the way we ended up with the current state, and my
> hypothesis about what happened, based on my debugging so far:
> # We started with a Mongo NodeStore and an external FileDataStore as the blob
> store. The FileDataStore was set up with minRecordLength=4096. The Mongo
> store stores blobs below minRecordLength as special "in-memory" blobIDs where
> the data itself is baked into the ID string in hex.
> # We have executed a sidegrade of the Mongo store into a Segment-Tar store.
> Our datastore is over 1TB in size, so copying the binaries wasn't an option.
> The new repository is simply reusing the existing datastore. The "in-memory"
> blobIDs still look like external blobIDs to the sidegrade process, so they
> were copied into the Segment-Tar repository as-is, instead of being converted
> into the efficient in-line format.
> # The server started up without issues on the new Segment-Tar store. The
> migrated "in-memory" blob IDs seem to work fine, if a bit sub-optimal.
> # At this point, we have created a cold standby instance by copying the files
> of the stopped primary instance and making the necessary config changes on
> both servers.
> # Everything worked fine until the primary server started its first round of
> online revision GC. After that process completed, the standby node started
> throwing exceptions about missing segments, and eventually stopped
> altogether. In the meantime, the following warning showed up in the primary
> log:
> {code:java}
> 29.09.2017 06:12:08.088 *WARN* [nioEventLoopGroup-3-10]
> org.apache.jackrabbit.oak.segment.standby.server.ExceptionHandler Exception
> caught on the server
> io.netty.handler.codec.TooLongFrameException: frame length (8208) exceeds the
> allowed maximum (8192)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:146)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:142)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:99)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:75)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at
>
[jira] [Commented] (OAK-7904) Exporting query duration per index metrics with Sling Metrics / DropWizard
[ https://issues.apache.org/jira/browse/OAK-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746302#comment-16746302 ] Paul Chibulcuteanu commented on OAK-7904: - [~tmueller], I've added [^OAK-7904.1.patch] which for now only measures the execution time. > Exporting query duration per index metrics with Sling Metrics / DropWizard > -- > > Key: OAK-7904 > URL: https://issues.apache.org/jira/browse/OAK-7904 > Project: Jackrabbit Oak > Issue Type: Task > Components: indexing, query >Reporter: Paul Chibulcuteanu >Assignee: Paul Chibulcuteanu >Priority: Major > Fix For: 1.9.10, 1.12 > > Attachments: OAK-7904.0.patch, OAK-7904.1.patch > > > Purpose of this task is to evaluate & create metric which calculates the > average duration of query for each index. > This metric can be later used to evaluate which index(s) need to be optimised. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7904) Exporting query duration per index metrics with Sling Metrics / DropWizard
[ https://issues.apache.org/jira/browse/OAK-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Chibulcuteanu updated OAK-7904: Attachment: OAK-7904.1.patch > Exporting query duration per index metrics with Sling Metrics / DropWizard > -- > > Key: OAK-7904 > URL: https://issues.apache.org/jira/browse/OAK-7904 > Project: Jackrabbit Oak > Issue Type: Task > Components: indexing, query >Reporter: Paul Chibulcuteanu >Assignee: Paul Chibulcuteanu >Priority: Major > Fix For: 1.9.10, 1.12 > > Attachments: OAK-7904.0.patch, OAK-7904.1.patch > > > Purpose of this task is to evaluate & create metric which calculates the > average duration of query for each index. > This metric can be later used to evaluate which index(s) need to be optimised. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OAK-7995) Composite node store: reference index not working as expected
Thomas Mueller created OAK-7995: --- Summary: Composite node store: reference index not working as expected Key: OAK-7995 URL: https://issues.apache.org/jira/browse/OAK-7995 Project: Jackrabbit Oak Issue Type: Bug Components: composite, indexing Reporter: Thomas Mueller Queries in the global store don't find indexed nodes in the read-only store. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7975) Facet extraction fails while requesting multiple facets and one of the requested facets doesn't have indexed values
[
https://issues.apache.org/jira/browse/OAK-7975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746169#comment-16746169
]
Davide Giannella commented on OAK-7975:
---
bq. is there some way to fix release notes for 1.6.16 itself?
while we could update the tag in svn I don't like the idea for this minor
things. Additionally it will create discrepancies between the announcements
sent along the public release and the actual release notes we'd have in tag.
I'd say that keeping both versions is a good one along with your comment.
What you could do more is to broadcast the situation on oak-dev just to make
sure that the community knows about this aspect.
> Facet extraction fails while requesting multiple facets and one of the
> requested facets doesn't have indexed values
> ---
>
> Key: OAK-7975
> URL: https://issues.apache.org/jira/browse/OAK-7975
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.10.0, 1.6.16, 1.8.11, 1.6.17, 1.11.0
>
>
> Consider a content like
> {noformat}
> + /test
>- foo=bar
> {noformat}
> with index def faceting multiple properties - something like
> {noformat}
> + /oak:index/foo/indexRules/nt:base/properties
>+ foo
> - propertyIndex=true
> - facets = true
> + bar
> - facets = true
> {noformat}
> Then a query like
> {noformat}
> SELECT [rep:facet(foo)], [rep:facete(bar)] FROM [nt:base]
> {noformat}
> should not fail.
> Note that the failure requires requesting on multiple facets and one of them
> must not have any indexed value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-6749) Segment-Tar standby sync fails with "in-memory" blobs present in the source repo
[
https://issues.apache.org/jira/browse/OAK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746153#comment-16746153
]
Andrei Dulceanu commented on OAK-6749:
--
[~frm], the previous commits and the patch LGTM.
> Segment-Tar standby sync fails with "in-memory" blobs present in the source
> repo
>
>
> Key: OAK-6749
> URL: https://issues.apache.org/jira/browse/OAK-6749
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob, tarmk-standby
>Affects Versions: 1.6.2
>Reporter: Csaba Varga
>Assignee: Francesco Mari
>Priority: Major
> Attachments: OAK-6749-01.patch, OAK-6749-02.patch
>
>
> We have run into some issue when trying to transition from an active/active
> Mongo NodeStore cluster to a single Segment-Tar server with cold standby. The
> issue itself manifests when the standby server tries to pull changes from the
> primary after the first round of online revision GC.
> Let me summarize the way we ended up with the current state, and my
> hypothesis about what happened, based on my debugging so far:
> # We started with a Mongo NodeStore and an external FileDataStore as the blob
> store. The FileDataStore was set up with minRecordLength=4096. The Mongo
> store stores blobs below minRecordLength as special "in-memory" blobIDs where
> the data itself is baked into the ID string in hex.
> # We have executed a sidegrade of the Mongo store into a Segment-Tar store.
> Our datastore is over 1TB in size, so copying the binaries wasn't an option.
> The new repository is simply reusing the existing datastore. The "in-memory"
> blobIDs still look like external blobIDs to the sidegrade process, so they
> were copied into the Segment-Tar repository as-is, instead of being converted
> into the efficient in-line format.
> # The server started up without issues on the new Segment-Tar store. The
> migrated "in-memory" blob IDs seem to work fine, if a bit sub-optimal.
> # At this point, we have created a cold standby instance by copying the files
> of the stopped primary instance and making the necessary config changes on
> both servers.
> # Everything worked fine until the primary server started its first round of
> online revision GC. After that process completed, the standby node started
> throwing exceptions about missing segments, and eventually stopped
> altogether. In the meantime, the following warning showed up in the primary
> log:
> {code:java}
> 29.09.2017 06:12:08.088 *WARN* [nioEventLoopGroup-3-10]
> org.apache.jackrabbit.oak.segment.standby.server.ExceptionHandler Exception
> caught on the server
> io.netty.handler.codec.TooLongFrameException: frame length (8208) exceeds the
> allowed maximum (8192)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:146)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:142)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:99)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:75)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at
>
[jira] [Commented] (OAK-6749) Segment-Tar standby sync fails with "in-memory" blobs present in the source repo
[
https://issues.apache.org/jira/browse/OAK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746129#comment-16746129
]
Francesco Mari commented on OAK-6749:
-
I incorporated [~tmueller]'s suggestions - both the one given here and other
feedback provided offline - into the second version of the patch. The method
{{shouldFetchBinary}} is definitely longer than the previous implementation,
but it's well commented and I think it conveys a better description of the
shortcuts and the worst case involved in such a check.
> Segment-Tar standby sync fails with "in-memory" blobs present in the source
> repo
>
>
> Key: OAK-6749
> URL: https://issues.apache.org/jira/browse/OAK-6749
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob, tarmk-standby
>Affects Versions: 1.6.2
>Reporter: Csaba Varga
>Assignee: Francesco Mari
>Priority: Major
> Attachments: OAK-6749-01.patch, OAK-6749-02.patch
>
>
> We have run into some issue when trying to transition from an active/active
> Mongo NodeStore cluster to a single Segment-Tar server with cold standby. The
> issue itself manifests when the standby server tries to pull changes from the
> primary after the first round of online revision GC.
> Let me summarize the way we ended up with the current state, and my
> hypothesis about what happened, based on my debugging so far:
> # We started with a Mongo NodeStore and an external FileDataStore as the blob
> store. The FileDataStore was set up with minRecordLength=4096. The Mongo
> store stores blobs below minRecordLength as special "in-memory" blobIDs where
> the data itself is baked into the ID string in hex.
> # We have executed a sidegrade of the Mongo store into a Segment-Tar store.
> Our datastore is over 1TB in size, so copying the binaries wasn't an option.
> The new repository is simply reusing the existing datastore. The "in-memory"
> blobIDs still look like external blobIDs to the sidegrade process, so they
> were copied into the Segment-Tar repository as-is, instead of being converted
> into the efficient in-line format.
> # The server started up without issues on the new Segment-Tar store. The
> migrated "in-memory" blob IDs seem to work fine, if a bit sub-optimal.
> # At this point, we have created a cold standby instance by copying the files
> of the stopped primary instance and making the necessary config changes on
> both servers.
> # Everything worked fine until the primary server started its first round of
> online revision GC. After that process completed, the standby node started
> throwing exceptions about missing segments, and eventually stopped
> altogether. In the meantime, the following warning showed up in the primary
> log:
> {code:java}
> 29.09.2017 06:12:08.088 *WARN* [nioEventLoopGroup-3-10]
> org.apache.jackrabbit.oak.segment.standby.server.ExceptionHandler Exception
> caught on the server
> io.netty.handler.codec.TooLongFrameException: frame length (8208) exceeds the
> allowed maximum (8192)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:146)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:142)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:99)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:75)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
>
[jira] [Updated] (OAK-6749) Segment-Tar standby sync fails with "in-memory" blobs present in the source repo
[
https://issues.apache.org/jira/browse/OAK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francesco Mari updated OAK-6749:
Attachment: OAK-6749-02.patch
> Segment-Tar standby sync fails with "in-memory" blobs present in the source
> repo
>
>
> Key: OAK-6749
> URL: https://issues.apache.org/jira/browse/OAK-6749
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob, tarmk-standby
>Affects Versions: 1.6.2
>Reporter: Csaba Varga
>Assignee: Francesco Mari
>Priority: Major
> Attachments: OAK-6749-01.patch, OAK-6749-02.patch
>
>
> We have run into some issue when trying to transition from an active/active
> Mongo NodeStore cluster to a single Segment-Tar server with cold standby. The
> issue itself manifests when the standby server tries to pull changes from the
> primary after the first round of online revision GC.
> Let me summarize the way we ended up with the current state, and my
> hypothesis about what happened, based on my debugging so far:
> # We started with a Mongo NodeStore and an external FileDataStore as the blob
> store. The FileDataStore was set up with minRecordLength=4096. The Mongo
> store stores blobs below minRecordLength as special "in-memory" blobIDs where
> the data itself is baked into the ID string in hex.
> # We have executed a sidegrade of the Mongo store into a Segment-Tar store.
> Our datastore is over 1TB in size, so copying the binaries wasn't an option.
> The new repository is simply reusing the existing datastore. The "in-memory"
> blobIDs still look like external blobIDs to the sidegrade process, so they
> were copied into the Segment-Tar repository as-is, instead of being converted
> into the efficient in-line format.
> # The server started up without issues on the new Segment-Tar store. The
> migrated "in-memory" blob IDs seem to work fine, if a bit sub-optimal.
> # At this point, we have created a cold standby instance by copying the files
> of the stopped primary instance and making the necessary config changes on
> both servers.
> # Everything worked fine until the primary server started its first round of
> online revision GC. After that process completed, the standby node started
> throwing exceptions about missing segments, and eventually stopped
> altogether. In the meantime, the following warning showed up in the primary
> log:
> {code:java}
> 29.09.2017 06:12:08.088 *WARN* [nioEventLoopGroup-3-10]
> org.apache.jackrabbit.oak.segment.standby.server.ExceptionHandler Exception
> caught on the server
> io.netty.handler.codec.TooLongFrameException: frame length (8208) exceeds the
> allowed maximum (8192)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:146)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:142)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:99)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:75)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:611)
> at
>
[jira] [Commented] (OAK-7989) Build Jackrabbit Oak #1891 failed
[ https://issues.apache.org/jira/browse/OAK-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746099#comment-16746099 ] Hudson commented on OAK-7989: - Previously failing build now is OK. Passed run: [Jackrabbit Oak #1897|https://builds.apache.org/job/Jackrabbit%20Oak/1897/] [console log|https://builds.apache.org/job/Jackrabbit%20Oak/1897/console] > Build Jackrabbit Oak #1891 failed > - > > Key: OAK-7989 > URL: https://issues.apache.org/jira/browse/OAK-7989 > Project: Jackrabbit Oak > Issue Type: Bug > Components: continuous integration >Reporter: Hudson >Priority: Major > > No description is provided > The build Jackrabbit Oak #1891 has failed. > First failed run: [Jackrabbit Oak > #1891|https://builds.apache.org/job/Jackrabbit%20Oak/1891/] [console > log|https://builds.apache.org/job/Jackrabbit%20Oak/1891/console] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7993) CompositeAuthorizationConfiguration.getRestrictionProvider() should filter duplications
[
https://issues.apache.org/jira/browse/OAK-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746079#comment-16746079
]
Alex Deparvu commented on OAK-7993:
---
looks good.
> CompositeAuthorizationConfiguration.getRestrictionProvider() should filter
> duplications
> ---
>
> Key: OAK-7993
> URL: https://issues.apache.org/jira/browse/OAK-7993
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, security
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Fix For: 1.12, 1.11.0
>
> Attachments: OAK-7993.patch
>
>
> The current implementation of
> {{CompositeAuthorizationConfiguration.getRestrictionProvider()}} creates a
> list populated with all non-empty providers returned by the aggregated
> authorization modules. This may lead to duplication of providers in case
> multiple modules reference the same provider.
> cc: [~stillalex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7937) Implement CugAccessControlManager.getEffectivePolicies(Set principals)
[
https://issues.apache.org/jira/browse/OAK-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745927#comment-16745927
]
angela commented on OAK-7937:
-
[~stillalex] tried to find something about the immutability... what i found so
far is the requirement that policies passed to {{setPolicy/removePolicy}} must
be obtained through {{getPolicies}} or {{getApplicablePolicies}}. The problem
when effective policies get passed to setPolicy/removePolicy is that they don't
reflect the same transient state than the editing session.
> Implement CugAccessControlManager.getEffectivePolicies(Set
> principals)
> -
>
> Key: OAK-7937
> URL: https://issues.apache.org/jira/browse/OAK-7937
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: angela
>Assignee: Alex Deparvu
>Priority: Major
> Fix For: 1.12
>
>
> today CugAccessControlManager.getEffectivePolicies(Set principals)
> returns an empty array and has a comment stating that this is not implemented.
> having thought this through again, i think there was some benefit in having
> the implementation. as long as the given set of principal does NOT include
> everyone the return value should just include the CUG-policies that
> explicitly list any of principals. IF _everyone_ was part of the set, the
> return-value basically includes _all_ CUG-policies, because every CUG will
> deny read-access for everyone except for the principals explicitly listed in
> the CUG-policy... if we do the latter as lazy as possible it might still be
> doable even in a scenario, when there are tons of CUG-policies specified.
> [~stillalex], wdyt? do you want to take care of this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OAK-7994) Principal Management APIs don't allow for search pagination
Alex Deparvu created OAK-7994: - Summary: Principal Management APIs don't allow for search pagination Key: OAK-7994 URL: https://issues.apache.org/jira/browse/OAK-7994 Project: Jackrabbit Oak Issue Type: Improvement Components: security, security-spi Reporter: Alex Deparvu Assignee: Alex Deparvu Current apis are limited to passing a search token, I'd like to also add range (offset, limit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7937) Implement CugAccessControlManager.getEffectivePolicies(Set principals)
[ https://issues.apache.org/jira/browse/OAK-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Deparvu updated OAK-7937: -- Component/s: security authorization-cug > Implement CugAccessControlManager.getEffectivePolicies(Set > principals) > - > > Key: OAK-7937 > URL: https://issues.apache.org/jira/browse/OAK-7937 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: authorization-cug, security >Reporter: angela >Assignee: Alex Deparvu >Priority: Major > Fix For: 1.12 > > > today CugAccessControlManager.getEffectivePolicies(Set principals) > returns an empty array and has a comment stating that this is not implemented. > having thought this through again, i think there was some benefit in having > the implementation. as long as the given set of principal does NOT include > everyone the return value should just include the CUG-policies that > explicitly list any of principals. IF _everyone_ was part of the set, the > return-value basically includes _all_ CUG-policies, because every CUG will > deny read-access for everyone except for the principals explicitly listed in > the CUG-policy... if we do the latter as lazy as possible it might still be > doable even in a scenario, when there are tons of CUG-policies specified. > [~stillalex], wdyt? do you want to take care of this? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7937) Implement CugAccessControlManager.getEffectivePolicies(Set principals)
[ https://issues.apache.org/jira/browse/OAK-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745923#comment-16745923 ] angela commented on OAK-7937: - [~stillalex] fine with me... so my comments here. anyway just 3 of them: # query engine should be build from latestroot not from getRoot() in order to make sure you only find persisted cugs (nothing transient should be in the effective policies). as you had it in the first patch # i vaguely remember that the effective policies should be immutable. in other words, one should not be able to modify or pass them to setPolicy/removePolicy. let's verify that against the spec again and if true also fix it for the getEffectivePolicies(String path) as well # tests: i would love to have a bit more tests: * verify that effective policies don't return transient cugs * that also policies that are not located at the supported path directly are included * immutability (if i remember correctly) * ...and some benchmarks for performance otherwise looks good if we decide to go for the query-traversal approach but it would definitely be good to have some benchmarks. the benefit of the query is that it's automatically using the editing session... if we went for the look-at-hidden HIDDEN_NESTED_CUGS properties we would need to do so on the nodestate level (or the readonly-tree) and then read the relevant cug-trees again with the editing contentsession to make sure we have access rights respected. and here more details on the alterative option: as far as i remember the NestedCugHook maintains a hidden structure that points to the next cug down the hierarchy... that's needed for the permission evaluation to be able to now where the top cugs are located and if and where they contain nested cugs that cancel the effect of a given cug... that should allow us to specifically look at nodes that are known to contain cugs instead of traversing everything and look at all nodes that might potentically contains cugs (but most likely don't have one except for some very edge cases where the majority of nodes withing the supported paths contain cugs... which i would consider pretty unlikely and a problem with the content design). does that info help? avoiding introducing new index to avoid troubles: fine with me... but let's make sure we properly benchmark any of the traversing solutions we end up with (see above). if it turns out to be very inefficient we would offer a configuration option that allowed customers to trigger the index-creation. > Implement CugAccessControlManager.getEffectivePolicies(Set > principals) > - > > Key: OAK-7937 > URL: https://issues.apache.org/jira/browse/OAK-7937 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: angela >Assignee: Alex Deparvu >Priority: Major > Fix For: 1.12 > > > today CugAccessControlManager.getEffectivePolicies(Set principals) > returns an empty array and has a comment stating that this is not implemented. > having thought this through again, i think there was some benefit in having > the implementation. as long as the given set of principal does NOT include > everyone the return value should just include the CUG-policies that > explicitly list any of principals. IF _everyone_ was part of the set, the > return-value basically includes _all_ CUG-policies, because every CUG will > deny read-access for everyone except for the principals explicitly listed in > the CUG-policy... if we do the latter as lazy as possible it might still be > doable even in a scenario, when there are tons of CUG-policies specified. > [~stillalex], wdyt? do you want to take care of this? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6749) Segment-Tar standby sync fails with "in-memory" blobs present in the source repo
[
https://issues.apache.org/jira/browse/OAK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745885#comment-16745885
]
Thomas Mueller commented on OAK-6749:
-
This patch most likely works. In theory, it might not work if the blob store
returns an input stream without verifying the file is there (opens the file
lazily). To ensure that can't happen, I suggest to read at least one byte, for
example something like this (needs to be tested):
{noformat}
...
+try {
+data = blobStore.getInputStream(blobId);
+data.read();
...
{noformat}
That should be more secure.
That said, it's probably a bit slow. It will open and close the file, if there
is one. I'm not sure if that could be a performance problem or not... If it is,
then I'm afraid I don't know what is the easiest way to speed it up. The
BlobStore API is rather limited.
> Segment-Tar standby sync fails with "in-memory" blobs present in the source
> repo
>
>
> Key: OAK-6749
> URL: https://issues.apache.org/jira/browse/OAK-6749
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: blob, tarmk-standby
>Affects Versions: 1.6.2
>Reporter: Csaba Varga
>Assignee: Francesco Mari
>Priority: Major
> Attachments: OAK-6749-01.patch
>
>
> We have run into some issue when trying to transition from an active/active
> Mongo NodeStore cluster to a single Segment-Tar server with cold standby. The
> issue itself manifests when the standby server tries to pull changes from the
> primary after the first round of online revision GC.
> Let me summarize the way we ended up with the current state, and my
> hypothesis about what happened, based on my debugging so far:
> # We started with a Mongo NodeStore and an external FileDataStore as the blob
> store. The FileDataStore was set up with minRecordLength=4096. The Mongo
> store stores blobs below minRecordLength as special "in-memory" blobIDs where
> the data itself is baked into the ID string in hex.
> # We have executed a sidegrade of the Mongo store into a Segment-Tar store.
> Our datastore is over 1TB in size, so copying the binaries wasn't an option.
> The new repository is simply reusing the existing datastore. The "in-memory"
> blobIDs still look like external blobIDs to the sidegrade process, so they
> were copied into the Segment-Tar repository as-is, instead of being converted
> into the efficient in-line format.
> # The server started up without issues on the new Segment-Tar store. The
> migrated "in-memory" blob IDs seem to work fine, if a bit sub-optimal.
> # At this point, we have created a cold standby instance by copying the files
> of the stopped primary instance and making the necessary config changes on
> both servers.
> # Everything worked fine until the primary server started its first round of
> online revision GC. After that process completed, the standby node started
> throwing exceptions about missing segments, and eventually stopped
> altogether. In the meantime, the following warning showed up in the primary
> log:
> {code:java}
> 29.09.2017 06:12:08.088 *WARN* [nioEventLoopGroup-3-10]
> org.apache.jackrabbit.oak.segment.standby.server.ExceptionHandler Exception
> caught on the server
> io.netty.handler.codec.TooLongFrameException: frame length (8208) exceeds the
> allowed maximum (8192)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:146)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.fail(LineBasedFrameDecoder.java:142)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:99)
> at
> io.netty.handler.codec.LineBasedFrameDecoder.decode(LineBasedFrameDecoder.java:75)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
>
[jira] [Updated] (OAK-7993) CompositeAuthorizationConfiguration.getRestrictionProvider() should filter duplications
[
https://issues.apache.org/jira/browse/OAK-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angela updated OAK-7993:
Fix Version/s: 1.11.0
1.12
> CompositeAuthorizationConfiguration.getRestrictionProvider() should filter
> duplications
> ---
>
> Key: OAK-7993
> URL: https://issues.apache.org/jira/browse/OAK-7993
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, security
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Fix For: 1.12, 1.11.0
>
> Attachments: OAK-7993.patch
>
>
> The current implementation of
> {{CompositeAuthorizationConfiguration.getRestrictionProvider()}} creates a
> list populated with all non-empty providers returned by the aggregated
> authorization modules. This may lead to duplication of providers in case
> multiple modules reference the same provider.
> cc: [~stillalex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-7993) CompositeAuthorizationConfiguration.getRestrictionProvider() should filter duplications
[
https://issues.apache.org/jira/browse/OAK-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angela updated OAK-7993:
Attachment: OAK-7993.patch
> CompositeAuthorizationConfiguration.getRestrictionProvider() should filter
> duplications
> ---
>
> Key: OAK-7993
> URL: https://issues.apache.org/jira/browse/OAK-7993
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, security
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Attachments: OAK-7993.patch
>
>
> The current implementation of
> {{CompositeAuthorizationConfiguration.getRestrictionProvider()}} creates a
> list populated with all non-empty providers returned by the aggregated
> authorization modules. This may lead to duplication of providers in case
> multiple modules reference the same provider.
> cc: [~stillalex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7993) CompositeAuthorizationConfiguration.getRestrictionProvider() should filter duplications
[
https://issues.apache.org/jira/browse/OAK-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745876#comment-16745876
]
angela commented on OAK-7993:
-
[~stillalex], proposed patch with tests illustrating the issue and the fix
attached. wdyt?
> CompositeAuthorizationConfiguration.getRestrictionProvider() should filter
> duplications
> ---
>
> Key: OAK-7993
> URL: https://issues.apache.org/jira/browse/OAK-7993
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, security
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Attachments: OAK-7993.patch
>
>
> The current implementation of
> {{CompositeAuthorizationConfiguration.getRestrictionProvider()}} creates a
> list populated with all non-empty providers returned by the aggregated
> authorization modules. This may lead to duplication of providers in case
> multiple modules reference the same provider.
> cc: [~stillalex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OAK-7993) CompositeAuthorizationConfiguration.getRestrictionProvider() should filter duplications
angela created OAK-7993:
---
Summary:
CompositeAuthorizationConfiguration.getRestrictionProvider() should filter
duplications
Key: OAK-7993
URL: https://issues.apache.org/jira/browse/OAK-7993
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, security
Reporter: angela
Assignee: angela
The current implementation of
{{CompositeAuthorizationConfiguration.getRestrictionProvider()}} creates a list
populated with all non-empty providers returned by the aggregated authorization
modules. This may lead to duplication of providers in case multiple modules
reference the same provider.
cc: [~stillalex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7937) Implement CugAccessControlManager.getEffectivePolicies(Set principals)
[ https://issues.apache.org/jira/browse/OAK-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745865#comment-16745865 ] Alex Deparvu commented on OAK-7937: --- bq. comment directly on github. I would prefer to keep the discussion here, like a single source of truth. bq. regarding risking re-index: agree that we should not create the index on existing installations. would be an option to create the index if the module is freshly installed (i.e. when the node types are being installed) but omit index creation if the node type is already there? I would not mess with index definitions out of fear or wrongly triggering a full reindex of the node type index. could we simply document the need to add this node type to the index? bq. an alternative approach to the traversal-ok flag (in case the index is missing or if we go without index altogether) might be to use the hidden structure to hop to nodes with cugs only instead of traversing everything... basically a manual search... wdyt? I don't follow, could you add more details to this option? > Implement CugAccessControlManager.getEffectivePolicies(Set > principals) > - > > Key: OAK-7937 > URL: https://issues.apache.org/jira/browse/OAK-7937 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: angela >Assignee: Alex Deparvu >Priority: Major > Fix For: 1.12 > > > today CugAccessControlManager.getEffectivePolicies(Set principals) > returns an empty array and has a comment stating that this is not implemented. > having thought this through again, i think there was some benefit in having > the implementation. as long as the given set of principal does NOT include > everyone the return value should just include the CUG-policies that > explicitly list any of principals. IF _everyone_ was part of the set, the > return-value basically includes _all_ CUG-policies, because every CUG will > deny read-access for everyone except for the principals explicitly listed in > the CUG-policy... if we do the latter as lazy as possible it might still be > doable even in a scenario, when there are tons of CUG-policies specified. > [~stillalex], wdyt? do you want to take care of this? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
