RE: Strategies around storing blobs in Mongo

2013-10-30 Thread Marcel Reutegger
Hi, Currently we are storing blobs by breaking them into small chunks and then storing those chunks in MongoDB as part of blobs collection. This approach would cause issues as Mongo maintains a global exclusive write locks on a per database level [1]. So even writing multiple small chunks of

Re: Strategies around storing blobs in Mongo

2013-10-30 Thread Thomas Mueller
Hi, as Mongo maintains a global exclusive write locks on a per database level I think this is not necessarily a huge problem. As far as I understand, it limits write concurrency within one shard only, so it does not block scalability. Open questions are, what is the write thoughput for one

Re: Strategies around storing blobs in Mongo

2013-10-30 Thread Ian Boston
On 30 October 2013 07:55, Thomas Mueller muel...@adobe.com wrote: Hi, as Mongo maintains a global exclusive write locks on a per database level I think this is not necessarily a huge problem. As far as I understand, it limits write concurrency within one shard only, so it does not block

Re: Strategies around storing blobs in Mongo

2013-10-30 Thread Chetan Mehrotra
Open questions are, what is the write thoughput for one shard, does the write lock also block reads (I guess not), does the write As Ian mentioned above write locks block all reads. So even adding a 2 MB chunk on a sharded system over remote connection would block read for that complete

Re: Strategies around storing blobs in Mongo

2013-10-30 Thread Chetan Mehrotra
sounds reasonable. what is the impact of such a design when it comes to map-reduce features? I was thinking that we could use it e.g. for garbage collection, but I don't know if this is still an option when data is spread across multiple databases. Would investigate that aspect further

Re: Strategies around storing blobs in Mongo

2013-10-30 Thread Amit Jain
So even adding a 2 MB chunk on a sharded system over remote connection would block read for that complete duration. So at minimum we should be avoiding that. I guess if there are read replicas in the shard replica set then, it will mitigate the effect to some extent On Wed, Oct 30, 2013 at

jackrabbit-oak build #2501: Broken

2013-10-30 Thread Travis CI
Build Update for apache/jackrabbit-oak - Build: #2501 Status: Broken Duration: 757 seconds Commit: 9647955a9108773e6ecfc4d33568f40a8f0d4e9b (trunk) Author: Michael Duerig Message: OAK-: Node#setProperty(String, Calendar) doesn't take time zone in account

jackrabbit-oak build #2502: Fixed

2013-10-30 Thread Travis CI
Build Update for apache/jackrabbit-oak - Build: #2502 Status: Fixed Duration: 1878 seconds Commit: 2f95b81fbbb914955e7bf8abf5ea9b2757745050 (trunk) Author: Marcel Reutegger Message: OAK-1129: Repeated MongoMK.rebase() always adds new revision git-svn-id:

Re: Strategies around storing blobs in Mongo

2013-10-30 Thread Michael Marth
Hi Chetan, 3. Bring back the JR2 DataStore implementation and just save metadata related to binaries in Mongo. We already have S3 based implementation there and they would continue to work with Oak also I think we will need the data store impl for Oak in any case (regardless the outcome

jackrabbit-oak build #2503: Fixed

2013-10-30 Thread Travis CI
Build Update for apache/jackrabbit-oak - Build: #2503 Status: Fixed Duration: 1899 seconds Commit: 9416e90f32bc1efa5971fa5ed33debaf19c20b4b (trunk) Author: Marcel Reutegger Message: OAK-1080: MongoMK: improved concurrency git-svn-id:

jackrabbit-oak build #2504: Fixed

2013-10-30 Thread Travis CI
Build Update for apache/jackrabbit-oak - Build: #2504 Status: Fixed Duration: 1978 seconds Commit: dd64305877e48d22e46b2f94eed6dcaf38065548 (trunk) Author: Thomas Mueller Message: OAK-1128 Conditions of the form (x=1 or x=2) or y=3 are not processed correctly

Re: Strategies around storing blobs in Mongo

2013-10-30 Thread Jukka Zitting
Hi, On Wed, Oct 30, 2013 at 2:50 AM, Chetan Mehrotra chetan.mehro...@gmail.com wrote: Currently we are storing blobs by breaking them into small chunks and then storing those chunks in MongoDB as part of blobs collection. This approach would cause issues as Mongo maintains a global exclusive

jackrabbit-oak build #2505: Fixed

2013-10-30 Thread Travis CI
Build Update for apache/jackrabbit-oak - Build: #2505 Status: Fixed Duration: 1696 seconds Commit: 11cccb00e0cece37956caebd102d5d0c79bb5323 (trunk) Author: Thomas Mueller Message: OAK-1128 Conditions of the form (x=1 or x=2) or y=3 are not processed correctly

Re: Clarifying: Support for properties and child nodes with the same name

2013-10-30 Thread Jukka Zitting
Hi, On Wed, Oct 30, 2013 at 6:43 AM, Stefan Guggisberg stefan.guggisb...@gmail.com wrote: SNNP was introduced with JSR 283, mainly due to pressure from vendors with legacy systems. we (day) were opposed since SNNP renders JCR paths ambiguous (IMO a serious problem). BTW SNNP are an optional

Re: Oak JCR Observation scalability aspects and concerns

2013-10-30 Thread Alexander Klimetschek
I created OAK-1133 for the Observation listener PLUS as I proposed it, to separate it from the Sling-specific case. https://issues.apache.org/jira/browse/OAK-1133 Cheers, Alex On 27.10.2013, at 23:57, Thomas Mueller muel...@adobe.com wrote: Hi, I've created OAK-1120 to start with the

Versionable paths

2013-10-30 Thread Tobias Bocanegra
Hi, oak now has rep:VersionablePaths which record the paths (per workspace) of the versionable node. I think this is wrong to store this on the nt:versionHistory node, since a node can be moved in between checkins. so I think it should be recorded on the nt:version node. I also find it

WeakReferences Query

2013-10-30 Thread Tobias Bocanegra
Hi, I'm looking at a strange problem where a WeakRef reverse lookup results in a Node that does not point to the respective reference: org.apache.jackrabbit.oak.spi.query.Cursors$TraversingCursor Traversed 11000 nodes with filter Filter(query=SELECT * FROM [nt:base] WHERE PROPERTY([ref],

Re: Clarifying: Support for properties and child nodes with the same name

2013-10-30 Thread Angela Schreiber
hi just a quick update on the status quo in response to our discussion during our weekly oak meeting related to OAK-1126: if my tests are correct SNNP actually does work in OAK and we could theoretically just flip the descriptor to return true. then the descriptor would match the implementation

Re: WeakReferences Query

2013-10-30 Thread Tobias Bocanegra
Actually, I was wrong. I think the iterator is correct and only reports properties that actually point to the respective node. but due to an NPE in the permission evaluation in the version store, I thought that was an offending result. So I think the query is correct. Nevertheless, I think we