RE: [jira] Commented: (JCR-926) Global data store for binaries
We are not planning to backport the data store patch to 1.3 release. We'd like to know if is it better, in terms of stability, to have the patch applied to 1.3.1 release instead of the revision the latest version of the patch was applied to (revision # 553213) Thanks, Pablo -Original Message- From: Jukka Zitting [mailto:[EMAIL PROTECTED] Sent: Friday, July 27, 2007 2:31 PM To: [email protected] Subject: Re: [jira] Commented: (JCR-926) Global data store for binaries Hi, On 7/27/07, Pablo Rios <[EMAIL PROTECTED]> wrote: > I'm asking just for a the patch (i.e.: "dataStore5.patch"), that can > be applied to the revision of JR 1.3.1 release. If you need the data store functionality already now, before Jackrabbit 1.4, I would strongly suggest that you use a snapshot version compiled directly from svn trunk instead of backporting the data store patches to 1.3.x. The svn trunk is usually quite stable and should be good for any development and testing work. I even know some people that are running Jackrabbit snapshots in production environments, but I wouldn't recommend going that far. BR, Jukka Zitting Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it.
Re: [jira] Commented: (JCR-926) Global data store for binaries
Hi, On 7/27/07, Pablo Rios <[EMAIL PROTECTED]> wrote: > I'm asking just for a the patch (i.e.: "dataStore5.patch"), that can be > applied to the revision of JR 1.3.1 release. If you need the data store functionality already now, before Jackrabbit 1.4, I would strongly suggest that you use a snapshot version compiled directly from svn trunk instead of backporting the data store patches to 1.3.x. The svn trunk is usually quite stable and should be good for any development and testing work. I even know some people that are running Jackrabbit snapshots in production environments, but I wouldn't recommend going that far. BR, Jukka Zitting
RE: [jira] Commented: (JCR-926) Global data store for binaries
> Did you already try out this feature? Yes. The serialization issue with large binary objects is truly critical for our project. Pablo -Original Message- From: Thomas Mueller [mailto:[EMAIL PROTECTED] Sent: Friday, July 27, 2007 12:29 PM To: [email protected] Subject: Re: [jira] Commented: (JCR-926) Global data store for binaries Hi, Sure, I can create such a patch. However it will take some time. Did you already try out this feature? Thomas On 7/27/07, Pablo Rios <[EMAIL PROTECTED]> wrote: > I'm asking just for a the patch (i.e.: "dataStore5.patch"), that can > be applied to the revision of JR 1.3.1 release. > > Thanks > > -Original Message- > From: Thomas Mueller [mailto:[EMAIL PROTECTED] > Sent: Friday, July 27, 2007 12:12 PM > To: [email protected] > Subject: Re: [jira] Commented: (JCR-926) Global data store for > binaries > > Hi, > > I'm the wrong person to ask, but I don't think so... It's a new > feature and requires changes in a few internal APIs. > > Jukka? > > Thomas > > Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it. > Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it.
RE: [jira] Commented: (JCR-926) Global data store for binaries
Yes, it would be great ! We've started working on a DataStore implementation based on a dbms. We can share that work when it get in well form. Thanks, Pablo -Original Message- From: Thomas Mueller [mailto:[EMAIL PROTECTED] Sent: Friday, July 27, 2007 12:15 PM To: [email protected] Subject: Re: [jira] Commented: (JCR-926) Global data store for binaries Sorry, maybe I misunderstood the question. Do you like me to update the patch for Jackrabbit 1.3.1? Thomas On 7/27/07, Thomas Mueller <[EMAIL PROTECTED]> wrote: > Hi, > > I'm the wrong person to ask, but I don't think so... It's a new > feature and requires changes in a few internal APIs. > > Jukka? > > Thomas > Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it.
Re: [jira] Commented: (JCR-926) Global data store for binaries
Hi, Sure, I can create such a patch. However it will take some time. Did you already try out this feature? Thomas On 7/27/07, Pablo Rios <[EMAIL PROTECTED]> wrote: > I'm asking just for a the patch (i.e.: "dataStore5.patch"), that can be > applied to the revision of JR 1.3.1 release. > > Thanks > > -Original Message- > From: Thomas Mueller [mailto:[EMAIL PROTECTED] > Sent: Friday, July 27, 2007 12:12 PM > To: [email protected] > Subject: Re: [jira] Commented: (JCR-926) Global data store for binaries > > Hi, > > I'm the wrong person to ask, but I don't think so... It's a new feature > and requires changes in a few internal APIs. > > Jukka? > > Thomas > > Notice: This email message, together with any attachments, may contain > information of BEA Systems, Inc., its subsidiaries and affiliated > entities, that may be confidential, proprietary, copyrighted and/or > legally privileged, and is intended solely for the use of the individual or > entity named in this message. If you are not the intended recipient, and have > received this message in error, please immediately return this by email and > then delete it. >
RE: [jira] Commented: (JCR-926) Global data store for binaries
I'm asking just for a the patch (i.e.: "dataStore5.patch"), that can be applied to the revision of JR 1.3.1 release. Thanks -Original Message- From: Thomas Mueller [mailto:[EMAIL PROTECTED] Sent: Friday, July 27, 2007 12:12 PM To: [email protected] Subject: Re: [jira] Commented: (JCR-926) Global data store for binaries Hi, I'm the wrong person to ask, but I don't think so... It's a new feature and requires changes in a few internal APIs. Jukka? Thomas Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it.
Re: [jira] Commented: (JCR-926) Global data store for binaries
Sorry, maybe I misunderstood the question. Do you like me to update the patch for Jackrabbit 1.3.1? Thomas On 7/27/07, Thomas Mueller <[EMAIL PROTECTED]> wrote: > Hi, > > I'm the wrong person to ask, but I don't think so... It's a new > feature and requires changes in a few internal APIs. > > Jukka? > > Thomas >
Re: [jira] Commented: (JCR-926) Global data store for binaries
Hi, I'm the wrong person to ask, but I don't think so... It's a new feature and requires changes in a few internal APIs. Jukka? Thomas
RE: [jira] Commented: (JCR-926) Global data store for binaries
Thomas, will the binaries data store patch be available for the revision of Jackrabbit 1.3.1 release ? Thanks, Pablo -Original Message- From: Thomas Mueller [mailto:[EMAIL PROTECTED] Sent: Thursday, July 19, 2007 6:57 AM To: [email protected] Subject: Re: [jira] Commented: (JCR-926) Global data store for binaries > I'm sorry I haven't had the cycles lately to properly review the patches. Sure, I understand. > Would it be possible to split them to smaller semi-independent pieces? > For example we could commit the data store implementation independent > of the integration to rest of the Jackrabbit core. Sure, just the ..core.data package. Do you need a new patch or can you just commit this subset? The real problem will be the API changes (introducing the DataStore in many constructors and methods). Can I help to integrate those changes? Or change it to make integration easier? Thomas Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it.
Re: [jira] Commented: (JCR-926) Global data store for binaries
Hi, On 7/19/07, Thomas Mueller <[EMAIL PROTECTED]> wrote: > Would it be possible to split them to smaller semi-independent pieces? > For example we could commit the data store implementation independent > of the integration to rest of the Jackrabbit core. Sure, just the ..core.data package. Do you need a new patch or can you just commit this subset? I can take care of it. The real problem will be the API changes (introducing the DataStore in many constructors and methods). Can I help to integrate those changes? Or change it to make integration easier? I'll give the latest patch a look and get back to you on the best way forward. BR, Jukka Zitting
Re: [jira] Commented: (JCR-926) Global data store for binaries
I'm sorry I haven't had the cycles lately to properly review the patches. Sure, I understand. Would it be possible to split them to smaller semi-independent pieces? For example we could commit the data store implementation independent of the integration to rest of the Jackrabbit core. Sure, just the ..core.data package. Do you need a new patch or can you just commit this subset? The real problem will be the API changes (introducing the DataStore in many constructors and methods). Can I help to integrate those changes? Or change it to make integration easier? Thomas
Re: [jira] Commented: (JCR-926) Global data store for binaries
Hi, Thomas, your patch references org.apache.jackrabbit.benchmark.RandomInputStream, wich is not included in the patch, and I couldn't find it anywhere. You are right, I saw that as well. This class is used for testing, you can ignore it (the test doesn't do anything useful anyway). The next patch will contain all classes. Thomas Regards > Global data store for binaries > -- > > Key: JCR-926 > URL: https://issues.apache.org/jira/browse/JCR-926 > Project: Jackrabbit > Issue Type: New Feature > Components: core >Reporter: Jukka Zitting > Attachments: dataStore.patch, DataStore.patch, DataStore2.patch, dataStore3.patch, dataStore4.zip, internalValue.patch, ReadWhileSaveTest.patch > > > There are three main problems with the way Jackrabbit currently handles large binary values: > 1) Persisting a large binary value blocks access to the persistence layer for extended amounts of time (see JCR-314) > 2) At least two copies of binary streams are made when saving them through the JCR API: one in the transient space, and one when persisting the value > 3) Versioining and copy operations on nodes or subtrees that contain large binary values can quickly end up consuming excessive amounts of storage space. > To solve these issues (and to get other nice benefits), I propose that we implement a global "data store" concept in the repository. A data store is an append-only set of binary values that uses short identifiers to identify and access the stored binary values. The data store would trivially fit the requirements of transient space and transaction handling due to the append-only nature. An explicit mark-and-sweep garbage collection process could be added to avoid concerns about storing garbage values. > See the recent NGP value record discussion, especially [1], for more background on this idea. > [1] http://mail-archives.apache.org/mod_mbox/jackrabbit-dev/200705.mbox/[EMAIL PROTECTED] -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (JCR-926) Global data store for binaries
Hi, With the data store feature disabled all TCK tests passed, Same for me. with this feature enabled I got 46 errors and 10 failures. It worked for me. I suppose these test failures are related with the steps above. Probably. How run the tests? Which is the revision the last version of the binary data store patch should be applied to ? ... I didn't write that down. I will merge my changes today and create a new patch, where I will tell the revision I used. Given the append-only nature of this feature, when a node with a binary property is removed binary content remains in the data store. When do you expect to have a garbage collection process of binary content (files) in the file system ? I started to implement that yesterday. I have some ideas how to make it faster than simply 'scan through the whole repository, mark all blobs, delete unmarked'. However the first implementation will be about like that. Also, the garbage needs to be a background process (unlike memory garbage collection). The patch will contain more information about that. Are you planning to provide a database-backed implementation of the data store ? So far I didn't plan to implement that, but it should be trivial (the DataStore API is very simple). Thomas
