Re: Segment Store GC failing on Windows
Hi, This is a know issue for some Windows environments. A workaround is to set tarmk.mode=32 in the configuration of the SegmentNodeStoreService. See also the "Tar storage" section at https://helpx.adobe.com/experience-manager/kb/performance-tuning-tips.html. Michael https://helpx.adobe.com/experience-manager/kb/performance-tuning-tips.html On 13.09.17 14:27, Yegor Kozlov wrote: Hi Every time Segment Store GC runs I get a bunch of these exceptions: 04.09.2017 07:41:53.157 *WARN* [TarMK filer reaper [C:\Users\yegor\aem\segmentstore]] org.apache.jackrabbit.oak.segment.file.FileReaper Unable to remove file C:\Users\yegor\aem\segmentstore\data00163a.tar java.nio.file.FileSystemException: E:\Inetpub\adobe\aem\authorrepository\repository\segmentstore\data00163a.tar: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108) at java.nio.file.Files.deleteIfExists(Files.java:1165) at org.apache.jackrabbit.oak.segment.file.FileReaper.reap(FileReaper.java:73) at org.apache.jackrabbit.oak.segment.file.FileStore$3.run(FileStore.java:245) at org.apache.jackrabbit.oak.segment.file.SafeRunnable.run(SafeRunnable.java:67) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ... I'm running Oak 1.6.2 from AEM 6.3 SP1 on 64-bit Windows 7. The TarMK mode is 64 which defaults from the 'sun.arch.data.model' system property. The problem is reproducible on Windows Server 2012 as well. It smells like I'm hitting OAK-4274. I perfectly understand it is a JDK issue and it cannot be fixed from the Oak code. My point is that SegmentNodeStoreService picks up a wrong TarMK mode on Windows. In the current implementation the default value is taken from the 'sun.arch.data.model' system property which is always 64 on 64-bit platforms. IMO it should always default to 32 on Windows and use 'sun.arch.data.model' on other operating systems. Regards, Yegor
Re: OAK-6575 - A word of caution
Hi Angela, On 13 September 2017 at 06:50, Angela Schreiber wrote: > Hi Ian > > The new proposal looks a lot better to me. > > The only concern from a security perspective I could come up with is the > one we expressed already with the very first proposal (see initial word of > caution mail sent by Francesco): applications built on top of Oak can up > to now be sure that all access to the repository content is subject to the > same permission evaluation as configured with the repository setup. This > is no longer guaranteed when we offer the ability to plug application code > that may or may not by-pass the evaluation by allowing it to directly > access binaries. > I don't think this patch bypasses Oak security, and since the API can only be implemented by Oak itself. I am sure any future patch would be subject to the same scrutiny. If it can be implemented outside Oak, then Oak has already been breached, something I can see no evidence of. In this case, the signed url is only issued after Oak security has granted access to the binary, and only returned over the JCR API to the JCR Session that made the call, in the same way that an InputStream allows the bytes of the binary to be read by that session. The URL only allows read access. What the session does with that data, is outside the control of Oak. Unlike the byte[] from the that has no protection, the signed URL is protected. It may only be used unmodified for the purpose it was intended by Oak and only for a well defined period of time. In that respect, arguably, its is more secure than the byte[] or InputStream. > > While I know that this is actually the goal of the whole exercise, we have > to be aware that this also is a change in our Oak security model. As such > this may look like a security breach and I have been told by my colleagues > at Adobe that the 'single-way-to-access' is a relevant security question > with a lot of potential customers. > > That doesn't mean that I am opposed to the patch in it's current form as I > see the benefits from an Oak pov, I just want to highlight that we are > going to make a fundamental change and we should treat and document it > with the necessary care... maybe we should take this opportunity to > finally create a threat model for Oak? Doing so at this stage would allow > us to visualise the proposed change to all parties involved. > > wdyt? > Agreed. Having a fully developed threat model which clarified all the risks for every aspect of Oak would, imho, be much better than not defining the risks that exist. Even the most secure application has risks, best exposed in a threat model, however brief. Unfortunately Oak now exists in a world which is distributed, where applications need to embrace the network. This is a fundamental change, which Oak has to embrace. An Oak Threat model that recognises this will be a great step forwards. On the other hand, if you are saying that the Oak Threat model has to be developed and agreed, before this patch can be added, then I am concerned that will take too long. Doing justice to an Oak Treat model will require resource. Best `Regards Ian > > Kind regards > Angela > > > On 07/09/17 16:39, "Ian Boston" wrote: > > >On 7 September 2017 at 14:41, Francesco Mari > >wrote: > > > >> On Thu, Sep 7, 2017 at 11:05 AM, Ian Boston wrote: > >> > On 7 September 2017 at 07:22, Ian Boston wrote: > >> > > >> >> Hi, > >> >> > >> >> On 6 September 2017 at 22:43, Michael Dürig > >>wrote: > >> >> > >> >>> > >> >>> > >> >>> On 06.09.17 23:08, Michael Dürig wrote: > >> >>> > >> > >> Hi, > >> > >> On 05.09.17 14:09, Ian Boston wrote: > >> > >> > Repeating the comment to on OAK-6575 here for further discussion. > >>2 > >> new > >> > Patches exploring both options. > >> > > >> > >> I would actually prefer the original patch ( > >> https://github.com/ieb/jackrabbit-oak/compare/trunk...ieb:O > >> AK-6575?expand=1) in most parts. However I have concerns regarding > >>the > >> generality of the new OakConversionService API as mentioned in my > >> previous > >> mail. I would be more comfortable if this could be restricted to > >> something > >> that resembles more like a "URIProvider", which given a blob > >>returns > >> an URI. > >> > >> On the implementation side, why do we need to introduce the > >>adaptable > >> machinery? Couldn't we re-use the Whiteboard and OSGiWhiteBoard > >> mechanisms > >> instead? I think these could be used to track URIProvider instances > >> registered by the various blob stores. > >> > >> > >> >>> See https://github.com/mduerig/jackrabbit-oak/commit/2709c097b01 > >> >>> a006784b7011135efcbbe3ce1ba88 for a *really* quickly hacked together > >> and > >> >>> entirely untested POC. But it should get the idea across though. > >> >> > >> >> > >> >> > >> >> Thank you. > >> >> That makes sense. > >> >> I think it only needs the java/org/apache/jackrabbi
Re: OAK-6575 - A word of caution
Hi Ian The new proposal looks a lot better to me. The only concern from a security perspective I could come up with is the one we expressed already with the very first proposal (see initial word of caution mail sent by Francesco): applications built on top of Oak can up to now be sure that all access to the repository content is subject to the same permission evaluation as configured with the repository setup. This is no longer guaranteed when we offer the ability to plug application code that may or may not by-pass the evaluation by allowing it to directly access binaries. While I know that this is actually the goal of the whole exercise, we have to be aware that this also is a change in our Oak security model. As such this may look like a security breach and I have been told by my colleagues at Adobe that the 'single-way-to-access' is a relevant security question with a lot of potential customers. That doesn't mean that I am opposed to the patch in it's current form as I see the benefits from an Oak pov, I just want to highlight that we are going to make a fundamental change and we should treat and document it with the necessary care... maybe we should take this opportunity to finally create a threat model for Oak? Doing so at this stage would allow us to visualise the proposed change to all parties involved. wdyt? Kind regards Angela On 07/09/17 16:39, "Ian Boston" wrote: >On 7 September 2017 at 14:41, Francesco Mari >wrote: > >> On Thu, Sep 7, 2017 at 11:05 AM, Ian Boston wrote: >> > On 7 September 2017 at 07:22, Ian Boston wrote: >> > >> >> Hi, >> >> >> >> On 6 September 2017 at 22:43, Michael Dürig >>wrote: >> >> >> >>> >> >>> >> >>> On 06.09.17 23:08, Michael Dürig wrote: >> >>> >> >> Hi, >> >> On 05.09.17 14:09, Ian Boston wrote: >> >> > Repeating the comment to on OAK-6575 here for further discussion. >>2 >> new >> > Patches exploring both options. >> > >> >> I would actually prefer the original patch ( >> https://github.com/ieb/jackrabbit-oak/compare/trunk...ieb:O >> AK-6575?expand=1) in most parts. However I have concerns regarding >>the >> generality of the new OakConversionService API as mentioned in my >> previous >> mail. I would be more comfortable if this could be restricted to >> something >> that resembles more like a "URIProvider", which given a blob >>returns >> an URI. >> >> On the implementation side, why do we need to introduce the >>adaptable >> machinery? Couldn't we re-use the Whiteboard and OSGiWhiteBoard >> mechanisms >> instead? I think these could be used to track URIProvider instances >> registered by the various blob stores. >> >> >> >>> See https://github.com/mduerig/jackrabbit-oak/commit/2709c097b01 >> >>> a006784b7011135efcbbe3ce1ba88 for a *really* quickly hacked together >> and >> >>> entirely untested POC. But it should get the idea across though. >> >> >> >> >> >> >> >> Thank you. >> >> That makes sense. >> >> I think it only needs the java/org/apache/jackrabbit/ >> >> oak/blob/cloud/aws/s3/CloudFrontS3SignedUrlAdapterFactory.java and >>the >> >> API to be inside Oak, everything else can be in Sling. >> >> I'll update my patch and do a 2 options for Sling. >> >> >> > >> > >> > >> > >> > https://github.com/ieb/jackrabbit-oak/compare/trunk.. >> .ieb:OAK-6575-3?expand=1 >> > >> > and >> > >> > >>https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3?expand=1 >> > >> > wdyt ? >> >> I like this a lot. It keeps Oak's side simple and cleanly integrates >> Oak's lower-level services in Sling. >> > > >Good news. >I think we should hold off committing the patch until Monday or Tuesday to >give those who may be offline this week a chance to comment. In particular >I have not seen a comment from Angela who I would expect to have a view as >this is acl/security related. That is assuming she is back online next >week. > >Best Regards >Ian > > >> >> > Obviously the second patch needs to be discussed with Sling dev, but >>is >> > should not be too contentious. >> > >> > Best Regards >> > Ian >> > >> > >> > >> >> >> >> I think that should address others concerns since it drops all signs >>of >> >> any generic object to object conversion from Oak (Francesco), and >> doesn't >> >> require wide scale fragile changes with implied requirements being >> placed >> >> on how intermediate classes are connected and behave (mine). >> >> >> >> Best Regards >> >> Ian >> >> >> >> >> >>> >> >>> Michael >> >>> >> >> >> >> >>
Segment Store GC failing on Windows
Hi Every time Segment Store GC runs I get a bunch of these exceptions: 04.09.2017 07:41:53.157 *WARN* [TarMK filer reaper [C:\Users\yegor\aem\segmentstore]] org.apache.jackrabbit.oak.segment.file.FileReaper Unable to remove file C:\Users\yegor\aem\segmentstore\data00163a.tar java.nio.file.FileSystemException: E:\Inetpub\adobe\aem\authorrepository\repository\segmentstore\data00163a.tar: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108) at java.nio.file.Files.deleteIfExists(Files.java:1165) at org.apache.jackrabbit.oak.segment.file.FileReaper.reap(FileReaper.java:73) at org.apache.jackrabbit.oak.segment.file.FileStore$3.run(FileStore.java:245) at org.apache.jackrabbit.oak.segment.file.SafeRunnable.run(SafeRunnable.java:67) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ... I'm running Oak 1.6.2 from AEM 6.3 SP1 on 64-bit Windows 7. The TarMK mode is 64 which defaults from the 'sun.arch.data.model' system property. The problem is reproducible on Windows Server 2012 as well. It smells like I'm hitting OAK-4274. I perfectly understand it is a JDK issue and it cannot be fixed from the Oak code. My point is that SegmentNodeStoreService picks up a wrong TarMK mode on Windows. In the current implementation the default value is taken from the 'sun.arch.data.model' system property which is always 64 on 64-bit platforms. IMO it should always default to 32 on Windows and use 'sun.arch.data.model' on other operating systems. Regards, Yegor
Re: Intent to backport OAK-6656 to 1.6 and 1.4 branch
Ack. Would continue with the backport then :). On Wed, Sep 13, 2017 at 2:22 PM, Chetan Mehrotra wrote: > Its was +0 ;) > Chetan Mehrotra > > > On Wed, Sep 13, 2017 at 2:15 PM, Vikas Saurabh > wrote: >> Hi Chetan, >> >> Was your concern a -1 or a +/- 0? >> >> Thanks, >> Vikas
Re: Intent to backport OAK-6656 to 1.6 and 1.4 branch
Its was +0 ;) Chetan Mehrotra On Wed, Sep 13, 2017 at 2:15 PM, Vikas Saurabh wrote: > Hi Chetan, > > Was your concern a -1 or a +/- 0? > > Thanks, > Vikas
Re: Intent to backport OAK-6656 to 1.6 and 1.4 branch
Hi Chetan, Was your concern a -1 or a +/- 0? Thanks, Vikas
Re: Intent to backport OAK-6656 to 1.6 and 1.4 branch
On 13/09/2017 09:22, Vikas Saurabh wrote: > Hi, > > On Wed, Sep 13, 2017 at 9:32 AM, Chetan Mehrotra > wrote: > >> Would the backport be of use now? As any upgrade I think would happen >> first to initial release from that branch where this fix would not be >> present > Well, from arguments pov, I think one can always package upgraded > setup with latest oak (yes, I understand that's not the case for AEM). > But, my hidden agenda is that "this is a bad bug + the fix is simple > => we should fix it". > But, sure, if it doesn't feel right, then I agree it's a not a major > bug from backport pov. +1 for backporting. It's trivial, well test covered and it's definitely better to have this as we don't know exactly what an Oak deployment could be. D.
Re: OAK-6650 release changes testing
On 12/09/2017 15:55, Robert Munteanu wrote: > -deploy > +package Gosh! I didn't see the `phase` binding... Didn't try it out yet but it should definitely work :) D.
Re: Intent to backport OAK-6656 to 1.6 and 1.4 branch
Hi, On Wed, Sep 13, 2017 at 9:32 AM, Chetan Mehrotra wrote: > Would the backport be of use now? As any upgrade I think would happen > first to initial release from that branch where this fix would not be > present Well, from arguments pov, I think one can always package upgraded setup with latest oak (yes, I understand that's not the case for AEM). But, my hidden agenda is that "this is a bad bug + the fix is simple => we should fix it". But, sure, if it doesn't feel right, then I agree it's a not a major bug from backport pov. Thanks, Vikas
