Hi,
> >> One more thing:
> >>
> >> You said "(although it should not be needed). Can you confirm that? I
> mean,
> >> if you remove that line, does the test still passes with -
> >> Djdk.io.permissionsUseCanonicalPath=true? If so, we will need to look
> into
> >> other reasons, since I did see in many other places file permission on
> >> /home/jenkins/..../morfologik-polish-2.1.0.jar itself is granted.
> >
> > See my other mail! It should not be needed, because Class#getResource
> does not throw or document any Security implications. But in fact if you don't
> wrap a doPrivileged around the call, it will also fail without our extra
> permission. It was added to work around this issue with 3rd party JARs.
>
> You mean this was observed before the new FilePermission change in
> 8164705?
Yes. The reason why we have the extra FilePermission policy in our policy file
is caused by the fact that there are many 3rd party JARs that read resources
and don't wrap their getClass().getResource() calls with doPrivileged. Because
of this, the code does not have access to its own resources because it is
running in the context of the calling code (which is a different JAR file).
Because of this we have the whole lib directory as a "workaround" permission
added (this is the Apache Ivy cache dir @ ${user.home}/.ivy/cache/* where
downloaded Maven Central artifacts are stored):
https://github.com/apache/lucene-solr/blob/master/lucene/tools/junit4/tests.policy#L25
>From my perspective this looks wrong, because there is no security
>implications documented on Class#getResource, so it is completely unclear that
>you actually need a doPrivileged when calling Class#getResource[AsStream]().
>This is a separate issue and has nothing to do with your changes. It was and
>still is broken, IMHO.
Previously, the extra permission added using ${user.home} was working fine
because paths were canonicalized. Which is no longer the case with your change.
> If so, this means before the code change, you need FilePermissions on both
> /var/lib/.../- and /home/jenkins/.../thejar to make sure the jar is
> accessible.
> Here it seems the 2nd one should be enough but actually not, and the 1st is
> added to guarantee it. And after the code change, the 1st is useless and the
> test starts to fail.
It was the other way round. Before the change it worked because ${user.home}
was normalized to its canonical form while reading the policy file. After your
change this broke, because user.home used the symlink, while the actual JAR
file permission checks used the canonical path (maybe because of how Ant/Ivy
resolves the classpath). Apache Ant generally canonicalizes all paths before it
passes them as property or classpath down to sub processes.
> I would consider adding a compatibility layer to make the 1st also useful, but
> at the same time, if we can fix the 2nd, that will be very helpful.
I'd add a layer just when reading the policy file: If the policy file reader
sees a FilePermission it creates it, but also calculates the canonical path of
the referenced file. If this one differs, it adds a second permission using the
canonic path to the PermissionCollection, so both observed paths are granted.
This would fix a lot of issues (not all), but would make it much more
predictable what happens especially for system administrators.
The other issue about Class#getResource() not documenting that it has security
implications (and the stupid "null" return value instead of something more
useful like NoSuchFileException or SecurityException), is a different thing!
IMHO, Class#getResource(AsStream) should use the security context of the code
source calling the method so the stupid doPrivileged (which nobody ever does)
is not needed.
> Thanks
> Max
>
>
> >
> > I just blew up with the symlink on home dir.
> >
> > Uwe
> >
> >> Thanks
> >> Max
> >>
> >>
> >>> On Oct 17, 2016, at 11:17 PM, Wang Weijun <weijun.w...@oracle.com>
> >> wrote:
> >>>
> >>> Yes, this is where the problem is.
> >>>
> >>> So it looks like the permission is granted in a policy file instead of
> >>> being
> >> granted by the class loader itself. In this case, the path of the
> >> permission
> >> must match how you access that file.
> >>>
> >>> I'll think about your suggestion. However, can you guarantee the code
> >> always accesses the file using the canonicalized path? For example,
> suppose
> >> the actual file is /x, both /a and /b symlink to it, and you grant the
> permission
> >> on /a in a policy file. Even if I canonicalize /a to /x and grant
> >> permissions
> on
> >> both /a and /x, you will still see a failure if the code access /b.
> >>>
> >>> --Max
> >>>
> >>>> On Oct 17, 2016, at 11:03 PM, Uwe Schindler <uschind...@apache.org>
> >> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> I attached the debugging logs ("all") for the working (conaonicalized)
> and
> >> non-working case to this mail. Please keep in mind that around the
> >> getResource() is no doPrivileged, so it looks like it uses privileges of
> >> outer
> JAR.
> >>>>
> >>>> I also checked, we have the following explicitely in our policy file,
> >>>> which
> >> explicitly allows everything in the IVY cache (although it should not be
> >> needed) - so it should really work anyways!!!
> >>>>
> >>>> permission java.io.FilePermission "${user.home}${/}.ivy2${/}cache${/}-
> ",
> >> "read";
> >>>>
> >>>> But I think I know the issue - which is really stupid - but it’s a fact
> affecting
> >> maaaany people! Here is what the policy file parsing says above the
> >> permission given before (please compare first line and second line):
> >>>>
> >>>> [junit4] 2> Active Principals: []
> >>>> [junit4] 2> policy: granting ("java.io.FilePermission"
> >> "/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/-" "read")
> >>>> [junit4] 2> policy: granting ("java.io.FilePermission"
> >> "/var/lib/jenkins/.ivy2/cache/-" "read")
> >>>> [junit4] 2> policy: granting ("java.io.FilePermission"
> >> "/home/jenkins/tools/java/64bit/jdk-9-ea+140/-" "read,execute")
> >>>>
> >>>> Unfortunately, the user changed its home directory, the old one
> >> (var/lib/jenkins) symlinked to the new one, but /etc/passwd was not
> >> updated. So ${user.home} looks different. This is the reason why it did not
> >> happen locally on my windows machine!
> >>>>
> >>>> IMHO: When reading the policy file, it should canonicalize all paths,
> >> especially stuff like ${user.home}. At runtime when actually checking the
> >> permissions it should not canonicalize for performance reasons. Not sure
> if
> >> this is a bug, but this could hit lots of people. I know that other Jenkins
> >> servers where Lucene runs their tests (not yet on Java 9) are setup in
> similar
> >> ways!
> >>>>
> >>>> Uwe
> >>>>
> >>>> -----
> >>>> Uwe Schindler
> >>>> H.-H.-Meier-Allee 63, D-28213 Bremen
> >>>> http://www.thetaphi.de
> >>>> eMail: u...@thetaphi.de
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Wang Weijun [mailto:weijun.w...@oracle.com]
> >>>>> Sent: Monday, October 17, 2016 4:03 PM
> >>>>> To: dev@lucene.apache.org
> >>>>> Cc: Dalibor Topic <dalibor.to...@oracle.com>; Balchandra Vaidya
> >>>>> <balchandra.vai...@oracle.com>; Muneer Kolarkunnu
> >>>>> <abdul.kolarku...@oracle.com>; Rory O'Donnell
> >>>>> <rory.odonn...@oracle.com>
> >>>>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-
> ea+140)
> >> -
> >>>>> Build # 18064 - Unstable!
> >>>>>
> >>>>> OK, I'll try it tomorrow.
> >>>>>
> >>>>> At the same time can you try -Djava.security.debug=all or -
> >>>>> Djava.security.debug=scl and see if it shows what permission is
> granted?
> >>>>>
> >>>>> Thanks
> >>>>> Max
> >>>>>
> >>>>>> On Oct 17, 2016, at 9:07 PM, Uwe Schindler
> <uschind...@apache.org>
> >>>>> wrote:
> >>>>>>
> >>>>>> FYI,
> >>>>>>
> >>>>>> With:
> >>>>>> $ ant test -Dtests.useSecurityManager=true -Dargs='-
> >>>>> Djava.security.debug=access -
> >> Djdk.io.permissionsUseCanonicalPath=true'
> >>>>>>
> >>>>>> I can see the following line:
> >>>>>>
> >>>>>> [junit4] 2> access: access allowed ("java.io.FilePermission"
> >>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
> >>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
> >>>>>>
> >>>>>> So there is a difference!!!
> >>>>>>
> >>>>>> Uwe
> >>>>>>
> >>>>>> -----
> >>>>>> Uwe Schindler
> >>>>>> uschind...@apache.org
> >>>>>> ASF Member, Apache Lucene PMC / Committer
> >>>>>> Bremen, Germany
> >>>>>> http://lucene.apache.org/
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Uwe Schindler [mailto:uschind...@apache.org]
> >>>>>>> Sent: Monday, October 17, 2016 2:44 PM
> >>>>>>> To: dev@lucene.apache.org; 'Wang Weijun'
> >> <weijun.w...@oracle.com>
> >>>>>>> Cc: 'Dalibor Topic' <dalibor.to...@oracle.com>; 'Balchandra Vaidya'
> >>>>>>> <balchandra.vai...@oracle.com>; 'Muneer Kolarkunnu'
> >>>>>>> <abdul.kolarku...@oracle.com>; 'Rory O'Donnell'
> >>>>>>> <rory.odonn...@oracle.com>
> >>>>>>> Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-
> >> ea+140) -
> >>>>>>> Build # 18064 - Unstable!
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I attached the log of a test run. As you see it says "access denied"
> >> while
> >>>>>>> reading the morphologic JAR file:
> >>>>>>>
> >>>>>>> jenkins@serv1:~/workspace/Lucene-Solr-master-
> >>>>>>> Linux/lucene/analysis/morfologik$ ant test -Dargs='-
> >>>>>>> Djava.security.debug=access' -
> Dtestcase=TestMorfologikFilterFactory
> >>>>>>> [...]
> >>>>>>> [junit4] 2> access: access denied ("java.io.FilePermission"
> >>>>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
> >>>>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
> >>>>>>> [junit4] 2> access: access denied ("java.io.FilePermission"
> >>>>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
> >>>>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
> >>>>>>> [...]
> >>>>>>>
> >>>>>>> To reproduce, do the following:
> >>>>>>> Check out Lucene's repository from Github:
> >>>>>>> https://github.com/apache/lucene-solr
> >>>>>>>
> >>>>>>> Go to root directory of it and run (Apache Ant required): "ant ivy-
> >>>>> bootstrap" -
> >>>>>>>> this installs the IVY downloader. Be sure to have internet
> connection
> >> to
> >>>>>>> access Maven Central!
> >>>>>>>
> >>>>>>> The cd into lucene/analysis/morfologik. From there you can run
> tests
> >>>>> using
> >>>>>>> "ant test".
> >>>>>>>
> >>>>>>> To pass extra JDK args, pass e.g., $ ant test -Dargs='-
> >>>>>>> Djava.security.debug=access'
> >>>>>>> The test passes with the JDK option canonicalized paths.
> >>>>>>>
> >>>>>>> From the above error message you see that the code tries to read
> the
> >> JAR
> >>>>> file
> >>>>>>> of morphologic, which is completely outside the build directory in
> the
> >> IVY
> >>>>>>> cache folder, but it is part of the
> >>>>>>>
> >>>>>>> I have seen that you have a short test for the same in your checkin
> to
> >> JDK,
> >>>>>>> but there is a slight difference: You are not using a JAR file, just
> >>>>>>> plain
> >> class
> >>>>>>> files in your test. And this works also for Lucene. It is just
> >>>>> JARURLConnection
> >>>>>>> that breaks. The reason why other parts of Lucene don't fail is
> because
> >>>>>>> Lucene's own tests use class file (not yet packaged). In always fails
> >> when
> >>>>> you
> >>>>>>> have 3rd party JAR files.
> >>>>>>>
> >>>>>>> Uwe
> >>>>>>>
> >>>>>>> -----
> >>>>>>> Uwe Schindler
> >>>>>>> uschind...@apache.org
> >>>>>>> ASF Member, Apache Lucene PMC / Committer
> >>>>>>> Bremen, Germany
> >>>>>>> http://lucene.apache.org/
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Wang Weijun [mailto:weijun.w...@oracle.com]
> >>>>>>>> Sent: Monday, October 17, 2016 2:20 PM
> >>>>>>>> To: Uwe Schindler <u...@thetaphi.de>
> >>>>>>>> Cc: dev@lucene.apache.org; Dalibor Topic
> >> <dalibor.to...@oracle.com>;
> >>>>>>>> Balchandra Vaidya <balchandra.vai...@oracle.com>; Muneer
> >>>>> Kolarkunnu
> >>>>>>>> <abdul.kolarku...@oracle.com>; Rory O'Donnell
> >>>>>>>> <rory.odonn...@oracle.com>
> >>>>>>>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-
> >> ea+140)
> >>>>> -
> >>>>>>>> Build # 18064 - Unstable!
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Oct 17, 2016, at 6:24 PM, Rory O'Donnell
> >>>>>>> <rory.odonn...@oracle.com>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Adding Max to discussion.
> >>>>>>>>>
> >>>>>>>>> Rgds,Rory
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 17/10/2016 08:26, Dawid Weiss wrote:
> >>>>>>>>>> Yeah, this doesn't look so good. This class and the resource it
> tries
> >>>>>>>>>> to access are within the same JAR file -- if I understand correctly
> >>>>>>>>>> the JDK issue you quoted as the source of the problem, the code
> >> here
> >>>>>>>>>> should be able to pass regardless of file restrictions/ policies?
> >>>>>>>>
> >>>>>>>> Correct. A class should always be able to read a file that's on its
> >>>>> classpath,
> >>>>>>> no
> >>>>>>>> matter if it's inside a jar or a .class file.
> >>>>>>>>
> >>>>>>>> Can someone tell me how to reproduce this error? I have never
> >> played
> >>>>> with
> >>>>>>>> Lucene before. Also, please try adding -
> Djava.security.debug=access
> >> and
> >>>>>>> see
> >>>>>>>> if you can get some extra debug info.
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> Max
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Dawid
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Oct 16, 2016 at 10:09 PM, Uwe Schindler
> >> <u...@thetaphi.de>
> >>>>>>>> wrote:
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> I reverted the Lucene builds to build Java 9 138 for now. I will
> >> later
> >>>>>>> check
> >>>>>>>> if this also happens with build 139, which I have to download first.
> I
> >> will
> >>>>>>> also
> >>>>>>>> debug locally.
> >>>>>>>>>>>
> >>>>>>>>>>> The code fails because this code hits "null" on getResource() at
> >>>>>>>>
> >>>>>
> >> morfologik.stemming.polish.PolishStemmer.<init>(PolishStemmer.java:34)
> >>>>>>>>>>>
> >>>>>>>>>>> https://github.com/morfologik/morfologik-
> >>>>>>>> stemming/blob/master/morfologik-
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>
> polish/src/main/java/morfologik/stemming/polish/PolishStemmer.java#L32
> >>>>>>>>>>>
> >>>>>>>>>>> This is impossible to happen, because the dict file is in same
> >> package.
> >>>>> I
> >>>>>>>> have no idea why this only fails here and not at other places in
> >> Lucene.
> >>>>> The
> >>>>>>>> main difference looks like the use of URL instead of
> >>>>> getResourceAsStream()
> >>>>>>>> like other places in Lucene.
> >>>>>>>>>>>
> >>>>>>>>>>> So this seems to be a major regression in Java 9 build 140.
> >>>>>>>>>>>
> >>>>>>>>>>> Uwe
> >>>>>>>>>>>
> >>>>>>>>>>> -----
> >>>>>>>>>>> Uwe Schindler
> >>>>>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
> >>>>>>>>>>> http://www.thetaphi.de
> >>>>>>>>>>> eMail: u...@thetaphi.de
> >>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Uwe Schindler [mailto:u...@thetaphi.de]
> >>>>>>>>>>>> Sent: Sunday, October 16, 2016 8:38 PM
> >>>>>>>>>>>> To: dev@lucene.apache.org
> >>>>>>>>>>>> Cc: dalibor.to...@oracle.com;
> balchandra.vai...@oracle.com;
> >>>>>>>> 'Muneer
> >>>>>>>>>>>> Kolarkunnu' <abdul.kolarku...@oracle.com>; 'Dawid Weiss'
> >>>>>>>>>>>> <dawid.we...@cs.put.poznan.pl>; dev@lucene.apache.org
> >>>>>>>>>>>> Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-
> 9-
> >>>>>>>> ea+140) -
> >>>>>>>>>>>> Build # 18064 - Unstable!
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> this seems to be a new regression in Java 9 ea build 140.
> >>>>> Interestingly
> >>>>>>>> this
> >>>>>>>>>>>> only affects 2 libraries (morphologic and commons-codec
> >> phonetic).
> >>>>>>> We
> >>>>>>>> use
> >>>>>>>>>>>> loading of resources from classloaders at many places; it is
> >> unclear
> >>>>> to
> >>>>>>>> me,
> >>>>>>>>>>>> why it only fails here. I will look into the code, but this is
> outside
> >> of
> >>>>>>>> Lucene. I
> >>>>>>>>>>>> think it might be some crazyness like using context class
> loader
> >> in
> >>>>> non-
> >>>>>>>> proper
> >>>>>>>>>>>> ways or similar.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Maybe it is a new bug in JDK 9 build 139 or build 140 (the last
> >>>>> working
> >>>>>>>> one
> >>>>>>>>>>>> was build 138).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Uwe
> >>>>>>>>>>>>
> >>>>>>>>>>>> -----
> >>>>>>>>>>>> Uwe Schindler
> >>>>>>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
> >>>>>>>>>>>> http://www.thetaphi.de
> >>>>>>>>>>>> eMail: u...@thetaphi.de
> >>>>>>>>>>>>
> >>>>>>>>>>>>> -----Original Message-----
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: dev-h...@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
