Hi,

> One more thing:
> 
> You said "(although it should not be needed). Can you confirm that? I mean,
> if you remove that line, does the test still passes with -
> Djdk.io.permissionsUseCanonicalPath=true? If so, we will need to look into
> other reasons, since I did see in many other places file permission on
> /home/jenkins/..../morfologik-polish-2.1.0.jar itself is granted.

See my other mail! It should not be needed, because Class#getResource does not 
throw or document any Security implications. But in fact if you don't wrap a 
doPrivileged around the call, it will also fail without our extra permission. 
It was added to work around this issue with 3rd party JARs.

I just blew up with the symlink on home dir.

Uwe

> Thanks
> Max
> 
> 
> > On Oct 17, 2016, at 11:17 PM, Wang Weijun <weijun.w...@oracle.com>
> wrote:
> >
> > Yes, this is where the problem is.
> >
> > So it looks like the permission is granted in a policy file instead of being
> granted by the class loader itself. In this case, the path of the permission
> must match how you access that file.
> >
> > I'll think about your suggestion. However, can you guarantee the code
> always accesses the file using the canonicalized path? For example, suppose
> the actual file is /x, both /a and /b symlink to it, and you grant the 
> permission
> on /a in a policy file. Even if I canonicalize /a to /x and grant permissions 
> on
> both /a and /x, you will still see a failure if the code access /b.
> >
> > --Max
> >
> >> On Oct 17, 2016, at 11:03 PM, Uwe Schindler <uschind...@apache.org>
> wrote:
> >>
> >> Hi,
> >>
> >> I attached the debugging logs ("all") for the working (conaonicalized) and
> non-working case to this mail. Please keep in mind that around the
> getResource() is no doPrivileged, so it looks like it uses privileges of 
> outer JAR.
> >>
> >> I also checked, we have the following explicitely in our policy file, which
> explicitly allows everything in the IVY cache (although it should not be
> needed) - so it should really work anyways!!!
> >>
> >> permission java.io.FilePermission "${user.home}${/}.ivy2${/}cache${/}-",
> "read";
> >>
> >> But I think I know the issue - which is really stupid - but it’s a fact 
> >> affecting
> maaaany people! Here is what the policy file parsing says above the
> permission given before (please compare first line and second line):
> >>
> >>  [junit4]   2>     Active Principals: []
> >>  [junit4]   2> policy:   granting ("java.io.FilePermission"
> "/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/-" "read")
> >>  [junit4]   2> policy:   granting ("java.io.FilePermission"
> "/var/lib/jenkins/.ivy2/cache/-" "read")
> >>  [junit4]   2> policy:   granting ("java.io.FilePermission"
> "/home/jenkins/tools/java/64bit/jdk-9-ea+140/-" "read,execute")
> >>
> >> Unfortunately, the user changed its home directory, the old one
> (var/lib/jenkins) symlinked to the new one, but /etc/passwd was not
> updated. So ${user.home} looks different. This is the reason why it did not
> happen locally on my windows machine!
> >>
> >> IMHO: When reading the policy file, it should canonicalize all paths,
> especially stuff like ${user.home}. At runtime when actually checking the
> permissions it should not canonicalize for performance reasons. Not sure if
> this is a bug, but this could hit lots of people. I know that other Jenkins
> servers where Lucene runs their tests (not yet on Java 9) are setup in similar
> ways!
> >>
> >> Uwe
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: u...@thetaphi.de
> >>
> >>> -----Original Message-----
> >>> From: Wang Weijun [mailto:weijun.w...@oracle.com]
> >>> Sent: Monday, October 17, 2016 4:03 PM
> >>> To: dev@lucene.apache.org
> >>> Cc: Dalibor Topic <dalibor.to...@oracle.com>; Balchandra Vaidya
> >>> <balchandra.vai...@oracle.com>; Muneer Kolarkunnu
> >>> <abdul.kolarku...@oracle.com>; Rory O'Donnell
> >>> <rory.odonn...@oracle.com>
> >>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140)
> -
> >>> Build # 18064 - Unstable!
> >>>
> >>> OK, I'll try it tomorrow.
> >>>
> >>> At the same time can you try -Djava.security.debug=all or -
> >>> Djava.security.debug=scl and see if it shows what permission is granted?
> >>>
> >>> Thanks
> >>> Max
> >>>
> >>>> On Oct 17, 2016, at 9:07 PM, Uwe Schindler <uschind...@apache.org>
> >>> wrote:
> >>>>
> >>>> FYI,
> >>>>
> >>>> With:
> >>>> $ ant test -Dtests.useSecurityManager=true -Dargs='-
> >>> Djava.security.debug=access -
> Djdk.io.permissionsUseCanonicalPath=true'
> >>>>
> >>>> I can see the following line:
> >>>>
> >>>> [junit4]   2> access: access allowed ("java.io.FilePermission"
> >>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
> >>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
> >>>>
> >>>> So there is a difference!!!
> >>>>
> >>>> Uwe
> >>>>
> >>>> -----
> >>>> Uwe Schindler
> >>>> uschind...@apache.org
> >>>> ASF Member, Apache Lucene PMC / Committer
> >>>> Bremen, Germany
> >>>> http://lucene.apache.org/
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Uwe Schindler [mailto:uschind...@apache.org]
> >>>>> Sent: Monday, October 17, 2016 2:44 PM
> >>>>> To: dev@lucene.apache.org; 'Wang Weijun'
> <weijun.w...@oracle.com>
> >>>>> Cc: 'Dalibor Topic' <dalibor.to...@oracle.com>; 'Balchandra Vaidya'
> >>>>> <balchandra.vai...@oracle.com>; 'Muneer Kolarkunnu'
> >>>>> <abdul.kolarku...@oracle.com>; 'Rory O'Donnell'
> >>>>> <rory.odonn...@oracle.com>
> >>>>> Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-
> ea+140) -
> >>>>> Build # 18064 - Unstable!
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I attached the log of a test run. As you see it says "access denied"
> while
> >>>>> reading the morphologic JAR file:
> >>>>>
> >>>>> jenkins@serv1:~/workspace/Lucene-Solr-master-
> >>>>> Linux/lucene/analysis/morfologik$ ant test -Dargs='-
> >>>>> Djava.security.debug=access' -Dtestcase=TestMorfologikFilterFactory
> >>>>> [...]
> >>>>> [junit4]   2> access: access denied ("java.io.FilePermission"
> >>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
> >>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
> >>>>> [junit4]   2> access: access denied ("java.io.FilePermission"
> >>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
> >>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
> >>>>> [...]
> >>>>>
> >>>>> To reproduce, do the following:
> >>>>> Check out Lucene's repository from Github:
> >>>>> https://github.com/apache/lucene-solr
> >>>>>
> >>>>> Go to root directory of it and run (Apache Ant required): "ant ivy-
> >>> bootstrap" -
> >>>>>> this installs the IVY downloader. Be sure to have internet connection
> to
> >>>>> access Maven Central!
> >>>>>
> >>>>> The cd into lucene/analysis/morfologik. From there you can run tests
> >>> using
> >>>>> "ant test".
> >>>>>
> >>>>> To pass extra JDK args, pass e.g., $ ant test -Dargs='-
> >>>>> Djava.security.debug=access'
> >>>>> The test passes with the JDK option canonicalized paths.
> >>>>>
> >>>>> From the above error message you see that the code tries to read the
> JAR
> >>> file
> >>>>> of morphologic, which is completely outside the build  directory in the
> IVY
> >>>>> cache folder, but it is part of the
> >>>>>
> >>>>> I have seen that you have a short test for the same in your checkin to
> JDK,
> >>>>> but there is a slight difference: You are not using a JAR file, just 
> >>>>> plain
> class
> >>>>> files in your test. And this works also for Lucene. It is just
> >>> JARURLConnection
> >>>>> that breaks. The reason why other parts of Lucene don't fail is because
> >>>>> Lucene's own tests use class file (not yet packaged). In always fails
> when
> >>> you
> >>>>> have 3rd party JAR files.
> >>>>>
> >>>>> Uwe
> >>>>>
> >>>>> -----
> >>>>> Uwe Schindler
> >>>>> uschind...@apache.org
> >>>>> ASF Member, Apache Lucene PMC / Committer
> >>>>> Bremen, Germany
> >>>>> http://lucene.apache.org/
> >>>>>> -----Original Message-----
> >>>>>> From: Wang Weijun [mailto:weijun.w...@oracle.com]
> >>>>>> Sent: Monday, October 17, 2016 2:20 PM
> >>>>>> To: Uwe Schindler <u...@thetaphi.de>
> >>>>>> Cc: dev@lucene.apache.org; Dalibor Topic
> <dalibor.to...@oracle.com>;
> >>>>>> Balchandra Vaidya <balchandra.vai...@oracle.com>; Muneer
> >>> Kolarkunnu
> >>>>>> <abdul.kolarku...@oracle.com>; Rory O'Donnell
> >>>>>> <rory.odonn...@oracle.com>
> >>>>>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-
> ea+140)
> >>> -
> >>>>>> Build # 18064 - Unstable!
> >>>>>>
> >>>>>>
> >>>>>>> On Oct 17, 2016, at 6:24 PM, Rory O'Donnell
> >>>>> <rory.odonn...@oracle.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Adding Max to discussion.
> >>>>>>>
> >>>>>>> Rgds,Rory
> >>>>>>>
> >>>>>>>
> >>>>>>> On 17/10/2016 08:26, Dawid Weiss wrote:
> >>>>>>>> Yeah, this doesn't look so good. This class and the resource it tries
> >>>>>>>> to access are within the same JAR file -- if I understand correctly
> >>>>>>>> the JDK issue you quoted as the source of the problem, the code
> here
> >>>>>>>> should be able to pass regardless of file restrictions/ policies?
> >>>>>>
> >>>>>> Correct. A class should always be able to read a file that's on its
> >>> classpath,
> >>>>> no
> >>>>>> matter if it's inside a jar or a .class file.
> >>>>>>
> >>>>>> Can someone tell me how to reproduce this error? I have never
> played
> >>> with
> >>>>>> Lucene before. Also, please try adding -Djava.security.debug=access
> and
> >>>>> see
> >>>>>> if you can get some extra debug info.
> >>>>>>
> >>>>>> Thanks
> >>>>>> Max
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>> Dawid
> >>>>>>>>
> >>>>>>>> On Sun, Oct 16, 2016 at 10:09 PM, Uwe Schindler
> <u...@thetaphi.de>
> >>>>>> wrote:
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> I reverted the Lucene builds to build Java 9 138 for now. I will
> later
> >>>>> check
> >>>>>> if this also happens with build 139, which I have to download first. I
> will
> >>>>> also
> >>>>>> debug locally.
> >>>>>>>>>
> >>>>>>>>> The code fails because this code hits "null" on getResource() at
> >>>>>>
> >>>
> morfologik.stemming.polish.PolishStemmer.<init>(PolishStemmer.java:34)
> >>>>>>>>>
> >>>>>>>>> https://github.com/morfologik/morfologik-
> >>>>>> stemming/blob/master/morfologik-
> >>>>>>
> >>>>>
> >>>
> polish/src/main/java/morfologik/stemming/polish/PolishStemmer.java#L32
> >>>>>>>>>
> >>>>>>>>> This is impossible to happen, because the dict file is in same
> package.
> >>> I
> >>>>>> have no idea why this only fails here and not at other places in
> Lucene.
> >>> The
> >>>>>> main difference looks like the use of URL instead of
> >>> getResourceAsStream()
> >>>>>> like other places in Lucene.
> >>>>>>>>>
> >>>>>>>>> So this seems to be a major regression in Java 9 build 140.
> >>>>>>>>>
> >>>>>>>>> Uwe
> >>>>>>>>>
> >>>>>>>>> -----
> >>>>>>>>> Uwe Schindler
> >>>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
> >>>>>>>>> http://www.thetaphi.de
> >>>>>>>>> eMail: u...@thetaphi.de
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Uwe Schindler [mailto:u...@thetaphi.de]
> >>>>>>>>>> Sent: Sunday, October 16, 2016 8:38 PM
> >>>>>>>>>> To: dev@lucene.apache.org
> >>>>>>>>>> Cc: dalibor.to...@oracle.com; balchandra.vai...@oracle.com;
> >>>>>> 'Muneer
> >>>>>>>>>> Kolarkunnu' <abdul.kolarku...@oracle.com>; 'Dawid Weiss'
> >>>>>>>>>> <dawid.we...@cs.put.poznan.pl>; dev@lucene.apache.org
> >>>>>>>>>> Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-
> >>>>>> ea+140) -
> >>>>>>>>>> Build # 18064 - Unstable!
> >>>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> this seems to be a new regression in Java 9 ea build 140.
> >>> Interestingly
> >>>>>> this
> >>>>>>>>>> only affects 2 libraries (morphologic and commons-codec
> phonetic).
> >>>>> We
> >>>>>> use
> >>>>>>>>>> loading of resources from classloaders at many places; it is
> unclear
> >>> to
> >>>>>> me,
> >>>>>>>>>> why it only fails here. I will look into the code, but this is 
> >>>>>>>>>> outside
> of
> >>>>>> Lucene. I
> >>>>>>>>>> think it might be some crazyness like using context class loader
> in
> >>> non-
> >>>>>> proper
> >>>>>>>>>> ways or similar.
> >>>>>>>>>>
> >>>>>>>>>> Maybe it is a new bug in JDK 9 build 139 or build 140 (the last
> >>> working
> >>>>>> one
> >>>>>>>>>> was build 138).
> >>>>>>>>>>
> >>>>>>>>>> Uwe
> >>>>>>>>>>
> >>>>>>>>>> -----
> >>>>>>>>>> Uwe Schindler
> >>>>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
> >>>>>>>>>> http://www.thetaphi.de
> >>>>>>>>>> eMail: u...@thetaphi.de
> >>>>>>>>>>
> >>>>>>>>>>> -----Original Message-----
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to