> On Oct 17, 2016, at 11:51 PM, Uwe Schindler <uschind...@apache.org> wrote:
> 
> Hi,
> 
>> One more thing:
>> 
>> You said "(although it should not be needed). Can you confirm that? I mean,
>> if you remove that line, does the test still passes with -
>> Djdk.io.permissionsUseCanonicalPath=true? If so, we will need to look into
>> other reasons, since I did see in many other places file permission on
>> /home/jenkins/..../morfologik-polish-2.1.0.jar itself is granted.
> 
> See my other mail! It should not be needed, because Class#getResource does 
> not throw or document any Security implications. But in fact if you don't 
> wrap a doPrivileged around the call, it will also fail without our extra 
> permission. It was added to work around this issue with 3rd party JARs.

You mean this was observed before the new FilePermission change in 8164705?

If so, this means before the code change, you need FilePermissions on both 
/var/lib/.../- and /home/jenkins/.../thejar to make sure the jar is accessible. 
Here it seems the 2nd one should be enough but actually not, and the 1st is 
added to guarantee it. And after the code change, the 1st is useless and the 
test starts to fail.

I would consider adding a compatibility layer to make the 1st also useful, but 
at the same time, if we can fix the 2nd, that will be very helpful.

Thanks
Max

   
> 
> I just blew up with the symlink on home dir.
> 
> Uwe
> 
>> Thanks
>> Max
>> 
>> 
>>> On Oct 17, 2016, at 11:17 PM, Wang Weijun <weijun.w...@oracle.com>
>> wrote:
>>> 
>>> Yes, this is where the problem is.
>>> 
>>> So it looks like the permission is granted in a policy file instead of being
>> granted by the class loader itself. In this case, the path of the permission
>> must match how you access that file.
>>> 
>>> I'll think about your suggestion. However, can you guarantee the code
>> always accesses the file using the canonicalized path? For example, suppose
>> the actual file is /x, both /a and /b symlink to it, and you grant the 
>> permission
>> on /a in a policy file. Even if I canonicalize /a to /x and grant 
>> permissions on
>> both /a and /x, you will still see a failure if the code access /b.
>>> 
>>> --Max
>>> 
>>>> On Oct 17, 2016, at 11:03 PM, Uwe Schindler <uschind...@apache.org>
>> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I attached the debugging logs ("all") for the working (conaonicalized) and
>> non-working case to this mail. Please keep in mind that around the
>> getResource() is no doPrivileged, so it looks like it uses privileges of 
>> outer JAR.
>>>> 
>>>> I also checked, we have the following explicitely in our policy file, which
>> explicitly allows everything in the IVY cache (although it should not be
>> needed) - so it should really work anyways!!!
>>>> 
>>>> permission java.io.FilePermission "${user.home}${/}.ivy2${/}cache${/}-",
>> "read";
>>>> 
>>>> But I think I know the issue - which is really stupid - but it’s a fact 
>>>> affecting
>> maaaany people! Here is what the policy file parsing says above the
>> permission given before (please compare first line and second line):
>>>> 
>>>> [junit4]   2>      Active Principals: []
>>>> [junit4]   2> policy:   granting ("java.io.FilePermission"
>> "/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/-" "read")
>>>> [junit4]   2> policy:   granting ("java.io.FilePermission"
>> "/var/lib/jenkins/.ivy2/cache/-" "read")
>>>> [junit4]   2> policy:   granting ("java.io.FilePermission"
>> "/home/jenkins/tools/java/64bit/jdk-9-ea+140/-" "read,execute")
>>>> 
>>>> Unfortunately, the user changed its home directory, the old one
>> (var/lib/jenkins) symlinked to the new one, but /etc/passwd was not
>> updated. So ${user.home} looks different. This is the reason why it did not
>> happen locally on my windows machine!
>>>> 
>>>> IMHO: When reading the policy file, it should canonicalize all paths,
>> especially stuff like ${user.home}. At runtime when actually checking the
>> permissions it should not canonicalize for performance reasons. Not sure if
>> this is a bug, but this could hit lots of people. I know that other Jenkins
>> servers where Lucene runs their tests (not yet on Java 9) are setup in 
>> similar
>> ways!
>>>> 
>>>> Uwe
>>>> 
>>>> -----
>>>> Uwe Schindler
>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>>> http://www.thetaphi.de
>>>> eMail: u...@thetaphi.de
>>>> 
>>>>> -----Original Message-----
>>>>> From: Wang Weijun [mailto:weijun.w...@oracle.com]
>>>>> Sent: Monday, October 17, 2016 4:03 PM
>>>>> To: dev@lucene.apache.org
>>>>> Cc: Dalibor Topic <dalibor.to...@oracle.com>; Balchandra Vaidya
>>>>> <balchandra.vai...@oracle.com>; Muneer Kolarkunnu
>>>>> <abdul.kolarku...@oracle.com>; Rory O'Donnell
>>>>> <rory.odonn...@oracle.com>
>>>>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140)
>> -
>>>>> Build # 18064 - Unstable!
>>>>> 
>>>>> OK, I'll try it tomorrow.
>>>>> 
>>>>> At the same time can you try -Djava.security.debug=all or -
>>>>> Djava.security.debug=scl and see if it shows what permission is granted?
>>>>> 
>>>>> Thanks
>>>>> Max
>>>>> 
>>>>>> On Oct 17, 2016, at 9:07 PM, Uwe Schindler <uschind...@apache.org>
>>>>> wrote:
>>>>>> 
>>>>>> FYI,
>>>>>> 
>>>>>> With:
>>>>>> $ ant test -Dtests.useSecurityManager=true -Dargs='-
>>>>> Djava.security.debug=access -
>> Djdk.io.permissionsUseCanonicalPath=true'
>>>>>> 
>>>>>> I can see the following line:
>>>>>> 
>>>>>> [junit4]   2> access: access allowed ("java.io.FilePermission"
>>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
>>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
>>>>>> 
>>>>>> So there is a difference!!!
>>>>>> 
>>>>>> Uwe
>>>>>> 
>>>>>> -----
>>>>>> Uwe Schindler
>>>>>> uschind...@apache.org
>>>>>> ASF Member, Apache Lucene PMC / Committer
>>>>>> Bremen, Germany
>>>>>> http://lucene.apache.org/
>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: Uwe Schindler [mailto:uschind...@apache.org]
>>>>>>> Sent: Monday, October 17, 2016 2:44 PM
>>>>>>> To: dev@lucene.apache.org; 'Wang Weijun'
>> <weijun.w...@oracle.com>
>>>>>>> Cc: 'Dalibor Topic' <dalibor.to...@oracle.com>; 'Balchandra Vaidya'
>>>>>>> <balchandra.vai...@oracle.com>; 'Muneer Kolarkunnu'
>>>>>>> <abdul.kolarku...@oracle.com>; 'Rory O'Donnell'
>>>>>>> <rory.odonn...@oracle.com>
>>>>>>> Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-
>> ea+140) -
>>>>>>> Build # 18064 - Unstable!
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I attached the log of a test run. As you see it says "access denied"
>> while
>>>>>>> reading the morphologic JAR file:
>>>>>>> 
>>>>>>> jenkins@serv1:~/workspace/Lucene-Solr-master-
>>>>>>> Linux/lucene/analysis/morfologik$ ant test -Dargs='-
>>>>>>> Djava.security.debug=access' -Dtestcase=TestMorfologikFilterFactory
>>>>>>> [...]
>>>>>>> [junit4]   2> access: access denied ("java.io.FilePermission"
>>>>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
>>>>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
>>>>>>> [junit4]   2> access: access denied ("java.io.FilePermission"
>>>>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
>>>>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
>>>>>>> [...]
>>>>>>> 
>>>>>>> To reproduce, do the following:
>>>>>>> Check out Lucene's repository from Github:
>>>>>>> https://github.com/apache/lucene-solr
>>>>>>> 
>>>>>>> Go to root directory of it and run (Apache Ant required): "ant ivy-
>>>>> bootstrap" -
>>>>>>>> this installs the IVY downloader. Be sure to have internet connection
>> to
>>>>>>> access Maven Central!
>>>>>>> 
>>>>>>> The cd into lucene/analysis/morfologik. From there you can run tests
>>>>> using
>>>>>>> "ant test".
>>>>>>> 
>>>>>>> To pass extra JDK args, pass e.g., $ ant test -Dargs='-
>>>>>>> Djava.security.debug=access'
>>>>>>> The test passes with the JDK option canonicalized paths.
>>>>>>> 
>>>>>>> From the above error message you see that the code tries to read the
>> JAR
>>>>> file
>>>>>>> of morphologic, which is completely outside the build  directory in the
>> IVY
>>>>>>> cache folder, but it is part of the
>>>>>>> 
>>>>>>> I have seen that you have a short test for the same in your checkin to
>> JDK,
>>>>>>> but there is a slight difference: You are not using a JAR file, just 
>>>>>>> plain
>> class
>>>>>>> files in your test. And this works also for Lucene. It is just
>>>>> JARURLConnection
>>>>>>> that breaks. The reason why other parts of Lucene don't fail is because
>>>>>>> Lucene's own tests use class file (not yet packaged). In always fails
>> when
>>>>> you
>>>>>>> have 3rd party JAR files.
>>>>>>> 
>>>>>>> Uwe
>>>>>>> 
>>>>>>> -----
>>>>>>> Uwe Schindler
>>>>>>> uschind...@apache.org
>>>>>>> ASF Member, Apache Lucene PMC / Committer
>>>>>>> Bremen, Germany
>>>>>>> http://lucene.apache.org/
>>>>>>>> -----Original Message-----
>>>>>>>> From: Wang Weijun [mailto:weijun.w...@oracle.com]
>>>>>>>> Sent: Monday, October 17, 2016 2:20 PM
>>>>>>>> To: Uwe Schindler <u...@thetaphi.de>
>>>>>>>> Cc: dev@lucene.apache.org; Dalibor Topic
>> <dalibor.to...@oracle.com>;
>>>>>>>> Balchandra Vaidya <balchandra.vai...@oracle.com>; Muneer
>>>>> Kolarkunnu
>>>>>>>> <abdul.kolarku...@oracle.com>; Rory O'Donnell
>>>>>>>> <rory.odonn...@oracle.com>
>>>>>>>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-
>> ea+140)
>>>>> -
>>>>>>>> Build # 18064 - Unstable!
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Oct 17, 2016, at 6:24 PM, Rory O'Donnell
>>>>>>> <rory.odonn...@oracle.com>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Adding Max to discussion.
>>>>>>>>> 
>>>>>>>>> Rgds,Rory
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 17/10/2016 08:26, Dawid Weiss wrote:
>>>>>>>>>> Yeah, this doesn't look so good. This class and the resource it tries
>>>>>>>>>> to access are within the same JAR file -- if I understand correctly
>>>>>>>>>> the JDK issue you quoted as the source of the problem, the code
>> here
>>>>>>>>>> should be able to pass regardless of file restrictions/ policies?
>>>>>>>> 
>>>>>>>> Correct. A class should always be able to read a file that's on its
>>>>> classpath,
>>>>>>> no
>>>>>>>> matter if it's inside a jar or a .class file.
>>>>>>>> 
>>>>>>>> Can someone tell me how to reproduce this error? I have never
>> played
>>>>> with
>>>>>>>> Lucene before. Also, please try adding -Djava.security.debug=access
>> and
>>>>>>> see
>>>>>>>> if you can get some extra debug info.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> Max
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Dawid
>>>>>>>>>> 
>>>>>>>>>> On Sun, Oct 16, 2016 at 10:09 PM, Uwe Schindler
>> <u...@thetaphi.de>
>>>>>>>> wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> I reverted the Lucene builds to build Java 9 138 for now. I will
>> later
>>>>>>> check
>>>>>>>> if this also happens with build 139, which I have to download first. I
>> will
>>>>>>> also
>>>>>>>> debug locally.
>>>>>>>>>>> 
>>>>>>>>>>> The code fails because this code hits "null" on getResource() at
>>>>>>>> 
>>>>> 
>> morfologik.stemming.polish.PolishStemmer.<init>(PolishStemmer.java:34)
>>>>>>>>>>> 
>>>>>>>>>>> https://github.com/morfologik/morfologik-
>>>>>>>> stemming/blob/master/morfologik-
>>>>>>>> 
>>>>>>> 
>>>>> 
>> polish/src/main/java/morfologik/stemming/polish/PolishStemmer.java#L32
>>>>>>>>>>> 
>>>>>>>>>>> This is impossible to happen, because the dict file is in same
>> package.
>>>>> I
>>>>>>>> have no idea why this only fails here and not at other places in
>> Lucene.
>>>>> The
>>>>>>>> main difference looks like the use of URL instead of
>>>>> getResourceAsStream()
>>>>>>>> like other places in Lucene.
>>>>>>>>>>> 
>>>>>>>>>>> So this seems to be a major regression in Java 9 build 140.
>>>>>>>>>>> 
>>>>>>>>>>> Uwe
>>>>>>>>>>> 
>>>>>>>>>>> -----
>>>>>>>>>>> Uwe Schindler
>>>>>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>>>>>>>>>> http://www.thetaphi.de
>>>>>>>>>>> eMail: u...@thetaphi.de
>>>>>>>>>>> 
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Uwe Schindler [mailto:u...@thetaphi.de]
>>>>>>>>>>>> Sent: Sunday, October 16, 2016 8:38 PM
>>>>>>>>>>>> To: dev@lucene.apache.org
>>>>>>>>>>>> Cc: dalibor.to...@oracle.com; balchandra.vai...@oracle.com;
>>>>>>>> 'Muneer
>>>>>>>>>>>> Kolarkunnu' <abdul.kolarku...@oracle.com>; 'Dawid Weiss'
>>>>>>>>>>>> <dawid.we...@cs.put.poznan.pl>; dev@lucene.apache.org
>>>>>>>>>>>> Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-
>>>>>>>> ea+140) -
>>>>>>>>>>>> Build # 18064 - Unstable!
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> this seems to be a new regression in Java 9 ea build 140.
>>>>> Interestingly
>>>>>>>> this
>>>>>>>>>>>> only affects 2 libraries (morphologic and commons-codec
>> phonetic).
>>>>>>> We
>>>>>>>> use
>>>>>>>>>>>> loading of resources from classloaders at many places; it is
>> unclear
>>>>> to
>>>>>>>> me,
>>>>>>>>>>>> why it only fails here. I will look into the code, but this is 
>>>>>>>>>>>> outside
>> of
>>>>>>>> Lucene. I
>>>>>>>>>>>> think it might be some crazyness like using context class loader
>> in
>>>>> non-
>>>>>>>> proper
>>>>>>>>>>>> ways or similar.
>>>>>>>>>>>> 
>>>>>>>>>>>> Maybe it is a new bug in JDK 9 build 139 or build 140 (the last
>>>>> working
>>>>>>>> one
>>>>>>>>>>>> was build 138).
>>>>>>>>>>>> 
>>>>>>>>>>>> Uwe
>>>>>>>>>>>> 
>>>>>>>>>>>> -----
>>>>>>>>>>>> Uwe Schindler
>>>>>>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>>>>>>>>>>> http://www.thetaphi.de
>>>>>>>>>>>> eMail: u...@thetaphi.de
>>>>>>>>>>>> 
>>>>>>>>>>>>> -----Original Message-----
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to