One more thing:
You said "(although it should not be needed). Can you confirm that? I mean, if
you remove that line, does the test still passes with
-Djdk.io.permissionsUseCanonicalPath=true? If so, we will need to look into
other reasons, since I did see in many other places file permission on
/home/jenkins/..../morfologik-polish-2.1.0.jar itself is granted.
Thanks
Max
> On Oct 17, 2016, at 11:17 PM, Wang Weijun <weijun.w...@oracle.com> wrote:
>
> Yes, this is where the problem is.
>
> So it looks like the permission is granted in a policy file instead of being
> granted by the class loader itself. In this case, the path of the permission
> must match how you access that file.
>
> I'll think about your suggestion. However, can you guarantee the code always
> accesses the file using the canonicalized path? For example, suppose the
> actual file is /x, both /a and /b symlink to it, and you grant the permission
> on /a in a policy file. Even if I canonicalize /a to /x and grant permissions
> on both /a and /x, you will still see a failure if the code access /b.
>
> --Max
>
>> On Oct 17, 2016, at 11:03 PM, Uwe Schindler <uschind...@apache.org> wrote:
>>
>> Hi,
>>
>> I attached the debugging logs ("all") for the working (conaonicalized) and
>> non-working case to this mail. Please keep in mind that around the
>> getResource() is no doPrivileged, so it looks like it uses privileges of
>> outer JAR.
>>
>> I also checked, we have the following explicitely in our policy file, which
>> explicitly allows everything in the IVY cache (although it should not be
>> needed) - so it should really work anyways!!!
>>
>> permission java.io.FilePermission "${user.home}${/}.ivy2${/}cache${/}-",
>> "read";
>>
>> But I think I know the issue - which is really stupid - but it’s a fact
>> affecting maaaany people! Here is what the policy file parsing says above
>> the permission given before (please compare first line and second line):
>>
>> [junit4] 2> Active Principals: []
>> [junit4] 2> policy: granting ("java.io.FilePermission"
>> "/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/-" "read")
>> [junit4] 2> policy: granting ("java.io.FilePermission"
>> "/var/lib/jenkins/.ivy2/cache/-" "read")
>> [junit4] 2> policy: granting ("java.io.FilePermission"
>> "/home/jenkins/tools/java/64bit/jdk-9-ea+140/-" "read,execute")
>>
>> Unfortunately, the user changed its home directory, the old one
>> (var/lib/jenkins) symlinked to the new one, but /etc/passwd was not updated.
>> So ${user.home} looks different. This is the reason why it did not happen
>> locally on my windows machine!
>>
>> IMHO: When reading the policy file, it should canonicalize all paths,
>> especially stuff like ${user.home}. At runtime when actually checking the
>> permissions it should not canonicalize for performance reasons. Not sure if
>> this is a bug, but this could hit lots of people. I know that other Jenkins
>> servers where Lucene runs their tests (not yet on Java 9) are setup in
>> similar ways!
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: u...@thetaphi.de
>>
>>> -----Original Message-----
>>> From: Wang Weijun [mailto:weijun.w...@oracle.com]
>>> Sent: Monday, October 17, 2016 4:03 PM
>>> To: dev@lucene.apache.org
>>> Cc: Dalibor Topic <dalibor.to...@oracle.com>; Balchandra Vaidya
>>> <balchandra.vai...@oracle.com>; Muneer Kolarkunnu
>>> <abdul.kolarku...@oracle.com>; Rory O'Donnell
>>> <rory.odonn...@oracle.com>
>>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) -
>>> Build # 18064 - Unstable!
>>>
>>> OK, I'll try it tomorrow.
>>>
>>> At the same time can you try -Djava.security.debug=all or -
>>> Djava.security.debug=scl and see if it shows what permission is granted?
>>>
>>> Thanks
>>> Max
>>>
>>>> On Oct 17, 2016, at 9:07 PM, Uwe Schindler <uschind...@apache.org>
>>> wrote:
>>>>
>>>> FYI,
>>>>
>>>> With:
>>>> $ ant test -Dtests.useSecurityManager=true -Dargs='-
>>> Djava.security.debug=access -Djdk.io.permissionsUseCanonicalPath=true'
>>>>
>>>> I can see the following line:
>>>>
>>>> [junit4] 2> access: access allowed ("java.io.FilePermission"
>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
>>>>
>>>> So there is a difference!!!
>>>>
>>>> Uwe
>>>>
>>>> -----
>>>> Uwe Schindler
>>>> uschind...@apache.org
>>>> ASF Member, Apache Lucene PMC / Committer
>>>> Bremen, Germany
>>>> http://lucene.apache.org/
>>>>
>>>>> -----Original Message-----
>>>>> From: Uwe Schindler [mailto:uschind...@apache.org]
>>>>> Sent: Monday, October 17, 2016 2:44 PM
>>>>> To: dev@lucene.apache.org; 'Wang Weijun' <weijun.w...@oracle.com>
>>>>> Cc: 'Dalibor Topic' <dalibor.to...@oracle.com>; 'Balchandra Vaidya'
>>>>> <balchandra.vai...@oracle.com>; 'Muneer Kolarkunnu'
>>>>> <abdul.kolarku...@oracle.com>; 'Rory O'Donnell'
>>>>> <rory.odonn...@oracle.com>
>>>>> Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) -
>>>>> Build # 18064 - Unstable!
>>>>>
>>>>> Hi,
>>>>>
>>>>> I attached the log of a test run. As you see it says "access denied" while
>>>>> reading the morphologic JAR file:
>>>>>
>>>>> jenkins@serv1:~/workspace/Lucene-Solr-master-
>>>>> Linux/lucene/analysis/morfologik$ ant test -Dargs='-
>>>>> Djava.security.debug=access' -Dtestcase=TestMorfologikFilterFactory
>>>>> [...]
>>>>> [junit4] 2> access: access denied ("java.io.FilePermission"
>>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
>>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
>>>>> [junit4] 2> access: access denied ("java.io.FilePermission"
>>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
>>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
>>>>> [...]
>>>>>
>>>>> To reproduce, do the following:
>>>>> Check out Lucene's repository from Github:
>>>>> https://github.com/apache/lucene-solr
>>>>>
>>>>> Go to root directory of it and run (Apache Ant required): "ant ivy-
>>> bootstrap" -
>>>>>> this installs the IVY downloader. Be sure to have internet connection to
>>>>> access Maven Central!
>>>>>
>>>>> The cd into lucene/analysis/morfologik. From there you can run tests
>>> using
>>>>> "ant test".
>>>>>
>>>>> To pass extra JDK args, pass e.g., $ ant test -Dargs='-
>>>>> Djava.security.debug=access'
>>>>> The test passes with the JDK option canonicalized paths.
>>>>>
>>>>> From the above error message you see that the code tries to read the JAR
>>> file
>>>>> of morphologic, which is completely outside the build directory in the
>>>>> IVY
>>>>> cache folder, but it is part of the
>>>>>
>>>>> I have seen that you have a short test for the same in your checkin to
>>>>> JDK,
>>>>> but there is a slight difference: You are not using a JAR file, just
>>>>> plain class
>>>>> files in your test. And this works also for Lucene. It is just
>>> JARURLConnection
>>>>> that breaks. The reason why other parts of Lucene don't fail is because
>>>>> Lucene's own tests use class file (not yet packaged). In always fails when
>>> you
>>>>> have 3rd party JAR files.
>>>>>
>>>>> Uwe
>>>>>
>>>>> -----
>>>>> Uwe Schindler
>>>>> uschind...@apache.org
>>>>> ASF Member, Apache Lucene PMC / Committer
>>>>> Bremen, Germany
>>>>> http://lucene.apache.org/
>>>>>> -----Original Message-----
>>>>>> From: Wang Weijun [mailto:weijun.w...@oracle.com]
>>>>>> Sent: Monday, October 17, 2016 2:20 PM
>>>>>> To: Uwe Schindler <u...@thetaphi.de>
>>>>>> Cc: dev@lucene.apache.org; Dalibor Topic <dalibor.to...@oracle.com>;
>>>>>> Balchandra Vaidya <balchandra.vai...@oracle.com>; Muneer
>>> Kolarkunnu
>>>>>> <abdul.kolarku...@oracle.com>; Rory O'Donnell
>>>>>> <rory.odonn...@oracle.com>
>>>>>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140)
>>> -
>>>>>> Build # 18064 - Unstable!
>>>>>>
>>>>>>
>>>>>>> On Oct 17, 2016, at 6:24 PM, Rory O'Donnell
>>>>> <rory.odonn...@oracle.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Adding Max to discussion.
>>>>>>>
>>>>>>> Rgds,Rory
>>>>>>>
>>>>>>>
>>>>>>> On 17/10/2016 08:26, Dawid Weiss wrote:
>>>>>>>> Yeah, this doesn't look so good. This class and the resource it tries
>>>>>>>> to access are within the same JAR file -- if I understand correctly
>>>>>>>> the JDK issue you quoted as the source of the problem, the code here
>>>>>>>> should be able to pass regardless of file restrictions/ policies?
>>>>>>
>>>>>> Correct. A class should always be able to read a file that's on its
>>> classpath,
>>>>> no
>>>>>> matter if it's inside a jar or a .class file.
>>>>>>
>>>>>> Can someone tell me how to reproduce this error? I have never played
>>> with
>>>>>> Lucene before. Also, please try adding -Djava.security.debug=access and
>>>>> see
>>>>>> if you can get some extra debug info.
>>>>>>
>>>>>> Thanks
>>>>>> Max
>>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> Dawid
>>>>>>>>
>>>>>>>> On Sun, Oct 16, 2016 at 10:09 PM, Uwe Schindler <u...@thetaphi.de>
>>>>>> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I reverted the Lucene builds to build Java 9 138 for now. I will later
>>>>> check
>>>>>> if this also happens with build 139, which I have to download first. I
>>>>>> will
>>>>> also
>>>>>> debug locally.
>>>>>>>>>
>>>>>>>>> The code fails because this code hits "null" on getResource() at
>>>>>>
>>> morfologik.stemming.polish.PolishStemmer.<init>(PolishStemmer.java:34)
>>>>>>>>>
>>>>>>>>> https://github.com/morfologik/morfologik-
>>>>>> stemming/blob/master/morfologik-
>>>>>>
>>>>>
>>> polish/src/main/java/morfologik/stemming/polish/PolishStemmer.java#L32
>>>>>>>>>
>>>>>>>>> This is impossible to happen, because the dict file is in same
>>>>>>>>> package.
>>> I
>>>>>> have no idea why this only fails here and not at other places in Lucene.
>>> The
>>>>>> main difference looks like the use of URL instead of
>>> getResourceAsStream()
>>>>>> like other places in Lucene.
>>>>>>>>>
>>>>>>>>> So this seems to be a major regression in Java 9 build 140.
>>>>>>>>>
>>>>>>>>> Uwe
>>>>>>>>>
>>>>>>>>> -----
>>>>>>>>> Uwe Schindler
>>>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>>>>>>>> http://www.thetaphi.de
>>>>>>>>> eMail: u...@thetaphi.de
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Uwe Schindler [mailto:u...@thetaphi.de]
>>>>>>>>>> Sent: Sunday, October 16, 2016 8:38 PM
>>>>>>>>>> To: dev@lucene.apache.org
>>>>>>>>>> Cc: dalibor.to...@oracle.com; balchandra.vai...@oracle.com;
>>>>>> 'Muneer
>>>>>>>>>> Kolarkunnu' <abdul.kolarku...@oracle.com>; 'Dawid Weiss'
>>>>>>>>>> <dawid.we...@cs.put.poznan.pl>; dev@lucene.apache.org
>>>>>>>>>> Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-
>>>>>> ea+140) -
>>>>>>>>>> Build # 18064 - Unstable!
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> this seems to be a new regression in Java 9 ea build 140.
>>> Interestingly
>>>>>> this
>>>>>>>>>> only affects 2 libraries (morphologic and commons-codec phonetic).
>>>>> We
>>>>>> use
>>>>>>>>>> loading of resources from classloaders at many places; it is unclear
>>> to
>>>>>> me,
>>>>>>>>>> why it only fails here. I will look into the code, but this is
>>>>>>>>>> outside of
>>>>>> Lucene. I
>>>>>>>>>> think it might be some crazyness like using context class loader in
>>> non-
>>>>>> proper
>>>>>>>>>> ways or similar.
>>>>>>>>>>
>>>>>>>>>> Maybe it is a new bug in JDK 9 build 139 or build 140 (the last
>>> working
>>>>>> one
>>>>>>>>>> was build 138).
>>>>>>>>>>
>>>>>>>>>> Uwe
>>>>>>>>>>
>>>>>>>>>> -----
>>>>>>>>>> Uwe Schindler
>>>>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>>>>>>>>> http://www.thetaphi.de
>>>>>>>>>> eMail: u...@thetaphi.de
>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
