Re: Performance in Solr 9 / Java 11

[Tomoko Uchida]

I believe mr-jar build is enabled in the 8x branch (LUCENE-7966), and the
workaround was dropped on the master branch when the minimum java version
was bumped up to java 11 (LUCENE-8738); if my understanding is correct.

$ jar tf core/lucene-core-8.6.1.jar | grep META-INF/versions
META-INF/versions/
META-INF/versions/9/
META-INF/versions/9/org/
META-INF/versions/9/org/apache/
...

$ jar tf core/build/libs/lucene-core-9.0.0-SNAPSHOT.jar | grep
META-INF/versions
// no outputs




2020年8月30日(日) 6:48 Mike Drob :

> Do you know if these mr-jars are built by default as part of the release
> process? I definitely had no idea about them when doing 8.5.2 and did not
> even think to verify anything about it.
>
> On Sat, Aug 29, 2020 at 4:05 PM Adrien Grand  wrote:
>
>> It may only be indirectly related to your question, but there is support
>> for vectorized operations of byte[] arrays that was added in JDK 13 (this
>> blog https://richardstartin.github.io/posts/vectorised-byte-operations 
>> explains
>> well what it is about) that we started leveraging for compressing terms
>> dictionaries in Lucene 8.5:
>> https://issues.apache.org/jira/browse/LUCENE-4702.
>>
>> I don't know how well this is known but our build also has logic to
>> create multi-release JARs. We don't use it in master today but it's used on
>> branch_8x, which requires Java 8, in order to use APIs that were introduced
>> in Java 9 such as Arrays#mismatch. See the "patch-mr-jar" target in the
>> branch_8x build:
>> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/common-build.xml#L602.
>> So if APIs that could help performance were introduced in say JDK 15, we
>> might still be able to leverage them in Lucene/Solr 9 using the same
>> mechanism.
>>
>>
>>
>> On Tue, Aug 11, 2020 at 1:12 AM Marcus Eagan 
>> wrote:
>>
>>> In my IDE, I have a few profiling tools that I bounce between that I
>>> started using in my work at Lucidworks but I continue to use in my current
>>> work today. I have suspicions that there may be some performance
>>> improvements in Java 11 that we can exploit further.  I'm curious as to if
>>> there has been any investigation, possibly Mark Miller or
>>> @u...@thetaphi.de ,  into performance improvements
>>> specific to the newer version of Java in Master? There are some obvious
>>> ones that we get for free, like a better GC, but curious as to prior work
>>> in this area before publishing anything that might be redundant or
>>> irrelevant.
>>>
>>> Best,
>>>
>>> --
>>> Marcus Eagan
>>>
>>>
>>>
>>>
>>
>> --
>> Adrien
>>
>>
>>

Re: Performance in Solr 9 / Java 11

[Mike Drob]

Do you know if these mr-jars are built by default as part of the release
process? I definitely had no idea about them when doing 8.5.2 and did not
even think to verify anything about it.

On Sat, Aug 29, 2020 at 4:05 PM Adrien Grand  wrote:

> It may only be indirectly related to your question, but there is support
> for vectorized operations of byte[] arrays that was added in JDK 13 (this
> blog https://richardstartin.github.io/posts/vectorised-byte-operations 
> explains
> well what it is about) that we started leveraging for compressing terms
> dictionaries in Lucene 8.5:
> https://issues.apache.org/jira/browse/LUCENE-4702.
>
> I don't know how well this is known but our build also has logic to create
> multi-release JARs. We don't use it in master today but it's used on
> branch_8x, which requires Java 8, in order to use APIs that were introduced
> in Java 9 such as Arrays#mismatch. See the "patch-mr-jar" target in the
> branch_8x build:
> https://github.com/apache/lucene-solr/blob/branch_8x/lucene/common-build.xml#L602.
> So if APIs that could help performance were introduced in say JDK 15, we
> might still be able to leverage them in Lucene/Solr 9 using the same
> mechanism.
>
>
>
> On Tue, Aug 11, 2020 at 1:12 AM Marcus Eagan 
> wrote:
>
>> In my IDE, I have a few profiling tools that I bounce between that I
>> started using in my work at Lucidworks but I continue to use in my current
>> work today. I have suspicions that there may be some performance
>> improvements in Java 11 that we can exploit further.  I'm curious as to if
>> there has been any investigation, possibly Mark Miller or
>> @u...@thetaphi.de ,  into performance improvements
>> specific to the newer version of Java in Master? There are some obvious
>> ones that we get for free, like a better GC, but curious as to prior work
>> in this area before publishing anything that might be redundant or
>> irrelevant.
>>
>> Best,
>>
>> --
>> Marcus Eagan
>>
>>
>>
>>
>
> --
> Adrien
>
>
>

Re: Performance in Solr 9 / Java 11

[Adrien Grand]

It may only be indirectly related to your question, but there is support
for vectorized operations of byte[] arrays that was added in JDK 13 (this
blog https://richardstartin.github.io/posts/vectorised-byte-operations explains
well what it is about) that we started leveraging for compressing terms
dictionaries in Lucene 8.5:
https://issues.apache.org/jira/browse/LUCENE-4702.

I don't know how well this is known but our build also has logic to create
multi-release JARs. We don't use it in master today but it's used on
branch_8x, which requires Java 8, in order to use APIs that were introduced
in Java 9 such as Arrays#mismatch. See the "patch-mr-jar" target in the
branch_8x build:
https://github.com/apache/lucene-solr/blob/branch_8x/lucene/common-build.xml#L602.
So if APIs that could help performance were introduced in say JDK 15, we
might still be able to leverage them in Lucene/Solr 9 using the same
mechanism.



On Tue, Aug 11, 2020 at 1:12 AM Marcus Eagan  wrote:

> In my IDE, I have a few profiling tools that I bounce between that I
> started using in my work at Lucidworks but I continue to use in my current
> work today. I have suspicions that there may be some performance
> improvements in Java 11 that we can exploit further.  I'm curious as to if
> there has been any investigation, possibly Mark Miller or @u...@thetaphi.de
> ,  into performance improvements specific to the newer
> version of Java in Master? There are some obvious ones that we get for
> free, like a better GC, but curious as to prior work in this area before
> publishing anything that might be redundant or irrelevant.
>
> Best,
>
> --
> Marcus Eagan
>
>

-- 
Adrien

Re: Annoying but harmless exceptions due to filepermissions when running tests

[Erick Erickson]

Well, as Uwe and I discussed offline, he’s right and I’m wrong.

In CoreContainer [364] there’s code like this:

Path userFilesPath = return solrHome.resolve("userfiles"); // TODO make 
configurable on cfg?
try {
  Files.createDirectories(userFilesPath); // does nothing if already exists
} catch (Exception e) {
  log.warn("Unable to create [{}].  Features requiring this directory may 
fail.", userFilesPath, e);
}

So I assumed it would happen on most every test, at least in cloud mode. But 
when I tried to make it happen on a different test, there was no exception.

I’ll have to poke some more to see what’s really happening…

Never Mind….

> On Aug 29, 2020, at 8:59 AM, Uwe Schindler  wrote:
> 
> Hi,
> 
> this is a bug in the test! It should never ever modify files outside its 
> sandbox. It should not even modify files in build directory. It is fully 
> valid to reject those writes - has nothing to do with users, it's just 
> forbidden by the test framework. Modifying this file is harmful, as it may 
> affect later tests.
> 
> So the correct way is to copy those files to the solr container running 
> inside test's sandbox. That's one of the main advantages of the Test sandbox: 
> No write access anywhere outside the test, see policy file.
> 
> Uwe
> 
> -
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: u...@thetaphi.de
> 
>> -Original Message-
>> From: Erick Erickson 
>> Sent: Saturday, August 29, 2020 2:54 PM
>> To: dev@lucene.apache.org
>> Subject: Annoying but harmless exceptions due to filepermissions when running
>> tests
>> 
>> These exceptions are handled in the code and don’t affect running tests, but
>> they can be a distraction when trying to figure out what’s causing a failure.
>> When CoreContainer is being initialized, these two paths get “Permission
>> Denied” errors since they try to create directories/files.
>> 
>> java.security.AccessControlException: access denied ("java.io.FilePermission"
>> "/Users/Erick/apache/solrJiras/master/solr/core/build/resources/test/solr/filest
>> ore" "write”)
>> 
>> java.security.AccessControlException: access denied ("java.io.FilePermission"
>> "/Users/Erick/apache/solrJiras/master/solr/core/build/resources/test/solr/userf
>> iles" "write”)
>> 
>> In both cases, the code logs a warning like "Features requiring this 
>> directory
>> may fail”.
>> 
>> “build” is permitted this way, so I guess gradle runs as some other user?
>> 
>> drwxr-xr-x  11 Erick  staff352 Aug 28 09:30 build
>> 
>> Any hints on an easy way to avoid these? It’s not worth much effort I don’t
>> think since they’re handled, but if it’s easy I’d like to not see them.
>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: Annoying but harmless exceptions due to filepermissions when running tests

[Uwe Schindler]

Hi,

this is a bug in the test! It should never ever modify files outside its 
sandbox. It should not even modify files in build directory. It is fully valid 
to reject those writes - has nothing to do with users, it's just forbidden by 
the test framework. Modifying this file is harmful, as it may affect later 
tests.

So the correct way is to copy those files to the solr container running inside 
test's sandbox. That's one of the main advantages of the Test sandbox: No write 
access anywhere outside the test, see policy file.

Uwe

-
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de

> -Original Message-
> From: Erick Erickson 
> Sent: Saturday, August 29, 2020 2:54 PM
> To: dev@lucene.apache.org
> Subject: Annoying but harmless exceptions due to filepermissions when running
> tests
> 
> These exceptions are handled in the code and don’t affect running tests, but
> they can be a distraction when trying to figure out what’s causing a failure.
> When CoreContainer is being initialized, these two paths get “Permission
> Denied” errors since they try to create directories/files.
> 
> java.security.AccessControlException: access denied ("java.io.FilePermission"
> "/Users/Erick/apache/solrJiras/master/solr/core/build/resources/test/solr/filest
> ore" "write”)
> 
> java.security.AccessControlException: access denied ("java.io.FilePermission"
> "/Users/Erick/apache/solrJiras/master/solr/core/build/resources/test/solr/userf
> iles" "write”)
> 
> In both cases, the code logs a warning like "Features requiring this directory
> may fail”.
> 
> “build” is permitted this way, so I guess gradle runs as some other user?
> 
> drwxr-xr-x  11 Erick  staff352 Aug 28 09:30 build
> 
> Any hints on an easy way to avoid these? It’s not worth much effort I don’t
> think since they’re handled, but if it’s easy I’d like to not see them.
> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Annoying but harmless exceptions due to filepermissions when running tests

[Erick Erickson]

These exceptions are handled in the code and don’t affect running tests, but 
they can be a distraction when trying to figure out what’s causing a failure. 
When CoreContainer is being initialized, these two paths get “Permission 
Denied” errors since they try to create directories/files.

java.security.AccessControlException: access denied ("java.io.FilePermission" 
"/Users/Erick/apache/solrJiras/master/solr/core/build/resources/test/solr/filestore"
 "write”)

java.security.AccessControlException: access denied ("java.io.FilePermission" 
"/Users/Erick/apache/solrJiras/master/solr/core/build/resources/test/solr/userfiles"
 "write”)

In both cases, the code logs a warning like "Features requiring this directory 
may fail”.

“build” is permitted this way, so I guess gradle runs as some other user?

drwxr-xr-x  11 Erick  staff352 Aug 28 09:30 build

Any hints on an easy way to avoid these? It’s not worth much effort I don’t 
think since they’re handled, but if it’s easy I’d like to not see them.



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: With ant removed, will GitHub PR job always fail 1st check?

[Uwe Schindler]

Thanks, I had no idea where this crazy script is configured. 

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: Tomás Fernández Löbbe  
Sent: Saturday, August 29, 2020 4:49 AM
To: dev@lucene.apache.org
Subject: Re: With ant removed, will GitHub PR job always fail 1st check?

 

I have this PR to remove the ant action:

https://github.com/apache/lucene-solr/pull/1798

 

On Fri, Aug 28, 2020 at 6:58 PM Alexandre Rafalovitch mailto:arafa...@gmail.com> > wrote:

For the Pull Request, GitHub is running both Ant and Gradle precommit.



Now that ant is gone, it is probably safe to remove that check as

well. It will always fail at "Ivy bootstrap" phase.



Regards,

   Alex.



-

To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org 
 

For additional commands, e-mail: dev-h...@lucene.apache.org