Re: [WARNING] Index corruption and crashes in Apache Lucene Core / Apache Solr with Java 7

2011-07-29 Thread Sanne Grinovero
Hello,
thanks for the warning, that's a pretty nasty bug.

A patch was made for OpenJDK, if anybody is interested to try it out
that would be great:
http://hg.openjdk.java.net/hsx/hotspot-comp/hotspot/rev/4e761e7e6e12

Regards,
Sanne

2011/7/28 Uwe Schindler uschind...@apache.org:
 Hello Apache Lucene  Apache Solr users,
 Hello users of other Java-based Apache projects,

 Oracle released Java 7 today. Unfortunately it contains hotspot compiler
 optimizations, which miscompile some loops. This can affect code of several
 Apache projects. Sometimes JVMs only crash, but in several cases, results
 calculated can be incorrect, leading to bugs in applications (see Hotspot
 bugs 7070134 [1], 7044738 [2], 7068051 [3]).

 Apache Lucene Core and Apache Solr are two Apache projects, which are
 affected by these bugs, namely all versions released until today. Solr users
 with the default configuration will have Java crashing with SIGSEGV as soon
 as they start to index documents, as one affected part is the well-known
 Porter stemmer (see LUCENE-3335 [4]). Other loops in Lucene may be
 miscompiled, too, leading to index corruption (especially on Lucene trunk
 with pulsing codec; other loops may be affected, too - LUCENE-3346 [5]).

 These problems were detected only 5 days before the official Java 7 release,
 so Oracle had no time to fix those bugs, affecting also many more
 applications. In response to our questions, they proposed to include the
 fixes into service release u2 (eventually into service release u1, see [6]).
 This means you cannot use Apache Lucene/Solr with Java 7 releases before
 Update 2! If you do, please don't open bug reports, it is not the
 committers' fault! At least disable loop optimizations using the
 -XX:-UseLoopPredicate JVM option to not risk index corruptions.

 Please note: Also Java 6 users are affected, if they use one of those JVM
 options, which are not enabled by default: -XX:+OptimizeStringConcat or
 -XX:+AggressiveOpts

 It is strongly recommended not to use any hotspot optimization switches in
 any Java version without extensive testing!

 In case you upgrade to Java 7, remember that you may have to reindex, as the
 unicode version shipped with Java 7 changed and tokenization behaves
 differently (e.g. lowercasing). For more information, read
 JRE_VERSION_MIGRATION.txt in your distribution package!

 On behalf of the Lucene project,
 Uwe

 [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7070134
 [2] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7044738
 [3] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7068051
 [4] https://issues.apache.org/jira/browse/LUCENE-3335
 [5] https://issues.apache.org/jira/browse/LUCENE-3346
 [6] http://s.apache.org/StQ

 -
 Uwe Schindler
 uschind...@apache.org
 Apache Lucene PMC Member / Committer
 Bremen, Germany
 http://lucene.apache.org/



 -
 To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
 For additional commands, e-mail: java-user-h...@lucene.apache.org




[WARNING] Index corruption and crashes in Apache Lucene Core / Apache Solr with Java 7

2011-07-28 Thread Uwe Schindler
Hello Apache Lucene  Apache Solr users,
Hello users of other Java-based Apache projects,

Oracle released Java 7 today. Unfortunately it contains hotspot compiler
optimizations, which miscompile some loops. This can affect code of several
Apache projects. Sometimes JVMs only crash, but in several cases, results
calculated can be incorrect, leading to bugs in applications (see Hotspot
bugs 7070134 [1], 7044738 [2], 7068051 [3]).

Apache Lucene Core and Apache Solr are two Apache projects, which are
affected by these bugs, namely all versions released until today. Solr users
with the default configuration will have Java crashing with SIGSEGV as soon
as they start to index documents, as one affected part is the well-known
Porter stemmer (see LUCENE-3335 [4]). Other loops in Lucene may be
miscompiled, too, leading to index corruption (especially on Lucene trunk
with pulsing codec; other loops may be affected, too - LUCENE-3346 [5]).

These problems were detected only 5 days before the official Java 7 release,
so Oracle had no time to fix those bugs, affecting also many more
applications. In response to our questions, they proposed to include the
fixes into service release u2 (eventually into service release u1, see [6]).
This means you cannot use Apache Lucene/Solr with Java 7 releases before
Update 2! If you do, please don't open bug reports, it is not the
committers' fault! At least disable loop optimizations using the
-XX:-UseLoopPredicate JVM option to not risk index corruptions.

Please note: Also Java 6 users are affected, if they use one of those JVM
options, which are not enabled by default: -XX:+OptimizeStringConcat or
-XX:+AggressiveOpts

It is strongly recommended not to use any hotspot optimization switches in
any Java version without extensive testing!

In case you upgrade to Java 7, remember that you may have to reindex, as the
unicode version shipped with Java 7 changed and tokenization behaves
differently (e.g. lowercasing). For more information, read
JRE_VERSION_MIGRATION.txt in your distribution package!

On behalf of the Lucene project,
Uwe

[1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7070134
[2] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7044738
[3] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7068051
[4] https://issues.apache.org/jira/browse/LUCENE-3335
[5] https://issues.apache.org/jira/browse/LUCENE-3346
[6] http://s.apache.org/StQ

-
Uwe Schindler
uschind...@apache.org 
Apache Lucene PMC Member / Committer
Bremen, Germany
http://lucene.apache.org/