Re: SSD experience

[Sanne Grinovero]

Indeed I would never actually use it, but symlinks do exist on Windows.

http://en.wikipedia.org/wiki/NTFS_symbolic_link

Sanne

2011/8/23 Peter Sturge :
> The Solr index directory lives directly on the SSD (running on Windows
> - where the word symlink does not appear in any dictionary within a
> 100 mile radius of Redmond :-)
>
> Currently, the main limiting factors of SSD are cost and size. SSDs
> will get larger over time. Splitting indexes across multiple shards on
> multiple SSDs is a wonderfully fast, if not slightly extravagant
> method of getting excellent IO performance.
> Regarding cost, I've seen many organizations where the use of fast
> SANs costs at least the same if not more per GB of storage than SSD.
> Hybrid drives can be a good cost-effective alternative as well.
>
> Peter
>
>
>
> On Tue, Aug 23, 2011 at 3:29 PM, Gerard Roos  wrote:
>> Interesting. Do you make a symlink to the indexes or is the whole Solr 
>> directory on SSD?
>>
>> thanks,
>> Gerard
>>
>> Op 23 aug. 2011, om 12:53 heeft Peter Sturge het volgende geschreven:
>>
>>> Just to add a few cents worth regarding SSD...
>>>
>>> We use Vertex SSD drives for storing indexes, and wow, they really
>>> scream compared to SATA/SAS/SAN. As we do some heavy commits, it's the
>>> commit times where we see the biggest performance boost.
>>> In tests, we found that locally attached 15k SAS drives are the next
>>> best for performance. SANs can work well, but should be FibreChannel.
>>> IP-based SANs are ok, as long they're not heavily taxed by other,
>>> non-Solr disk I/O.
>>> NAS is far and away the poorest performing - not recommended for real 
>>> indexes.
>>>
>>> HTH,
>>> Peter
>>>
>>>
>>>
>>> On Mon, Aug 22, 2011 at 3:54 PM, Rich Cariens  wrote:
 Ahoy ahoy!

 Does anyone have any experiences or stories they can share with the list
 about how SSDs impacted search performance for better or worse?

 I found a Lucene SSD performance benchmark
 docbut
 the wiki engine is refusing to let me view the attachment (I get "You
 are not allowed to do AttachFile on this page.").

 Thanks in advance!

>>>
>>>
>>
>>
>

Re: [WARNING] Index corruption and crashes in Apache Lucene Core / Apache Solr with Java 7

[Sanne Grinovero]

Hello,
thanks for the warning, that's a pretty nasty bug.

A patch was made for OpenJDK, if anybody is interested to try it out
that would be great:
http://hg.openjdk.java.net/hsx/hotspot-comp/hotspot/rev/4e761e7e6e12

Regards,
Sanne

2011/7/28 Uwe Schindler :
> Hello Apache Lucene & Apache Solr users,
> Hello users of other Java-based Apache projects,
>
> Oracle released Java 7 today. Unfortunately it contains hotspot compiler
> optimizations, which miscompile some loops. This can affect code of several
> Apache projects. Sometimes JVMs only crash, but in several cases, results
> calculated can be incorrect, leading to bugs in applications (see Hotspot
> bugs 7070134 [1], 7044738 [2], 7068051 [3]).
>
> Apache Lucene Core and Apache Solr are two Apache projects, which are
> affected by these bugs, namely all versions released until today. Solr users
> with the default configuration will have Java crashing with SIGSEGV as soon
> as they start to index documents, as one affected part is the well-known
> Porter stemmer (see LUCENE-3335 [4]). Other loops in Lucene may be
> miscompiled, too, leading to index corruption (especially on Lucene trunk
> with pulsing codec; other loops may be affected, too - LUCENE-3346 [5]).
>
> These problems were detected only 5 days before the official Java 7 release,
> so Oracle had no time to fix those bugs, affecting also many more
> applications. In response to our questions, they proposed to include the
> fixes into service release u2 (eventually into service release u1, see [6]).
> This means you cannot use Apache Lucene/Solr with Java 7 releases before
> Update 2! If you do, please don't open bug reports, it is not the
> committers' fault! At least disable loop optimizations using the
> -XX:-UseLoopPredicate JVM option to not risk index corruptions.
>
> Please note: Also Java 6 users are affected, if they use one of those JVM
> options, which are not enabled by default: -XX:+OptimizeStringConcat or
> -XX:+AggressiveOpts
>
> It is strongly recommended not to use any hotspot optimization switches in
> any Java version without extensive testing!
>
> In case you upgrade to Java 7, remember that you may have to reindex, as the
> unicode version shipped with Java 7 changed and tokenization behaves
> differently (e.g. lowercasing). For more information, read
> JRE_VERSION_MIGRATION.txt in your distribution package!
>
> On behalf of the Lucene project,
> Uwe
>
> [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7070134
> [2] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7044738
> [3] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7068051
> [4] https://issues.apache.org/jira/browse/LUCENE-3335
> [5] https://issues.apache.org/jira/browse/LUCENE-3346
> [6] http://s.apache.org/StQ
>
> -
> Uwe Schindler
> uschind...@apache.org
> Apache Lucene PMC Member / Committer
> Bremen, Germany
> http://lucene.apache.org/
>
>
>
> -
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>