Re: Deadlock in lucene?

2008-08-19 Thread Matthew Runo
I know this isn't really the place for this, so please forgive me -  
but does this patch look reasonably safe to use to skip the isDeleted  
check inside of FunctionQuery?


My reasoning behind this is that many people (us included) will be  
building the index on a separate server, and then using the  
replication scripts to publish the files out to several read-only  
servers. On those instances, deletedDocs would always be empty, since  
it's a read only instance - and so we can conveniently skip the Lucene  
code in question. This flag would also be good for other optimizations  
that can only be made when you assume the index is read-only.


Solr seems to work with the flag set - any reasons why this will crash  
and/or kill my kitten?


(please forgive my posting this here instead of in solr-dev!)

Index: src/java/org/apache/solr/search/FunctionQParser.java
===
--- src/java/org/apache/solr/search/FunctionQParser.java	(revision  
687135)
+++ src/java/org/apache/solr/search/FunctionQParser.java	Tue Aug 19  
11:08:45 PDT 2008

@@ -49,7 +49,7 @@
 }
 ***/

-return new FunctionQuery(vs);
+return new FunctionQuery(vs,  
req.getSchema().getSolrConfig().isReadOnly() );

   }

   /**
Index: src/java/org/apache/solr/search/function/FunctionQuery.java
===
--- src/java/org/apache/solr/search/function/FunctionQuery.java	 
(revision 687135)
+++ src/java/org/apache/solr/search/function/FunctionQuery.java	Tue  
Aug 19 11:08:45 PDT 2008

@@ -31,12 +31,14 @@
  */
 public class FunctionQuery extends Query {
   ValueSource func;
+  Boolean readOnly;

   /**
* @param func defines the function to be used for scoring
*/
-  public FunctionQuery(ValueSource func) {
+  public FunctionQuery(ValueSource func, Boolean readOnly) {
 this.func=func;
+this.readOnly=readOnly;
   }

   /** @return The associated ValueSource */
@@ -113,7 +115,7 @@
 if (doc=maxDoc) {
   return false;
 }
-if (reader.isDeleted(doc)) continue;
+if (!readOnly  reader.isDeleted(doc)) continue;
 // todo: maybe allow score() to throw a specific exception
 // and continue on to the next document if it is thrown...
 // that may be useful, but exceptions aren't really good
Index: src/java/org/apache/solr/core/Config.java
===
--- src/java/org/apache/solr/core/Config.java   (revision 687135)
+++ src/java/org/apache/solr/core/Config.java	Tue Aug 19 11:08:45 PDT  
2008

@@ -45,6 +45,8 @@
   private final String name;
   private final SolrResourceLoader loader;

+  private Boolean readOnly;
+
   /**
* @deprecated Use [EMAIL PROTECTED] #Config(SolrResourceLoader, String,  
InputStream, String)} instead.

*/
@@ -254,6 +256,19 @@
  return val!=null ? Double.parseDouble(val) : def;
}

+  /**
+   * Is the index set up to be readOnly? If so, this will cause the  
FunctionQuery stuff to not check

+   * for deleted documents.
+   * @return boolean readOnly
+   */
+   public boolean isReadOnly() {
+   if( this.readOnly == null ){
+   readOnly = getBool(/mainIndex/readOnly, false);
+   }
+
+   return readOnly;
+   }
+
   // The following functions were moved to ResourceLoader
   
//-

Index: example/solr/conf/solrconfig.xml
===
--- example/solr/conf/solrconfig.xml(revision 687135)
+++ example/solr/conf/solrconfig.xmlTue Aug 19 11:13:13 PDT 2008
@@ -114,6 +114,12 @@
  This is not needed if lock type is 'none' or 'single'
  --
 unlockOnStartupfalse/unlockOnStartup
+
+   !-- In the event that you are only using this index for reads,
+you can enable this flag. This will skip some checks that
+can cause performance issues when under high load
+   --
+   readOnlyfalse/readOnly
   /mainIndex

   !--	Enables JMX if and only if an existing MBeanServer is found,  
use




--- end patch ---

On Aug 18, 2008, at 8:04 PM, Yonik Seeley wrote:


It's not a deadlock (just a synchronization bottleneck) , but it is a
known issue in Lucene and there has been some progress in improving
the situation.
-Yonik


On Mon, Aug 18, 2008 at 10:55 PM, Matthew Runo [EMAIL PROTECTED]  
wrote:

Hello folks!

I was just wondering if anyone else has seen this issue under heavy  
load. We
had some servers set to very high thread limits (12 core servers  
with 32
gigs of ram), and found several threads would end up in this  
state


Name: http-8080-891
State: BLOCKED on [EMAIL PROTECTED]  
owned by:

http-8080-191
Total blocked: 97,926  Total waited: 16

Stack trace:
org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java: 
674)
org.apache.solr.search.function.FunctionQuery 

Re: Deadlock in lucene?

2008-08-19 Thread Yonik Seeley
It doesn't matter that it's executed on the read-only server... it
matters if any of the docs are marked as deleted.   That's the
condition that you probably want to check for.

-Yonik

On Tue, Aug 19, 2008 at 4:26 PM, Matthew Runo [EMAIL PROTECTED] wrote:
 I know this isn't really the place for this, so please forgive me - but does
 this patch look reasonably safe to use to skip the isDeleted check inside of
 FunctionQuery?

 My reasoning behind this is that many people (us included) will be building
 the index on a separate server, and then using the replication scripts to
 publish the files out to several read-only servers. On those instances,
 deletedDocs would always be empty, since it's a read only instance - and so
 we can conveniently skip the Lucene code in question. This flag would also
 be good for other optimizations that can only be made when you assume the
 index is read-only.

 Solr seems to work with the flag set - any reasons why this will crash
 and/or kill my kitten?

 (please forgive my posting this here instead of in solr-dev!)

 Index: src/java/org/apache/solr/search/FunctionQParser.java
 ===
 --- src/java/org/apache/solr/search/FunctionQParser.java(revision
 687135)
 +++ src/java/org/apache/solr/search/FunctionQParser.javaTue Aug 19
 11:08:45 PDT 2008
 @@ -49,7 +49,7 @@
 }
 ***/

 -return new FunctionQuery(vs);
 +return new FunctionQuery(vs,
 req.getSchema().getSolrConfig().isReadOnly() );
   }

   /**
 Index: src/java/org/apache/solr/search/function/FunctionQuery.java
 ===
 --- src/java/org/apache/solr/search/function/FunctionQuery.java (revision
 687135)
 +++ src/java/org/apache/solr/search/function/FunctionQuery.java Tue Aug 19
 11:08:45 PDT 2008
 @@ -31,12 +31,14 @@
  */
  public class FunctionQuery extends Query {
   ValueSource func;
 +  Boolean readOnly;

   /**
* @param func defines the function to be used for scoring
*/
 -  public FunctionQuery(ValueSource func) {
 +  public FunctionQuery(ValueSource func, Boolean readOnly) {
 this.func=func;
 +this.readOnly=readOnly;
   }

   /** @return The associated ValueSource */
 @@ -113,7 +115,7 @@
 if (doc=maxDoc) {
   return false;
 }
 -if (reader.isDeleted(doc)) continue;
 +if (!readOnly  reader.isDeleted(doc)) continue;
 // todo: maybe allow score() to throw a specific exception
 // and continue on to the next document if it is thrown...
 // that may be useful, but exceptions aren't really good
 Index: src/java/org/apache/solr/core/Config.java
 ===
 --- src/java/org/apache/solr/core/Config.java   (revision 687135)
 +++ src/java/org/apache/solr/core/Config.java   Tue Aug 19 11:08:45 PDT 2008
 @@ -45,6 +45,8 @@
   private final String name;
   private final SolrResourceLoader loader;

 +  private Boolean readOnly;
 +
   /**
* @deprecated Use [EMAIL PROTECTED] #Config(SolrResourceLoader, String, 
 InputStream,
 String)} instead.
*/
 @@ -254,6 +256,19 @@
  return val!=null ? Double.parseDouble(val) : def;
}

 +  /**
 +   * Is the index set up to be readOnly? If so, this will cause the
 FunctionQuery stuff to not check
 +   * for deleted documents.
 +   * @return boolean readOnly
 +   */
 +   public boolean isReadOnly() {
 +   if( this.readOnly == null ){
 +   readOnly = getBool(/mainIndex/readOnly, false);
 +   }
 +
 +   return readOnly;
 +   }
 +
   // The following functions were moved to ResourceLoader

 //-

 Index: example/solr/conf/solrconfig.xml
 ===
 --- example/solr/conf/solrconfig.xml(revision 687135)
 +++ example/solr/conf/solrconfig.xmlTue Aug 19 11:13:13 PDT 2008
 @@ -114,6 +114,12 @@
  This is not needed if lock type is 'none' or 'single'
  --
 unlockOnStartupfalse/unlockOnStartup
 +
 +   !-- In the event that you are only using this index for reads,
 +you can enable this flag. This will skip some checks that
 +can cause performance issues when under high load
 +   --
 +   readOnlyfalse/readOnly
   /mainIndex

   !-- Enables JMX if and only if an existing MBeanServer is found, use



 --- end patch ---

 On Aug 18, 2008, at 8:04 PM, Yonik Seeley wrote:

 It's not a deadlock (just a synchronization bottleneck) , but it is a
 known issue in Lucene and there has been some progress in improving
 the situation.
 -Yonik


 On Mon, Aug 18, 2008 at 10:55 PM, Matthew Runo [EMAIL PROTECTED] wrote:

 Hello folks!

 I was just wondering if anyone else has seen this issue under heavy load.
 We
 had some servers set to very high thread limits (12 core servers with 32
 gigs of ram), and found 

Re: Deadlock in lucene?

2008-08-19 Thread Yonik Seeley
FYI, I just slipped this optimization into trunk.

-Yonik

On Tue, Aug 19, 2008 at 4:37 PM, Yonik Seeley [EMAIL PROTECTED] wrote:
 It doesn't matter that it's executed on the read-only server... it
 matters if any of the docs are marked as deleted.   That's the
 condition that you probably want to check for.

 -Yonik

 On Tue, Aug 19, 2008 at 4:26 PM, Matthew Runo [EMAIL PROTECTED] wrote:
 I know this isn't really the place for this, so please forgive me - but does
 this patch look reasonably safe to use to skip the isDeleted check inside of
 FunctionQuery?

 My reasoning behind this is that many people (us included) will be building
 the index on a separate server, and then using the replication scripts to
 publish the files out to several read-only servers. On those instances,
 deletedDocs would always be empty, since it's a read only instance - and so
 we can conveniently skip the Lucene code in question. This flag would also
 be good for other optimizations that can only be made when you assume the
 index is read-only.

 Solr seems to work with the flag set - any reasons why this will crash
 and/or kill my kitten?

 (please forgive my posting this here instead of in solr-dev!)

 Index: src/java/org/apache/solr/search/FunctionQParser.java
 ===
 --- src/java/org/apache/solr/search/FunctionQParser.java(revision
 687135)
 +++ src/java/org/apache/solr/search/FunctionQParser.javaTue Aug 19
 11:08:45 PDT 2008
 @@ -49,7 +49,7 @@
 }
 ***/

 -return new FunctionQuery(vs);
 +return new FunctionQuery(vs,
 req.getSchema().getSolrConfig().isReadOnly() );
   }

   /**
 Index: src/java/org/apache/solr/search/function/FunctionQuery.java
 ===
 --- src/java/org/apache/solr/search/function/FunctionQuery.java (revision
 687135)
 +++ src/java/org/apache/solr/search/function/FunctionQuery.java Tue Aug 19
 11:08:45 PDT 2008
 @@ -31,12 +31,14 @@
  */
  public class FunctionQuery extends Query {
   ValueSource func;
 +  Boolean readOnly;

   /**
* @param func defines the function to be used for scoring
*/
 -  public FunctionQuery(ValueSource func) {
 +  public FunctionQuery(ValueSource func, Boolean readOnly) {
 this.func=func;
 +this.readOnly=readOnly;
   }

   /** @return The associated ValueSource */
 @@ -113,7 +115,7 @@
 if (doc=maxDoc) {
   return false;
 }
 -if (reader.isDeleted(doc)) continue;
 +if (!readOnly  reader.isDeleted(doc)) continue;
 // todo: maybe allow score() to throw a specific exception
 // and continue on to the next document if it is thrown...
 // that may be useful, but exceptions aren't really good
 Index: src/java/org/apache/solr/core/Config.java
 ===
 --- src/java/org/apache/solr/core/Config.java   (revision 687135)
 +++ src/java/org/apache/solr/core/Config.java   Tue Aug 19 11:08:45 PDT 2008
 @@ -45,6 +45,8 @@
   private final String name;
   private final SolrResourceLoader loader;

 +  private Boolean readOnly;
 +
   /**
* @deprecated Use [EMAIL PROTECTED] #Config(SolrResourceLoader, String, 
 InputStream,
 String)} instead.
*/
 @@ -254,6 +256,19 @@
  return val!=null ? Double.parseDouble(val) : def;
}

 +  /**
 +   * Is the index set up to be readOnly? If so, this will cause the
 FunctionQuery stuff to not check
 +   * for deleted documents.
 +   * @return boolean readOnly
 +   */
 +   public boolean isReadOnly() {
 +   if( this.readOnly == null ){
 +   readOnly = getBool(/mainIndex/readOnly, false);
 +   }
 +
 +   return readOnly;
 +   }
 +
   // The following functions were moved to ResourceLoader

 //-

 Index: example/solr/conf/solrconfig.xml
 ===
 --- example/solr/conf/solrconfig.xml(revision 687135)
 +++ example/solr/conf/solrconfig.xmlTue Aug 19 11:13:13 PDT 2008
 @@ -114,6 +114,12 @@
  This is not needed if lock type is 'none' or 'single'
  --
 unlockOnStartupfalse/unlockOnStartup
 +
 +   !-- In the event that you are only using this index for reads,
 +you can enable this flag. This will skip some checks that
 +can cause performance issues when under high load
 +   --
 +   readOnlyfalse/readOnly
   /mainIndex

   !-- Enables JMX if and only if an existing MBeanServer is found, use



 --- end patch ---

 On Aug 18, 2008, at 8:04 PM, Yonik Seeley wrote:

 It's not a deadlock (just a synchronization bottleneck) , but it is a
 known issue in Lucene and there has been some progress in improving
 the situation.
 -Yonik


 On Mon, Aug 18, 2008 at 10:55 PM, Matthew Runo [EMAIL PROTECTED] wrote:

 Hello folks!

 I was just wondering if anyone else has seen 

Re: Deadlock in lucene?

2008-08-19 Thread Fuad Efendi


I don't think it will help; for instance SegmentReader of Lucene:

public synchronized Document document(int n, FieldSelector fieldSelector)


Unsynchronized (in future) SOLR caching should help.

-Fuad



I know this isn't really the place for this, so please forgive me - but
does this patch look reasonably safe to use to skip the isDeleted check
inside of FunctionQuery?

My reasoning behind this is that many people (us included) will be
building the index on a separate server, and then using the replication
scripts to publish the files out to several read-only servers. On those
instances, deletedDocs would always be empty, since it's a read only
instance - and so we can conveniently skip the Lucene code in question.
This flag would also be good for other optimizations that can only be
made when you assume the index is read-only.

Solr seems to work with the flag set - any reasons why this will crash
and/or kill my kitten?

(please forgive my posting this here instead of in solr-dev!)

Index: src/java/org/apache/solr/search/FunctionQParser.java

...




Re: Deadlock in lucene?

2008-08-19 Thread Otis Gospodnetic
Matthew, just because an index is read-only on some server it doesn't mean it 
contains no deletes (no docs marked as deleted, but not yet removed from the 
index).  So you still want to check isDeleted(doc) *unless* you are certain the 
index has no docs marked as deleted (this happens after optimization).

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
 From: Matthew Runo [EMAIL PROTECTED]
 To: solr-user@lucene.apache.org
 Sent: Tuesday, August 19, 2008 4:26:59 PM
 Subject: Re: Deadlock in lucene?
 
 I know this isn't really the place for this, so please forgive me -  
 but does this patch look reasonably safe to use to skip the isDeleted  
 check inside of FunctionQuery?
 
 My reasoning behind this is that many people (us included) will be  
 building the index on a separate server, and then using the  
 replication scripts to publish the files out to several read-only  
 servers. On those instances, deletedDocs would always be empty, since  
 it's a read only instance - and so we can conveniently skip the Lucene  
 code in question. This flag would also be good for other optimizations  
 that can only be made when you assume the index is read-only.
 
 Solr seems to work with the flag set - any reasons why this will crash  
 and/or kill my kitten?
 
 (please forgive my posting this here instead of in solr-dev!)
 
 Index: src/java/org/apache/solr/search/FunctionQParser.java
 ===
 --- src/java/org/apache/solr/search/FunctionQParser.java(revision  
 687135)
 +++ src/java/org/apache/solr/search/FunctionQParser.javaTue Aug 19  
 11:08:45 PDT 2008
 @@ -49,7 +49,7 @@
   }
   ***/
 
 -return new FunctionQuery(vs);
 +return new FunctionQuery(vs,  
 req.getSchema().getSolrConfig().isReadOnly() );
 }
 
 /**
 Index: src/java/org/apache/solr/search/function/FunctionQuery.java
 ===
 --- src/java/org/apache/solr/search/function/FunctionQuery.java
 (revision 687135)
 +++ src/java/org/apache/solr/search/function/FunctionQuery.javaTue  
 Aug 19 11:08:45 PDT 2008
 @@ -31,12 +31,14 @@
*/
   public class FunctionQuery extends Query {
 ValueSource func;
 +  Boolean readOnly;
 
 /**
  * @param func defines the function to be used for scoring
  */
 -  public FunctionQuery(ValueSource func) {
 +  public FunctionQuery(ValueSource func, Boolean readOnly) {
   this.func=func;
 +this.readOnly=readOnly;
 }
 
 /** @return The associated ValueSource */
 @@ -113,7 +115,7 @@
   if (doc=maxDoc) {
 return false;
   }
 -if (reader.isDeleted(doc)) continue;
 +if (!readOnly  reader.isDeleted(doc)) continue;
   // todo: maybe allow score() to throw a specific exception
   // and continue on to the next document if it is thrown...
   // that may be useful, but exceptions aren't really good
 Index: src/java/org/apache/solr/core/Config.java
 ===
 --- src/java/org/apache/solr/core/Config.java(revision 687135)
 +++ src/java/org/apache/solr/core/Config.javaTue Aug 19 11:08:45 PDT  
 2008
 @@ -45,6 +45,8 @@
 private final String name;
 private final SolrResourceLoader loader;
 
 +  private Boolean readOnly;
 +
 /**
  * @deprecated Use [EMAIL PROTECTED] #Config(SolrResourceLoader, String,  
 InputStream, String)} instead.
  */
 @@ -254,6 +256,19 @@
return val!=null ? Double.parseDouble(val) : def;
  }
 
 +  /**
 +   * Is the index set up to be readOnly? If so, this will cause the  
 FunctionQuery stuff to not check
 +   * for deleted documents.
 +   * @return boolean readOnly
 +   */
 +   public boolean isReadOnly() {
 +   if( this.readOnly == null ){
 +   readOnly = getBool(/mainIndex/readOnly, false);
 +   }
 +
 +   return readOnly;
 +   }
 +
 // The following functions were moved to ResourceLoader
 
 //-
 
 Index: example/solr/conf/solrconfig.xml
 ===
 --- example/solr/conf/solrconfig.xml(revision 687135)
 +++ example/solr/conf/solrconfig.xmlTue Aug 19 11:13:13 PDT 2008
 @@ -114,6 +114,12 @@
This is not needed if lock type is 'none' or 'single'
--
   false
 +
 +
 +false
 
 
  use
 
 
 
 --- end patch ---
 
 On Aug 18, 2008, at 8:04 PM, Yonik Seeley wrote:
 
  It's not a deadlock (just a synchronization bottleneck) , but it is a
  known issue in Lucene and there has been some progress in improving
  the situation.
  -Yonik
 
 
  On Mon, Aug 18, 2008 at 10:55 PM, Matthew Runo   
  wrote:
  Hello folks!
 
  I was just wondering if anyone else has seen this issue under heavy  
  load. We
  had some servers set to very

Re: Deadlock in lucene?

2008-08-19 Thread Matthew Runo
Ouch, that's certainly a problem! I'll have to think some more on this  
one.


Thanks for your time!

Matthew Runo
Software Engineer, Zappos.com
[EMAIL PROTECTED] - 702-943-7833

On Aug 19, 2008, at 1:42 PM, Otis Gospodnetic wrote:

Matthew, just because an index is read-only on some server it  
doesn't mean it contains no deletes (no docs marked as deleted, but  
not yet removed from the index).  So you still want to check  
isDeleted(doc) *unless* you are certain the index has no docs marked  
as deleted (this happens after optimization).


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 

From: Matthew Runo [EMAIL PROTECTED]
To: solr-user@lucene.apache.org
Sent: Tuesday, August 19, 2008 4:26:59 PM
Subject: Re: Deadlock in lucene?

I know this isn't really the place for this, so please forgive me -
but does this patch look reasonably safe to use to skip the isDeleted
check inside of FunctionQuery?

My reasoning behind this is that many people (us included) will be
building the index on a separate server, and then using the
replication scripts to publish the files out to several read-only
servers. On those instances, deletedDocs would always be empty, since
it's a read only instance - and so we can conveniently skip the  
Lucene
code in question. This flag would also be good for other  
optimizations

that can only be made when you assume the index is read-only.

Solr seems to work with the flag set - any reasons why this will  
crash

and/or kill my kitten?

(please forgive my posting this here instead of in solr-dev!)

Index: src/java/org/apache/solr/search/FunctionQParser.java
===
--- src/java/org/apache/solr/search/FunctionQParser.java(revision
687135)
+++ src/java/org/apache/solr/search/FunctionQParser.javaTue Aug  
19

11:08:45 PDT 2008
@@ -49,7 +49,7 @@
 }
 ***/

-return new FunctionQuery(vs);
+return new FunctionQuery(vs,
req.getSchema().getSolrConfig().isReadOnly() );
   }

   /**
Index: src/java/org/apache/solr/search/function/FunctionQuery.java
===
--- src/java/org/apache/solr/search/function/FunctionQuery.java
(revision 687135)
+++ src/java/org/apache/solr/search/function/FunctionQuery.java 
Tue

Aug 19 11:08:45 PDT 2008
@@ -31,12 +31,14 @@
  */
 public class FunctionQuery extends Query {
   ValueSource func;
+  Boolean readOnly;

   /**
* @param func defines the function to be used for scoring
*/
-  public FunctionQuery(ValueSource func) {
+  public FunctionQuery(ValueSource func, Boolean readOnly) {
 this.func=func;
+this.readOnly=readOnly;
   }

   /** @return The associated ValueSource */
@@ -113,7 +115,7 @@
 if (doc=maxDoc) {
   return false;
 }
-if (reader.isDeleted(doc)) continue;
+if (!readOnly  reader.isDeleted(doc)) continue;
 // todo: maybe allow score() to throw a specific exception
 // and continue on to the next document if it is thrown...
 // that may be useful, but exceptions aren't really good
Index: src/java/org/apache/solr/core/Config.java
===
--- src/java/org/apache/solr/core/Config.java(revision 687135)
+++ src/java/org/apache/solr/core/Config.javaTue Aug 19  
11:08:45 PDT

2008
@@ -45,6 +45,8 @@
   private final String name;
   private final SolrResourceLoader loader;

+  private Boolean readOnly;
+
   /**
* @deprecated Use [EMAIL PROTECTED] #Config(SolrResourceLoader, String,
InputStream, String)} instead.
*/
@@ -254,6 +256,19 @@
  return val!=null ? Double.parseDouble(val) : def;
}

+  /**
+   * Is the index set up to be readOnly? If so, this will cause the
FunctionQuery stuff to not check
+   * for deleted documents.
+   * @return boolean readOnly
+   */
+   public boolean isReadOnly() {
+   if( this.readOnly == null ){
+   readOnly = getBool(/mainIndex/readOnly, false);
+   }
+
+   return readOnly;
+   }
+
   // The following functions were moved to ResourceLoader

//-

Index: example/solr/conf/solrconfig.xml
===
--- example/solr/conf/solrconfig.xml(revision 687135)
+++ example/solr/conf/solrconfig.xmlTue Aug 19 11:13:13 PDT 2008
@@ -114,6 +114,12 @@
  This is not needed if lock type is 'none' or 'single'
  --
 false
+
+
+false



use




--- end patch ---

On Aug 18, 2008, at 8:04 PM, Yonik Seeley wrote:

It's not a deadlock (just a synchronization bottleneck) , but it  
is a

known issue in Lucene and there has been some progress in improving
the situation.
-Yonik


On Mon, Aug 18, 2008 at 10:55 PM, Matthew Runo
wrote:

Hello folks!

I was just wondering if anyone else has seen this issue under heavy
load. We
had

Deadlock in lucene?

2008-08-18 Thread Matthew Runo

Hello folks!

I was just wondering if anyone else has seen this issue under heavy  
load. We had some servers set to very high thread limits (12 core  
servers with 32 gigs of ram), and found several threads would end up  
in this state


Name: http-8080-891
State: BLOCKED on [EMAIL PROTECTED] owned  
by: http-8080-191

Total blocked: 97,926  Total waited: 16

Stack trace:
org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:674)
org.apache.solr.search.function.FunctionQuery 
$AllScorer.next(FunctionQuery.java:116)
org 
.apache 
.lucene 
.util.ScorerDocQueue.topNextAndAdjustElsePop(ScorerDocQueue.java:116)
org 
.apache 
.lucene 
.search 
.DisjunctionSumScorer.advanceAfterCurrent(DisjunctionSumScorer.java:175)
org 
.apache 
.lucene.search.DisjunctionSumScorer.skipTo(DisjunctionSumScorer.java: 
228)

org.apache.lucene.search.ReqOptSumScorer.score(ReqOptSumScorer.java:76)
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:357)
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:320)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:137)
org.apache.lucene.search.Searcher.search(Searcher.java:126)
org.apache.lucene.search.Searcher.search(Searcher.java:105)
org 
.apache 
.solr 
.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java: 
1148)
org 
.apache 
.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:834)
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java: 
269)
org 
.apache 
.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
org 
.apache 
.solr 
.handler.component.SearchHandler.handleRequestBody(SearchHandler.java: 
169)
org 
.apache 
.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java: 
128)

org.apache.solr.core.SolrCore.execute(SolrCore.java:1143)
org 
.apache 
.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
org 
.apache 
.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:272)
org 
.apache 
.catalina 
.core 
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java: 
235)
org 
.apache 
.catalina 
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org 
.apache 
.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java: 
233)
org 
.apache 
.catalina.core.StandardContextValve.invoke(StandardContextValve.java: 
175)
org 
.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java: 
128)
org 
.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java: 
102)
org 
.apache 
.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java: 
286)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java: 
844)
org.apache.coyote.http11.Http11Protocol 
$Http11ConnectionHandler.process(Http11Protocol.java:583)

org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
java.lang.Thread.run(Thread.java:619)

Thanks for your time!

Matthew Runo
Software Engineer, Zappos.com
[EMAIL PROTECTED] - 702-943-7833



Re: Deadlock in lucene?

2008-08-18 Thread Yonik Seeley
It's not a deadlock (just a synchronization bottleneck) , but it is a
known issue in Lucene and there has been some progress in improving
the situation.
-Yonik


On Mon, Aug 18, 2008 at 10:55 PM, Matthew Runo [EMAIL PROTECTED] wrote:
 Hello folks!

 I was just wondering if anyone else has seen this issue under heavy load. We
 had some servers set to very high thread limits (12 core servers with 32
 gigs of ram), and found several threads would end up in this state

 Name: http-8080-891
 State: BLOCKED on [EMAIL PROTECTED] owned by:
 http-8080-191
 Total blocked: 97,926  Total waited: 16

 Stack trace:
 org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:674)
 org.apache.solr.search.function.FunctionQuery$AllScorer.next(FunctionQuery.java:116)
 org.apache.lucene.util.ScorerDocQueue.topNextAndAdjustElsePop(ScorerDocQueue.java:116)
 org.apache.lucene.search.DisjunctionSumScorer.advanceAfterCurrent(DisjunctionSumScorer.java:175)
 org.apache.lucene.search.DisjunctionSumScorer.skipTo(DisjunctionSumScorer.java:228)
 org.apache.lucene.search.ReqOptSumScorer.score(ReqOptSumScorer.java:76)
 org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:357)
 org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:320)
 org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:137)
 org.apache.lucene.search.Searcher.search(Searcher.java:126)
 org.apache.lucene.search.Searcher.search(Searcher.java:105)
 org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1148)
 org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:834)
 org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
 org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
 org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:128)
 org.apache.solr.core.SolrCore.execute(SolrCore.java:1143)
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:272)
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
 org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
 org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
 org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 java.lang.Thread.run(Thread.java:619)

 Thanks for your time!

 Matthew Runo
 Software Engineer, Zappos.com
 [EMAIL PROTECTED] - 702-943-7833