[jira] Commented: (SOLR-1482) Solr master and slave freeze after query
[
https://issues.apache.org/jira/browse/SOLR-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12761654#action_12761654
]
Bill Au commented on SOLR-1482:
---
You probably want to take a JVM thread dump (kill -3) while the JVM is hung to
find out what's going on.
Is your webapp app being reloaded? You can check the appserver log file to see
if that's happening. One common way of running out of PermGen space is a
classloader link which occurs when a webapp is reloaded.
Solr master and slave freeze after query
Key: SOLR-1482
URL: https://issues.apache.org/jira/browse/SOLR-1482
Project: Solr
Issue Type: Bug
Affects Versions: 1.4
Environment: Nightly 9/28/09.
14 individual instances per server, using JNDI.
replicateAfter commit, 5 min interval polling.
All caches are currently commented out, on both slave and master.
Lots of ongoing commits - large chunks of data, each accompanied by a commit.
This is to guarantee that anything we think is now in Solr remains there in
case the server crashes.
Reporter: Artem Russakovskii
Priority: Critical
We're having issues with the deployment of 2 master-slave setups.
One of the master-slave setups is OK (so far) but on the other both the
master and the slave keep freezing, but only after I send a query to them.
And by freezing I mean indefinite hanging, with almost no output to log, no
errors, nothing. It's as if there's some sort of a deadlock. The hanging
servers need to be killed with -9, otherwise they keep hanging.
The query I send queries all instances at the same time using the ?shards=
syntax.
On the slave, the logs just stop - nothing shows up anymore after the query
is issued. On the master, they're a bit more descriptive. This information
seeps through very-very slowly, as you can see from the timestamps:
{quote}
SEVERE: java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:16:00 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:37 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:37 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:39 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Exception in thread ContainerBackException in thread pool-29-threadOct 1,
2009 2:21:47 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
-22 java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Exception in thread http-8080-42 Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Exception in thread http-8080-26 Exception in thread http-8080-32
Exception in thread http-8080-25 Exception in thread http-8080-22
Exception in thread http-8080-15 Exception in thread http-8080-45
Exception in thread http-8080-13 Exception in thread http-8080-48
Exception in thread http-8080-7 Exception in thread http-8080-38
Exception in thread http-8080-39 Exception in thread http-8080-28
Exception in thread http-8080-1 Exception in thread http-8080-2 Exception
in thread http-8080-12 Exception in thread http-8080-44 Exception in
thread http-8080-47 Exception in thread http-8080-29 Exception in thread
http-8080-33 Exception in thread http-8080-27 Exception in thread
http-8080-36 Exception in thread http-8080-113 Exception in thread
http-8080-112
[jira] Commented: (SOLR-1482) Solr master and slave freeze after query
[
https://issues.apache.org/jira/browse/SOLR-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12761677#action_12761677
]
Artem Russakovskii commented on SOLR-1482:
--
Also, just saw this on the first slave:
{quote}
INFO: Closing searc...@3efceb09 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Oct 2, 2009 11:43:27 AM org.apache.solr.handler.SnapPuller doCommit
INFO: Force open index writer to make sure older index files get deleted
Oct 2, 2009 11:43:35 AM org.apache.solr.update.SolrIndexWriter finalize
SEVERE: SolrIndexWriter was not closed prior to finalize(), indicates a bug --
POSSIBLE RESOURCE LEAK!!!
{quote}
Solr master and slave freeze after query
Key: SOLR-1482
URL: https://issues.apache.org/jira/browse/SOLR-1482
Project: Solr
Issue Type: Bug
Affects Versions: 1.4
Environment: Nightly 9/28/09.
14 individual instances per server, using JNDI.
replicateAfter commit, 5 min interval polling.
All caches are currently commented out, on both slave and master.
Lots of ongoing commits - large chunks of data, each accompanied by a commit.
This is to guarantee that anything we think is now in Solr remains there in
case the server crashes.
Reporter: Artem Russakovskii
Priority: Critical
Attachments: catalina.out, catalina2.out
We're having issues with the deployment of 2 master-slave setups.
One of the master-slave setups is OK (so far) but on the other both the
master and the slave keep freezing, but only after I send a query to them.
And by freezing I mean indefinite hanging, with almost no output to log, no
errors, nothing. It's as if there's some sort of a deadlock. The hanging
servers need to be killed with -9, otherwise they keep hanging.
The query I send queries all instances at the same time using the ?shards=
syntax.
On the slave, the logs just stop - nothing shows up anymore after the query
is issued. On the master, they're a bit more descriptive. This information
seeps through very-very slowly, as you can see from the timestamps:
{quote}
SEVERE: java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:16:00 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:37 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:37 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:39 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Exception in thread ContainerBackException in thread pool-29-threadOct 1,
2009 2:21:47 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
-22 java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Exception in thread http-8080-42 Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Exception in thread http-8080-26 Exception in thread http-8080-32
Exception in thread http-8080-25 Exception in thread http-8080-22
Exception in thread http-8080-15 Exception in thread http-8080-45
Exception in thread http-8080-13 Exception in thread http-8080-48
Exception in thread http-8080-7 Exception in thread http-8080-38
Exception in thread http-8080-39 Exception in thread http-8080-28
[jira] Commented: (SOLR-1482) Solr master and slave freeze after query
[
https://issues.apache.org/jira/browse/SOLR-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12761449#action_12761449
]
Artem Russakovskii commented on SOLR-1482:
--
I'm getting an error even just trying to access a single shard's admin
interface, even after adjusting -XX:MaxPermSize=512m
{quote}
== catalina.out ==
Oct 1, 2009 6:47:06 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
at java.lang.Throwable.getStackTraceElement(Native Method)
at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
at java.lang.Throwable.printStackTrace(Throwable.java:510)
at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:72)
at org.apache.juli.FileHandler.publish(FileHandler.java:129)
at java.util.logging.Logger.log(Logger.java:458)
at java.util.logging.Logger.doLog(Logger.java:480)
at java.util.logging.Logger.logp(Logger.java:680)
at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:167)
at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:135)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:324)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:619)
{quote}
:-/
Solr master and slave freeze after query
Key: SOLR-1482
URL: https://issues.apache.org/jira/browse/SOLR-1482
Project: Solr
Issue Type: Bug
Affects Versions: 1.4
Environment: Nightly 9/28/09.
14 individual instances per server, using JNDI.
replicateAfter commit, 5 min interval polling.
All caches are currently commented out, on both slave and master.
Lots of ongoing commits - large chunks of data, each accompanied by a commit.
This is to guarantee that anything we think is now in Solr remains there in
case the server crashes.
Reporter: Artem Russakovskii
Priority: Critical
We're having issues with the deployment of 2 master-slave setups.
One of the master-slave setups is OK (so far) but on the other both the
master and the slave keep freezing, but only after I send a query to them.
And by freezing I mean indefinite hanging, with almost no output to log, no
errors, nothing. It's as if there's some sort of a deadlock. The hanging
servers need to be killed with -9, otherwise they keep hanging.
The query I send queries all instances at the same time using the ?shards=
syntax.
On the slave, the logs just stop - nothing shows up anymore after the query
is issued. On the master, they're a bit more descriptive. This information
seeps through very-very slowly, as you can see from the timestamps:
{quote}
SEVERE: java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:16:00 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:37 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:37 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:39 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Exception in thread ContainerBackException in thread pool-29-threadOct 1,
2009 2:21:47 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
-22 java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Exception in thread http-8080-42 Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
Re: [jira] Commented: (SOLR-1482) Solr master and slave freeze after query
What is your index size? if you have enough RAM , try out a bigger
perm gen size. If you shard, you will not be able to use the same
data dir. The data has to be split among shards. Which mean
reindexing.
--Noble
On Fri, Oct 2, 2009 at 7:20 AM, Artem Russakovskii (JIRA)
j...@apache.org wrote:
[
https://issues.apache.org/jira/browse/SOLR-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12761449#action_12761449
]
Artem Russakovskii commented on SOLR-1482:
--
I'm getting an error even just trying to access a single shard's admin
interface, even after adjusting -XX:MaxPermSize=512m
{quote}
== catalina.out ==
Oct 1, 2009 6:47:06 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
at java.lang.Throwable.getStackTraceElement(Native Method)
at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
at java.lang.Throwable.printStackTrace(Throwable.java:510)
at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:72)
at org.apache.juli.FileHandler.publish(FileHandler.java:129)
at java.util.logging.Logger.log(Logger.java:458)
at java.util.logging.Logger.doLog(Logger.java:480)
at java.util.logging.Logger.logp(Logger.java:680)
at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:167)
at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:135)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:324)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:619)
{quote}
:-/
Solr master and slave freeze after query
Key: SOLR-1482
URL: https://issues.apache.org/jira/browse/SOLR-1482
Project: Solr
Issue Type: Bug
Affects Versions: 1.4
Environment: Nightly 9/28/09.
14 individual instances per server, using JNDI.
replicateAfter commit, 5 min interval polling.
All caches are currently commented out, on both slave and master.
Lots of ongoing commits - large chunks of data, each accompanied by a
commit. This is to guarantee that anything we think is now in Solr remains
there in case the server crashes.
Reporter: Artem Russakovskii
Priority: Critical
We're having issues with the deployment of 2 master-slave setups.
One of the master-slave setups is OK (so far) but on the other both the
master and the slave keep freezing, but only after I send a query to them.
And by freezing I mean indefinite hanging, with almost no output to log, no
errors, nothing. It's as if there's some sort of a deadlock. The hanging
servers need to be killed with -9, otherwise they keep hanging.
The query I send queries all instances at the same time using the ?shards=
syntax.
On the slave, the logs just stop - nothing shows up anymore after the query
is issued. On the master, they're a bit more descriptive. This information
seeps through very-very slowly, as you can see from the timestamps:
{quote}
SEVERE: java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:16:00 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:37 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:37 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:19:39 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Exception in thread ContainerBackException in thread pool-29-threadOct 1,
2009 2:21:47 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError: PermGen space
-22 java.lang.OutOfMemoryError: PermGen space
Oct 1, 2009 2:21:47 PM
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
SEVERE: Error reading
