Re: Solr Nodes Killed During a ReIndexing Process on New VMs Out of Memory Error

[THADC]

Thanks, made heap size considerably larger and its fine now. Thank you



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr Nodes Killed During a ReIndexing Process on New VMs Out of Memory Error

[Shawn Heisey]

On 7/18/2018 8:31 AM, THADC wrote:

Thanks for the reply. I read the link you provided. I am currently not
specifying a heap size with solr so my understanding is that by default it
will just grow automatically. If I add more physical memory to the VM
without doing anything with heap size, won't that possibly fix the problem?


No, that is not how it works.  If Java is not given a heap size, then it 
will choose the heap size for you based on how much memory the machine 
has and its own internal algorithms, and limit itself to that amount.


Solr 5.0 and later, when started using the included scripts, asks Java 
for a 512MB heap by default.  This is an extremely small heap.  Nearly 
all Solr users must increase the heap size beyond 512MB.


Thanks,
Shawn

Re: Solr Nodes Killed During a ReIndexing Process on New VMs Out of Memory Error

[THADC]

Thanks for the reply. I read the link you provided. I am currently not
specifying a heap size with solr so my understanding is that by default it
will just grow automatically. If I add more physical memory to the VM
without doing anything with heap size, won't that possibly fix the problem?

Thanks



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr Nodes Killed During a ReIndexing Process on New VMs Out of Memory Error

[Shawn Heisey]

On 7/18/2018 7:10 AM, THADC wrote:

We performed a full reindex for the first time against our largest database
and on two new VMs dedicated to solr indexing. We have two solr nodes
(solrCloud/solr7.3) with a zookeeper cluster. Several hours into the
reindexing process, both solr nodes shut down with:

java.long.OutOfMemoryError: Java heap space

Running OOM killer script for process blah on port blah

Does this definitely indicate we need more memory or could it just be a heap
space adjustment issue? Is there a way to better diagnose what to do?


https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

There are exactly two ways to deal with OOME:  Increase the available 
amount of the resource that's running out (heap space in this case), or 
change something so the program requires less of that resource.  
Depending on the hardware and software configuration, either of these 
options might turn out to be impossible.


The rest of that wiki page discusses memory in general as well as heap 
memory.  If you have questions after reading the page, go ahead and ask 
them.


Thanks,
Shawn

Solr Nodes Killed During a ReIndexing Process on New VMs Out of Memory Error

[THADC]

Hi,

We performed a full reindex for the first time against our largest database
and on two new VMs dedicated to solr indexing. We have two solr nodes
(solrCloud/solr7.3) with a zookeeper cluster. Several hours into the
reindexing process, both solr nodes shut down with:

java.long.OutOfMemoryError: Java heap space

Running OOM killer script for process blah on port blah

Does this definitely indicate we need more memory or could it just be a heap
space adjustment issue? Is there a way to better diagnose what to do?

Thanks for any reply.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Out of memory error during full import

[Shawn Heisey]

On 2/4/2016 12:18 AM, Srinivas Kashyap wrote:
> I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the 
> child entities in data-config.xml. When i try to do full import, i'm getting 
> OutOfMemory error(Java Heap Space). I increased the HEAP allocation to the 
> maximum extent possible. Is there a workaround to do initial data load 
> without running into this error?
>
> I found that 'batchSize=-1' parameter needs to be specified in the datasource 
> for MySql, is there a way to specify for others Databases as well?

Setting batchSize to -1 in the DIH config translates to a 'setFetchSize'
on the JDBC object of Integer.MIN_VALUE.  This is how to turn on result
streaming in MySQL.

The method for doing this with other JDBC implementations is likely to
be different.  The Microsoft driver for SQL Server uses a URL parameter,
and newer versions of that particular driver have the streaming behavior
as default.  I have no idea how to do it for any other driver, you would
need to ask the author of the driver.

When you turn on caching (SortedMapBackedCache), you are asking Solr to
put all of the data received into memory -- very similar to what happens
if result streaming is not turned on.  When the SQL result is very
large, this can require a LOT of memory.  In situations like that,
you'll just have to remove the caching.  One alternative to child
entities is to do a query using JOIN in a single entity, so that all the
data you need is returned by a single SQL query, where the heavy lifting
is done by the database server instead of Solr.

The MySQL database that serves as the information source for *my* Solr
index is hundreds of gigabytes in size, so caching it is not possible
for me.  The batchSize=-1 option is the only way to get the import to work.

Thanks,
Shawn

Out of memory error during full import

[Srinivas Kashyap]

Hello,

I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the 
child entities in data-config.xml. When i try to do full import, i'm getting 
OutOfMemory error(Java Heap Space). I increased the HEAP allocation to the 
maximum extent possible. Is there a workaround to do initial data load without 
running into this error?

I found that 'batchSize=-1' parameter needs to be specified in the datasource 
for MySql, is there a way to specify for others Databases as well?

Thanks and Regards,
Srinivas Kashyap
DISCLAIMER: E-mails and attachments from Bamboo Rose, Inc. are confidential. If 
you are not the intended recipient, please notify the sender immediately by 
replying to the e-mail, and then delete it without making copies or using it in 
any way. No representation is made that this email or any attachments are free 
of viruses. Virus scanning is recommended and is the responsibility of the 
recipient.

Out of memory error during full import

[Srinivas Kashyap]

Hello,

I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the 
child entities in data-config.xml. When i try to do full import, i'm getting 
OutOfMemory error(Java Heap Space). I increased the HEAP allocation to the 
maximum extent possible. Is there a workaround to do initial data load without 
running into this error?

I found that 'batchSize=-1' parameter needs to be specified in the datasource 
for MySql, is there a way to specify for others Databases as well?

Thanks and Regards,
Srinivas Kashyap

DISCLAIMER: 
E-mails and attachments from TradeStone Software, Inc. are confidential.
If you are not the intended recipient, please notify the sender immediately by
replying to the e-mail, and then delete it without making copies or using it
in any way. No representation is made that this email or any attachments are
free of viruses. Virus scanning is recommended and is the responsibility of
the recipient.

Re: need help from hard core solr experts - out of memory error

[Candygram For Mongo]

We have tried using fetchSize and we still got the same out of memory
errors.


On Fri, Apr 18, 2014 at 9:39 PM, Shawn Heisey s...@elyograg.org wrote:

 On 4/18/2014 6:15 PM, Candygram For Mongo wrote:
  We are getting Out Of Memory errors when we try to execute a full import
  using the Data Import Handler.  This error originally occurred on a
  production environment with a database containing 27 million records.
  Heap
  memory was configured for 6GB and the server had 32GB of physical memory.
   We have been able to replicate the error on a local system with 6
 million
  records.  We set the memory heap size to 64MB to accelerate the error
  replication.  The indexing process has been failing in different
 scenarios.
   We have 9 test cases documented.  In some of the test cases we increased
  the heap size to 128MB.  In our first test case we set heap memory to
 512MB
  which also failed.

 One characteristic of a JDBC connection is that unless you tell it
 otherwise, it will try to retrieve the entire resultset into RAM before
 any results are delivered to the application.  It's not Solr doing this,
 it's JDBC.

 In this case, there are 27 million rows in the resultset.  It's highly
 unlikely that this much data (along with the rest of Solr's memory
 requirements) will fit in 6GB of heap.

 JDBC has a built-in way to deal with this.  It's called fetchSize.  By
 using the batchSize parameter on your JdbcDataSource config, you can set
 the JDBC fetchSize.  Set it to something small, between 100 and 1000,
 and you'll probably get rid of the OOM problem.

 http://wiki.apache.org/solr/DataImportHandler#Configuring_JdbcDataSource

 If you had been using MySQL, I would have recommended that you set
 batchSize to -1.  This sets fetchSize to Integer.MIN_VALUE, which tells
 the MySQL driver to stream results instead of trying to either batch
 them or return everything.  I'm pretty sure that the Oracle driver
 doesn't work this way -- you would have to modify the dataimport source
 code to use their streaming method.

 Thanks,
 Shawn

Re: need help from hard core solr experts - out of memory error

[Shawn Heisey]

On 4/20/2014 11:12 AM, Candygram For Mongo wrote:
 We have tried using fetchSize and we still got the same out of memory
 errors.

It needs to be batchSize, not fetchSize.  I mentioned too much of the
internal details.  The fetchSize name is only important if you're
writing source code that uses JDBC.

http://wiki.apache.org/solr/DataImportHandler#Configuring_JdbcDataSource

Thanks,
Shawn

Re: need help from hard core solr experts - out of memory error

[Mikhail Khludnev]

I noticed enormous number of commits, which reasonably triggers merges that
hits OOMe. Try to disable autocommits completely. Monitor commit
occurrences in the log.


On Sun, Apr 20, 2014 at 9:12 PM, Candygram For Mongo 
candygram.for.mo...@gmail.com wrote:

 We have tried using fetchSize and we still got the same out of memory
 errors.


 On Fri, Apr 18, 2014 at 9:39 PM, Shawn Heisey s...@elyograg.org wrote:

  On 4/18/2014 6:15 PM, Candygram For Mongo wrote:
   We are getting Out Of Memory errors when we try to execute a full
 import
   using the Data Import Handler.  This error originally occurred on a
   production environment with a database containing 27 million records.
   Heap
   memory was configured for 6GB and the server had 32GB of physical
 memory.
We have been able to replicate the error on a local system with 6
  million
   records.  We set the memory heap size to 64MB to accelerate the error
   replication.  The indexing process has been failing in different
  scenarios.
We have 9 test cases documented.  In some of the test cases we
 increased
   the heap size to 128MB.  In our first test case we set heap memory to
  512MB
   which also failed.
 
  One characteristic of a JDBC connection is that unless you tell it
  otherwise, it will try to retrieve the entire resultset into RAM before
  any results are delivered to the application.  It's not Solr doing this,
  it's JDBC.
 
  In this case, there are 27 million rows in the resultset.  It's highly
  unlikely that this much data (along with the rest of Solr's memory
  requirements) will fit in 6GB of heap.
 
  JDBC has a built-in way to deal with this.  It's called fetchSize.  By
  using the batchSize parameter on your JdbcDataSource config, you can set
  the JDBC fetchSize.  Set it to something small, between 100 and 1000,
  and you'll probably get rid of the OOM problem.
 
  http://wiki.apache.org/solr/DataImportHandler#Configuring_JdbcDataSource
 
  If you had been using MySQL, I would have recommended that you set
  batchSize to -1.  This sets fetchSize to Integer.MIN_VALUE, which tells
  the MySQL driver to stream results instead of trying to either batch
  them or return everything.  I'm pretty sure that the Oracle driver
  doesn't work this way -- you would have to modify the dataimport source
  code to use their streaming method.
 
  Thanks,
  Shawn
 
 




-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

http://www.griddynamics.com
 mkhlud...@griddynamics.com

need help from hard core solr experts - out of memory error

[Candygram For Mongo]

I have lots of log files and other files to support this issue (sometimes
referenced in the text below) but I am not sure the best way to submit.  I
don't want to overwhelm and I am not sure if this email will accept graphs
and charts.  Please provide direction and I will send them.


*Issue Description*



We are getting Out Of Memory errors when we try to execute a full import
using the Data Import Handler.  This error originally occurred on a
production environment with a database containing 27 million records.  Heap
memory was configured for 6GB and the server had 32GB of physical memory.
 We have been able to replicate the error on a local system with 6 million
records.  We set the memory heap size to 64MB to accelerate the error
replication.  The indexing process has been failing in different scenarios.
 We have 9 test cases documented.  In some of the test cases we increased
the heap size to 128MB.  In our first test case we set heap memory to 512MB
which also failed.





*Environment Values Used*



*SOLR/Lucene version: *4.2.1*

*JVM version:

Java(TM) SE Runtime Environment (build 1.7.0_07-b11)

Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)

*Indexer startup command:

set JVMARGS= -XX:MaxPermSize=364m -Xss256K –Xmx128m –Xms128m

java  %JVMARGS% ^

-Dcom.sun.management.jmxremote.port=1092 ^

-Dcom.sun.management.jmxremote.ssl=false ^

-Dcom.sun.management.jmxremote.authenticate=false ^

-jar start.jar

*SOLR indexing HTTP parameters request:

webapp=/solr path=/dataimport
params={clean=falsecommand=full-importwt=javabinversion=2}



The information we use for the database retrieve using the Data Import
Handler is as follows:



dataSource

name=org_only

type=JdbcDataSource

driver=oracle.jdbc.OracleDriver

url=jdbc:oracle:thin:@{server name}:1521:{database
name}

user={username}

password={password}

readOnly=false

/





*The Query (simple, single table)*



*select*



*NVL(cast(STU.ACCT_ADDRESS_ALL.R_ID as varchar2(100)), 'null')*

*as SOLR_ID,*



*'STU.ACCT_ADDRESS_ALL'*

*as SOLR_CATEGORY,*



*NVL(cast(STU.ACCT_ADDRESS_ALL.R_ID as varchar2(255)), ' ') as
ADDRESSALLRID,*

*NVL(cast(STU.ACCT_ADDRESS_ALL.ADDR_TYPE as varchar2(255)), ' ') as
ADDRESSALLADDRTYPECD,*

*NVL(cast(STU.ACCT_ADDRESS_ALL.LONGITUDE as varchar2(255)), ' ') as
ADDRESSALLLONGITUDE,*

*NVL(cast(STU.ACCT_ADDRESS_ALL.LATITUDE as varchar2(255)), ' ') as
ADDRESSALLLATITUDE,*

*NVL(cast(STU.ACCT_ADDRESS_ALL.ADDR_NAME as varchar2(255)), ' ') as
ADDRESSALLADDRNAME,*

*NVL(cast(STU.ACCT_ADDRESS_ALL.CITY as varchar2(255)), ' ') as
ADDRESSALLCITY,*

*NVL(cast(STU.ACCT_ADDRESS_ALL.STATE as varchar2(255)), ' ') as
ADDRESSALLSTATE,*

*NVL(cast(STU.ACCT_ADDRESS_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
ADDRESSALLEMAILADDR *



*from STU.ACCT_ADDRESS_ALL*



You can see this information in the database.xml file.



Our main solrconfig.xml file contains the following differences compared to
a new downloaded solrconfig.xml
filefile:///D:/Solr%20Full%20Indexing%20issue/solrconfig%20(default%20content).xml(the
original content).



config

lib dir=../../../dist/
regex=solr-dataimporthandler-.*\.jar /

!—Our libraries containing customized filters--

lib path=../../../../default/lib/common.jar /

lib path=../../../../default/lib/webapp.jar /

lib path=../../../../default/lib/commons-pool-1.4.jar /



abortOnConfigurationError${solr.abortOnConfigurationError:true}/abortOnConfigurationError



directoryFactory name=DirectoryFactory
class=org.apache.solr.core.StandardDirectoryFactory /



requestHandler name=/dataimport
class=org.apache.solr.handler.dataimport.DataImportHandler

lst name=defaults

str name=configdatabase.xml/str

/lst

/requestHandler

/config





*Custom Libraries*



The common.jar contains a customized TokenFiltersFactory implementation
that we use for indexing.  They do some special treatment to the fields
read from the database.  How those classes are used is described in the
schema.xml file.  The webapp.jar file contains other related classes.  The
commons-pool-1.4.jar is an API from apache used for instances reuse.



The logic used in the TokenFiltersFactory is contained in the following
files:




ConcatFilterFactory.javafile:///D:/Solr%20Full%20Indexing%20issue/source%20files/ConcatFilterFactory.java


ConcatFilter.javafile:///D:/Solr%20Full%20Indexing%20issue/source%20files/ConcatFilter.java


MDFilterSchemaFactory.javafile:///D:/Solr%20Full%20Indexing%20issue/source%20files/MDFilterSchemaFactory.java


MDFilter.javafile:///D:/Solr%20Full%20Indexing%20issue/source%20files/MDFilter.java


MDFilterPoolObjectFactory.javafile:///D:/Solr%20Full%20Indexing%20issue/source%20files/MDFilterPoolObjectFactory.java

Re: need help from hard core solr experts - out of memory error

[Walter Underwood]

I see heap size commands for 128 Meg and 512 Meg. That will certainly run out 
of memory. Why do you think you have 6G of heap with these settings?

–Xmx128m –Xms128m
–Xmx512m –Xms512m

wunder

On Apr 18, 2014, at 5:15 PM, Candygram For Mongo 
candygram.for.mo...@gmail.com wrote:

 I have lots of log files and other files to support this issue (sometimes
 referenced in the text below) but I am not sure the best way to submit.  I
 don't want to overwhelm and I am not sure if this email will accept graphs
 and charts.  Please provide direction and I will send them.
 
 
 *Issue Description*
 
 
 
 We are getting Out Of Memory errors when we try to execute a full import
 using the Data Import Handler.  This error originally occurred on a
 production environment with a database containing 27 million records.  Heap
 memory was configured for 6GB and the server had 32GB of physical memory.
 We have been able to replicate the error on a local system with 6 million
 records.  We set the memory heap size to 64MB to accelerate the error
 replication.  The indexing process has been failing in different scenarios.
 We have 9 test cases documented.  In some of the test cases we increased
 the heap size to 128MB.  In our first test case we set heap memory to 512MB
 which also failed.
 
 
 
 
 
 *Environment Values Used*
 
 
 
 *SOLR/Lucene version: *4.2.1*
 
 *JVM version:
 
 Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
 
 Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
 
 *Indexer startup command:
 
 set JVMARGS= -XX:MaxPermSize=364m -Xss256K –Xmx128m –Xms128m
 
 java  %JVMARGS% ^
 
 -Dcom.sun.management.jmxremote.port=1092 ^
 
 -Dcom.sun.management.jmxremote.ssl=false ^
 
 -Dcom.sun.management.jmxremote.authenticate=false ^
 
 -jar start.jar
 
 *SOLR indexing HTTP parameters request:
 
 webapp=/solr path=/dataimport
 params={clean=falsecommand=full-importwt=javabinversion=2}
 
 
 
 The information we use for the database retrieve using the Data Import
 Handler is as follows:
 
 
 
 dataSource
 
name=org_only
 
type=JdbcDataSource
 
driver=oracle.jdbc.OracleDriver
 
url=jdbc:oracle:thin:@{server name}:1521:{database
 name}
 
user={username}
 
password={password}
 
readOnly=false
 
/
 
 
 
 
 
 *The Query (simple, single table)*
 
 
 
 *select*
 
 
 
 *NVL(cast(STU.ACCT_ADDRESS_ALL.R_ID as varchar2(100)), 'null')*
 
 *as SOLR_ID,*
 
 
 
 *'STU.ACCT_ADDRESS_ALL'*
 
 *as SOLR_CATEGORY,*
 
 
 
 *NVL(cast(STU.ACCT_ADDRESS_ALL.R_ID as varchar2(255)), ' ') as
 ADDRESSALLRID,*
 
 *NVL(cast(STU.ACCT_ADDRESS_ALL.ADDR_TYPE as varchar2(255)), ' ') as
 ADDRESSALLADDRTYPECD,*
 
 *NVL(cast(STU.ACCT_ADDRESS_ALL.LONGITUDE as varchar2(255)), ' ') as
 ADDRESSALLLONGITUDE,*
 
 *NVL(cast(STU.ACCT_ADDRESS_ALL.LATITUDE as varchar2(255)), ' ') as
 ADDRESSALLLATITUDE,*
 
 *NVL(cast(STU.ACCT_ADDRESS_ALL.ADDR_NAME as varchar2(255)), ' ') as
 ADDRESSALLADDRNAME,*
 
 *NVL(cast(STU.ACCT_ADDRESS_ALL.CITY as varchar2(255)), ' ') as
 ADDRESSALLCITY,*
 
 *NVL(cast(STU.ACCT_ADDRESS_ALL.STATE as varchar2(255)), ' ') as
 ADDRESSALLSTATE,*
 
 *NVL(cast(STU.ACCT_ADDRESS_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
 ADDRESSALLEMAILADDR *
 
 
 
 *from STU.ACCT_ADDRESS_ALL*
 
 
 
 You can see this information in the database.xml file.
 
 
 
 Our main solrconfig.xml file contains the following differences compared to
 a new downloaded solrconfig.xml
 filefile:///D:/Solr%20Full%20Indexing%20issue/solrconfig%20(default%20content).xml(the
 original content).
 
 
 
 config
 
lib dir=../../../dist/
 regex=solr-dataimporthandler-.*\.jar /
 
!—Our libraries containing customized filters--
 
lib path=../../../../default/lib/common.jar /
 
lib path=../../../../default/lib/webapp.jar /
 
lib path=../../../../default/lib/commons-pool-1.4.jar /
 
 
 
 abortOnConfigurationError${solr.abortOnConfigurationError:true}/abortOnConfigurationError
 
 
 
 directoryFactory name=DirectoryFactory
 class=org.apache.solr.core.StandardDirectoryFactory /
 
 
 
 requestHandler name=/dataimport
 class=org.apache.solr.handler.dataimport.DataImportHandler
 
 lst name=defaults
 
 str name=configdatabase.xml/str
 
 /lst
 
 /requestHandler
 
 /config
 
 
 
 
 
 *Custom Libraries*
 
 
 
 The common.jar contains a customized TokenFiltersFactory implementation
 that we use for indexing.  They do some special treatment to the fields
 read from the database.  How those classes are used is described in the
 schema.xml file.  The webapp.jar file contains other related classes.  The
 commons-pool-1.4.jar is an API from apache used for instances reuse.
 
 
 
 The logic used in the TokenFiltersFactory is contained in the following
 files:
 
 
 

 

Re: need help from hard core solr experts - out of memory error

[Candygram For Mongo]

We consistently reproduce this problem on multiple systems configured with
6GB and 12GB of heap space.  To quickly reproduce many cases for
troubleshooting we reduced the heap space to 64, 128 and 512MB.  With 6 or
12GB configured it takes hours to see the error.


On Fri, Apr 18, 2014 at 5:54 PM, Walter Underwood wun...@wunderwood.orgwrote:

 I see heap size commands for 128 Meg and 512 Meg. That will certainly run
 out of memory. Why do you think you have 6G of heap with these settings?

 –Xmx128m –Xms128m
 –Xmx512m –Xms512m

 wunder

 On Apr 18, 2014, at 5:15 PM, Candygram For Mongo 
 candygram.for.mo...@gmail.com wrote:

  I have lots of log files and other files to support this issue (sometimes
  referenced in the text below) but I am not sure the best way to submit.
  I
  don't want to overwhelm and I am not sure if this email will accept
 graphs
  and charts.  Please provide direction and I will send them.
 
 
  *Issue Description*
 
 
 
  We are getting Out Of Memory errors when we try to execute a full import
  using the Data Import Handler.  This error originally occurred on a
  production environment with a database containing 27 million records.
  Heap
  memory was configured for 6GB and the server had 32GB of physical memory.
  We have been able to replicate the error on a local system with 6 million
  records.  We set the memory heap size to 64MB to accelerate the error
  replication.  The indexing process has been failing in different
 scenarios.
  We have 9 test cases documented.  In some of the test cases we increased
  the heap size to 128MB.  In our first test case we set heap memory to
 512MB
  which also failed.
 
 
 
 
 
  *Environment Values Used*
 
 
 
  *SOLR/Lucene version: *4.2.1*
 
  *JVM version:
 
  Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
 
  Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
 
  *Indexer startup command:
 
  set JVMARGS= -XX:MaxPermSize=364m -Xss256K –Xmx128m –Xms128m
 
  java  %JVMARGS% ^
 
  -Dcom.sun.management.jmxremote.port=1092 ^
 
  -Dcom.sun.management.jmxremote.ssl=false ^
 
  -Dcom.sun.management.jmxremote.authenticate=false ^
 
  -jar start.jar
 
  *SOLR indexing HTTP parameters request:
 
  webapp=/solr path=/dataimport
  params={clean=falsecommand=full-importwt=javabinversion=2}
 
 
 
  The information we use for the database retrieve using the Data Import
  Handler is as follows:
 
 
 
  dataSource
 
 name=org_only
 
 type=JdbcDataSource
 
 driver=oracle.jdbc.OracleDriver
 
 url=jdbc:oracle:thin:@{server
 name}:1521:{database
  name}
 
 user={username}
 
 password={password}
 
 readOnly=false
 
 /
 
 
 
 
 
  *The Query (simple, single table)*
 
 
 
  *select*
 
 
 
  *NVL(cast(STU.ACCT_ADDRESS_ALL.R_ID as varchar2(100)), 'null')*
 
  *as SOLR_ID,*
 
 
 
  *'STU.ACCT_ADDRESS_ALL'*
 
  *as SOLR_CATEGORY,*
 
 
 
  *NVL(cast(STU.ACCT_ADDRESS_ALL.R_ID as varchar2(255)), ' ') as
  ADDRESSALLRID,*
 
  *NVL(cast(STU.ACCT_ADDRESS_ALL.ADDR_TYPE as varchar2(255)), ' ') as
  ADDRESSALLADDRTYPECD,*
 
  *NVL(cast(STU.ACCT_ADDRESS_ALL.LONGITUDE as varchar2(255)), ' ') as
  ADDRESSALLLONGITUDE,*
 
  *NVL(cast(STU.ACCT_ADDRESS_ALL.LATITUDE as varchar2(255)), ' ') as
  ADDRESSALLLATITUDE,*
 
  *NVL(cast(STU.ACCT_ADDRESS_ALL.ADDR_NAME as varchar2(255)), ' ') as
  ADDRESSALLADDRNAME,*
 
  *NVL(cast(STU.ACCT_ADDRESS_ALL.CITY as varchar2(255)), ' ') as
  ADDRESSALLCITY,*
 
  *NVL(cast(STU.ACCT_ADDRESS_ALL.STATE as varchar2(255)), ' ') as
  ADDRESSALLSTATE,*
 
  *NVL(cast(STU.ACCT_ADDRESS_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
  ADDRESSALLEMAILADDR *
 
 
 
  *from STU.ACCT_ADDRESS_ALL*
 
 
 
  You can see this information in the database.xml file.
 
 
 
  Our main solrconfig.xml file contains the following differences compared
 to
  a new downloaded solrconfig.xml
 
 filefile:///D:/Solr%20Full%20Indexing%20issue/solrconfig%20(default%20content).xml(the
  original content).
 
 
 
  config
 
 lib dir=../../../dist/
  regex=solr-dataimporthandler-.*\.jar /
 
 !—Our libraries containing customized filters--
 
 lib path=../../../../default/lib/common.jar /
 
 lib path=../../../../default/lib/webapp.jar /
 
 lib path=../../../../default/lib/commons-pool-1.4.jar /
 
 
 
 
 abortOnConfigurationError${solr.abortOnConfigurationError:true}/abortOnConfigurationError
 
 
 
  directoryFactory name=DirectoryFactory
  class=org.apache.solr.core.StandardDirectoryFactory /
 
 
 
  requestHandler name=/dataimport
  class=org.apache.solr.handler.dataimport.DataImportHandler
 
  lst name=defaults
 
  str name=configdatabase.xml/str
 
  /lst
 
  /requestHandler
 
  /config
 
 
 
 
 
  *Custom Libraries*
 
 
 
  The common.jar contains a customized TokenFiltersFactory implementation
  that we use for indexing.  They do some 

Re: need help from hard core solr experts - out of memory error

[Candygram For Mongo]

I have uploaded several files including the problem description with
graphics to this link on Google drive:

https://drive.google.com/folderview?id=0B7UpFqsS5lSjWEhxRE1NN2tMNTQusp=sharing

I shared it with this address solr-user@lucene.apache.org so I am hoping
it can be accessed by people in the group.


On Fri, Apr 18, 2014 at 5:15 PM, Candygram For Mongo 
candygram.for.mo...@gmail.com wrote:

 I have lots of log files and other files to support this issue (sometimes
 referenced in the text below) but I am not sure the best way to submit.  I
 don't want to overwhelm and I am not sure if this email will accept graphs
 and charts.  Please provide direction and I will send them.


 *Issue Description*



 We are getting Out Of Memory errors when we try to execute a full import
 using the Data Import Handler.  This error originally occurred on a
 production environment with a database containing 27 million records.  Heap
 memory was configured for 6GB and the server had 32GB of physical memory.
  We have been able to replicate the error on a local system with 6 million
 records.  We set the memory heap size to 64MB to accelerate the error
 replication.  The indexing process has been failing in different scenarios.
  We have 9 test cases documented.  In some of the test cases we increased
 the heap size to 128MB.  In our first test case we set heap memory to 512MB
 which also failed.





 *Environment Values Used*



 *SOLR/Lucene version: *4.2.1*

 *JVM version:

 Java(TM) SE Runtime Environment (build 1.7.0_07-b11)

 Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)

 *Indexer startup command:

 set JVMARGS= -XX:MaxPermSize=364m -Xss256K –Xmx128m –Xms128m

 java  %JVMARGS% ^

 -Dcom.sun.management.jmxremote.port=1092 ^

 -Dcom.sun.management.jmxremote.ssl=false ^

 -Dcom.sun.management.jmxremote.authenticate=false ^

 -jar start.jar

 *SOLR indexing HTTP parameters request:

 webapp=/solr path=/dataimport
 params={clean=falsecommand=full-importwt=javabinversion=2}



 The information we use for the database retrieve using the Data Import
 Handler is as follows:



 dataSource

 name=org_only

 type=JdbcDataSource

 driver=oracle.jdbc.OracleDriver

 url=jdbc:oracle:thin:@{server
 name}:1521:{database name}

 user={username}

 password={password}

 readOnly=false

 /





 *The Query (simple, single table)*



 *select*



 *NVL(cast(STU.ACCT_ADDRESS_ALL.R_ID as varchar2(100)), 'null')*

 *as SOLR_ID,*



 *'STU.ACCT_ADDRESS_ALL'*

 *as SOLR_CATEGORY,*



 *NVL(cast(STU.ACCT_ADDRESS_ALL.R_ID as varchar2(255)), ' ') as
 ADDRESSALLRID,*

 *NVL(cast(STU.ACCT_ADDRESS_ALL.ADDR_TYPE as varchar2(255)), ' ') as
 ADDRESSALLADDRTYPECD,*

 *NVL(cast(STU.ACCT_ADDRESS_ALL.LONGITUDE as varchar2(255)), ' ') as
 ADDRESSALLLONGITUDE,*

 *NVL(cast(STU.ACCT_ADDRESS_ALL.LATITUDE as varchar2(255)), ' ') as
 ADDRESSALLLATITUDE,*

 *NVL(cast(STU.ACCT_ADDRESS_ALL.ADDR_NAME as varchar2(255)), ' ') as
 ADDRESSALLADDRNAME,*

 *NVL(cast(STU.ACCT_ADDRESS_ALL.CITY as varchar2(255)), ' ') as
 ADDRESSALLCITY,*

 *NVL(cast(STU.ACCT_ADDRESS_ALL.STATE as varchar2(255)), ' ') as
 ADDRESSALLSTATE,*

 *NVL(cast(STU.ACCT_ADDRESS_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
 ADDRESSALLEMAILADDR *



 *from STU.ACCT_ADDRESS_ALL*



 You can see this information in the database.xml file.



 Our main solrconfig.xml file contains the following differences compared
 to a new downloaded solrconfig.xml file (the original content).



 config

 lib dir=../../../dist/
 regex=solr-dataimporthandler-.*\.jar /

 !—Our libraries containing customized filters--

 lib path=../../../../default/lib/common.jar /

 lib path=../../../../default/lib/webapp.jar /

 lib path=../../../../default/lib/commons-pool-1.4.jar /




 abortOnConfigurationError${solr.abortOnConfigurationError:true}/abortOnConfigurationError



 directoryFactory name=DirectoryFactory
 class=org.apache.solr.core.StandardDirectoryFactory /



 requestHandler name=/dataimport
 class=org.apache.solr.handler.dataimport.DataImportHandler

 lst name=defaults

 str name=configdatabase.xml/str

 /lst

 /requestHandler

 /config





 *Custom Libraries*



 The common.jar contains a customized TokenFiltersFactory implementation
 that we use for indexing.  They do some special treatment to the fields
 read from the database.  How those classes are used is described in the
 schema.xml file.  The webapp.jar file contains other related classes.
 The commons-pool-1.4.jar is an API from apache used for instances reuse.



 The logic used in the TokenFiltersFactory is contained in the following
 files:



 ConcatFilterFactory.java

 ConcatFilter.java

 MDFilterSchemaFactory.java

 MDFilter.java

 

Re: need help from hard core solr experts - out of memory error

[Shawn Heisey]

On 4/18/2014 6:15 PM, Candygram For Mongo wrote:
 We are getting Out Of Memory errors when we try to execute a full import
 using the Data Import Handler.  This error originally occurred on a
 production environment with a database containing 27 million records.  Heap
 memory was configured for 6GB and the server had 32GB of physical memory.
  We have been able to replicate the error on a local system with 6 million
 records.  We set the memory heap size to 64MB to accelerate the error
 replication.  The indexing process has been failing in different scenarios.
  We have 9 test cases documented.  In some of the test cases we increased
 the heap size to 128MB.  In our first test case we set heap memory to 512MB
 which also failed.

One characteristic of a JDBC connection is that unless you tell it
otherwise, it will try to retrieve the entire resultset into RAM before
any results are delivered to the application.  It's not Solr doing this,
it's JDBC.

In this case, there are 27 million rows in the resultset.  It's highly
unlikely that this much data (along with the rest of Solr's memory
requirements) will fit in 6GB of heap.

JDBC has a built-in way to deal with this.  It's called fetchSize.  By
using the batchSize parameter on your JdbcDataSource config, you can set
the JDBC fetchSize.  Set it to something small, between 100 and 1000,
and you'll probably get rid of the OOM problem.

http://wiki.apache.org/solr/DataImportHandler#Configuring_JdbcDataSource

If you had been using MySQL, I would have recommended that you set
batchSize to -1.  This sets fetchSize to Integer.MIN_VALUE, which tells
the MySQL driver to stream results instead of trying to either batch
them or return everything.  I'm pretty sure that the Oracle driver
doesn't work this way -- you would have to modify the dataimport source
code to use their streaming method.

Thanks,
Shawn

Re: Facets with 5000 facet fields - Out of memory error during the query time

[sivaprasad]

I got more information with the responses.Now, It's time to re look into  the
number of facets to be configured.

Thanks,
Siva
http://smarttechies.wordpress.com/



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Facets-with-5000-facet-fields-Out-of-memory-error-during-the-query-time-tp4048450p4059079.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR 4.1 Out Of Memory error After commit of a few thousand Solr Docs

[Rahul Bishnoi]

Hi Shawn,

Thanks for your reply. After following your suggestions we were able to
index 30k documents. I have some queries:

1) What is stored in the RAM while only indexing is going on?  How to
calculate the RAM/heap requirements for our documents?
2) The document cache, filter cache, etc...are populated while querying.
Correct me if I am wrong. Are there any caches that are populated while
indexing?

Thanks,
Rahul



On Sat, Jan 26, 2013 at 11:46 PM, Shawn Heisey s...@elyograg.org wrote:

 On 1/26/2013 12:55 AM, Rahul Bishnoi wrote:

 Thanks for quick reply and addressing each point queried.

 Additional asked information is mentioned below:

 OS = Ubuntu 12.04 (64 bit)
 Sun Java 7 (64 bit)
 Total RAM = 8GB

 SolrConfig.xml is available at http://pastebin.com/SEFxkw2R


 Rahul,

 The MaxPermGenSize could be a contributing factor.  The documents where
 you have 1000 words are somewhat large, though your overall index size is
 pretty small.  I would try removing the MaxPermGenSize option and see what
 happens.  You can also try reducing the ramBufferSizeMB in solrconfig.xml.
  The default in previous versions of Solr was 32, which is big enough for
 most things, unless you are indexing HUGE documents like entire books.

 It looks like you have the cache sizes under query at values close to
 default.  I wouldn't decrease the documentCache any - in fact an increase
 might be a good thing there.  As for the others, you could probably reduce
 them.  The filterCache size I would start at 64 or 128.  Watch your cache
 hitratios to see whether the changes make things remarkably worse.

 If that doesn't help, try increasing the -Xmx option - first 3072m, next
 4096m.  You could go as high as 6GB and not run into any OS cache problems
 with your small index size, though you might run into long GC pauses.

 Indexing, especially big documents, is fairly memory intensive.  Some
 queries can be memory intensive as well, especially those using facets or a
 lot of clauses.

 Under normal operation, I could probably get away with a 3GB heap size,
 but I have it at 8GB because otherwise a full reindex (full-import from
 mysql) runs into OOM errors.

 Thanks,
 Shawn

Re: SOLR 4.1 Out Of Memory error After commit of a few thousand Solr Docs

[Shawn Heisey]

On 1/27/2013 10:28 PM, Rahul Bishnoi wrote:

Thanks for your reply. After following your suggestions we were able to
index 30k documents. I have some queries:

1) What is stored in the RAM while only indexing is going on?  How to
calculate the RAM/heap requirements for our documents?
2) The document cache, filter cache, etc...are populated while querying.
Correct me if I am wrong. Are there any caches that are populated while
indexing?


If anyone catches me making statements that are not true, please feel 
free to correct me.


The caches are indeed only used during querying.  If you are not making 
queries at all, they aren't much of a factor.


I can't give you any definitive answers to your question about RAM usage 
and how to calculate RAM/heap requirements.  I can make some general 
statements without looking at the code, just based on what I've learned 
so far about Solr, and about Java in general.


You would have an exact copy of the input text for each field initially, 
which would ultimately get used for the stored data (for those fields 
that are stored).  Each one is probably just a plain String, though I 
don't know as I haven't read the code.  If the field is not being stored 
or copied, then it would be possible to get rid of that data as soon as 
it is no longer required for indexing.  I don't have any idea whether 
Solr/Lucene code actually gets rid of the exact copy in this way.


If you are storing termvectors, additional memory would be needed for 
that.  I don't know if that involves lots of objects or if it's one 
object with index information.  Based on my experience, termvectors can 
be bigger than the stored data for the same field.


Tokenization and filtering is where I imagine that most of the memory 
would get used.  If you're using a filter like EdgeNGram, that's a LOT 
of tokens.  Even if you're just tokenizing words, it can add up.  There 
is also space required for the inverted index, norms, and other 
data/metadata.  If each token is a separate Java object (which I do not 
know), there would be a fair amount of memory overhead involved.


A String object in java has something like 40 bytes of overhead above 
and beyond the space required for the data.  Also, strings in Java are 
internally represented in UTF-16, so each character actually takes two 
bytes.


http://www.javamex.com/tutorials/memory/string_memory_usage.shtml

The finished documents stack up in the ramBufferSizeMB space until it 
gets full or a hard commit is issued, at which point they are flushed to 
disk as a Lucene segment.  One thing that I'm not sure about is whether 
an additional ram buffer is allocated for further indexing while the 
flush is happening, or if it flushes and then re-uses the buffer for 
subsequent documents.


Another way that it can use memory is when merging index segments.  I 
don't know how much memory gets used for this process.


On Solr 4 with the default directory factory, part of a flushed segment 
may remain in RAM until enough additional segment data is created.  The 
amount of memory used by this feature should be pretty small, unless you 
have a lot of cores on a single JVM.  That extra memory can be 
eliminated by using MMapDirectoryFactory instead of 
NRTCachingDirectoryFactory, at the expense of fast Near-RealTime index 
updates.


Thanks,
Shawn

Re: SOLR 4.1 Out Of Memory error After commit of a few thousand Solr Docs

[Shawn Heisey]

On 1/26/2013 12:55 AM, Rahul Bishnoi wrote:

Thanks for quick reply and addressing each point queried.

Additional asked information is mentioned below:

OS = Ubuntu 12.04 (64 bit)
Sun Java 7 (64 bit)
Total RAM = 8GB

SolrConfig.xml is available at http://pastebin.com/SEFxkw2R



Rahul,

The MaxPermGenSize could be a contributing factor.  The documents where 
you have 1000 words are somewhat large, though your overall index size 
is pretty small.  I would try removing the MaxPermGenSize option and see 
what happens.  You can also try reducing the ramBufferSizeMB in 
solrconfig.xml.  The default in previous versions of Solr was 32, which 
is big enough for most things, unless you are indexing HUGE documents 
like entire books.


It looks like you have the cache sizes under query at values close to 
default.  I wouldn't decrease the documentCache any - in fact an 
increase might be a good thing there.  As for the others, you could 
probably reduce them.  The filterCache size I would start at 64 or 128. 
 Watch your cache hitratios to see whether the changes make things 
remarkably worse.


If that doesn't help, try increasing the -Xmx option - first 3072m, next 
4096m.  You could go as high as 6GB and not run into any OS cache 
problems with your small index size, though you might run into long GC 
pauses.


Indexing, especially big documents, is fairly memory intensive.  Some 
queries can be memory intensive as well, especially those using facets 
or a lot of clauses.


Under normal operation, I could probably get away with a 3GB heap size, 
but I have it at 8GB because otherwise a full reindex (full-import from 
mysql) runs into OOM errors.


Thanks,
Shawn

SOLR 4.1 Out Of Memory error After commit of a few thousand Solr Docs

[Harish Verma]

we are testing solr 4.1 running inside tomcat 7 and java 7 with  following
options

JAVA_OPTS=-Xms256m -Xmx2048m -XX:MaxPermSize=1024m -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode -XX:+ParallelRefProcEnabled
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ubuntu/OOM_HeapDump

our source code looks like following:
/ START */
int noOfSolrDocumentsInBatch = 0;
for(int i=0 ; i5000 ; i++) {
SolrInputDocument solrInputDocument = getNextSolrInputDocument();
server.add(solrInputDocument);
noOfSolrDocumentsInBatch += 1;
if(noOfSolrDocumentsInBatch == 10) {
server.commit();
noOfSolrDocumentsInBatch = 0;
}
}
/ END */

the method getNextSolrInputDocument() generates a solr document with 100
fields (average). Around 50 of the fields are of text_general type.
Some of the test_general fields consist of approx 1000 words rest
consists of few words. Ouf of total fields there are around 35-40
multivalued fields (not of type text_general).
We are indexing all the fields but storing only 8 fields. Out of these 8
fields two are string type, five are long and one is boolean. So our index
size is only 394 MB. But the RAM occupied at time of OOM is around 2.5 GB.
Why the memory is so high even though the index size is small?
What is being stored in the memory? Our understanding is that after every
commit documents are flushed to the disk.So nothing should remain in RAM
after commit.

We are using the following settings:

server.commit() set waitForSearcher=true and waitForFlush=true
solrConfig.xml has following properties set:
directoryFactory = solr.MMapDirectoryFactory
maxWarmingSearchers = 1
text_general data type is being used as supplied in the schema.xml with the
solr setup.
maxIndexingThreads = 8(default)
autoCommitmaxTime15000/maxTimeopenSearcherfalse/openSearcher/autoCommit

We get Java heap Out Of Memory Error after commiting around 3990 solr
documents.Some of the snapshots of memory dump from profiler are uploaded
at following links.
http://s9.postimage.org/w7589t9e7/memorydump1.png
http://s7.postimage.org/p3abs6nuj/memorydump2.png

can somebody please suggest what should we do to minimize/optimize the
memory consumption in our case with the reasons?
also suggest what should be optimal values and reason for following
parameters of solrConfig.xml
useColdSearcher - true/false?
maxwarmingsearchers- number
spellcheck-on/off?
omitNorms=true/false?
omitTermFreqAndPositions?
mergefactor? we are using default value 10
java garbage collection tuning parameters ?


Regards
Harish Verma

Re: SOLR 4.1 Out Of Memory error After commit of a few thousand Solr Docs

[Shawn Heisey]

On 1/25/2013 4:49 AM, Harish Verma wrote:

we are testing solr 4.1 running inside tomcat 7 and java 7 with  following
options

JAVA_OPTS=-Xms256m -Xmx2048m -XX:MaxPermSize=1024m -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode -XX:+ParallelRefProcEnabled
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ubuntu/OOM_HeapDump

our source code looks like following:
/ START */
int noOfSolrDocumentsInBatch = 0;
for(int i=0 ; i5000 ; i++) {
 SolrInputDocument solrInputDocument = getNextSolrInputDocument();
 server.add(solrInputDocument);
 noOfSolrDocumentsInBatch += 1;
 if(noOfSolrDocumentsInBatch == 10) {
 server.commit();
 noOfSolrDocumentsInBatch = 0;
 }
}
/ END */

the method getNextSolrInputDocument() generates a solr document with 100
fields (average). Around 50 of the fields are of text_general type.
Some of the test_general fields consist of approx 1000 words rest
consists of few words. Ouf of total fields there are around 35-40
multivalued fields (not of type text_general).
We are indexing all the fields but storing only 8 fields. Out of these 8
fields two are string type, five are long and one is boolean. So our index
size is only 394 MB. But the RAM occupied at time of OOM is around 2.5 GB.
Why the memory is so high even though the index size is small?
What is being stored in the memory? Our understanding is that after every
commit documents are flushed to the disk.So nothing should remain in RAM
after commit.

We are using the following settings:

server.commit() set waitForSearcher=true and waitForFlush=true
solrConfig.xml has following properties set:
directoryFactory = solr.MMapDirectoryFactory
maxWarmingSearchers = 1
text_general data type is being used as supplied in the schema.xml with the
solr setup.
maxIndexingThreads = 8(default)
autoCommitmaxTime15000/maxTimeopenSearcherfalse/openSearcher/autoCommit

We get Java heap Out Of Memory Error after commiting around 3990 solr
documents.Some of the snapshots of memory dump from profiler are uploaded
at following links.
http://s9.postimage.org/w7589t9e7/memorydump1.png
http://s7.postimage.org/p3abs6nuj/memorydump2.png

can somebody please suggest what should we do to minimize/optimize the
memory consumption in our case with the reasons?
also suggest what should be optimal values and reason for following
parameters of solrConfig.xml
useColdSearcher - true/false?
 maxwarmingsearchers- number
 spellcheck-on/off?
 omitNorms=true/false?
 omitTermFreqAndPositions?
 mergefactor? we are using default value 10
 java garbage collection tuning parameters ?


Additional information is needed.  What OS platform?  Is the OS 64-bit?  
Is Java 64-bit?  How much total RAM?  We'll need your solrconfig.xml 
file, in particular the query and indexConfig sections. Use your 
favorite paste site (pastie.org, pastebin.com for example) to link the 
solrconfig.xml file.


General thoughts without the above information:

You are reserving half of your java heap for the permanent generation.  
I have a solr installation where Java has a max heap of 8GB, about 5GB 
of that is currently committed - actually allocated at the OS level.  My 
perm gen space is 65908KB.  This server handles a total index size of 
nearly 70GB.  I doubt you need 1GB for your perm gen size.


A 2GB heap is fairly small in the Solr world.  If you are using a 32 bit 
java, that's the biggest heap you can create, so 64-bit on both Java and 
OS is the way to go.  You can reduce memory requirements a small amount 
by using Jetty instead of Tomcat, but the difference is probably not big 
enough to really matter.


For the questions you asked at the end, most of them are personal 
preference, but maxWarmingSearchers should normally be kept low.  I 
think I have a value of 2 in my config.  Here are the GC tuning 
parameters that I am currently testing.  I have been having problems 
with long GC pauses that I am trying to fix:


-Xms1024M
-Xmx8192M
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:NewRatio=3
-XX:MaxTenuringThreshold=8
-XX:+CMSParallelRemarkEnable

You should only use CMSIncrementalMode if you only have one or two 
processor cores.  My reading has suggested that when you have more, it 
is not beneficial.


So far my GC parameters seem to be working really well, but I need to do 
a full reindex which should force usage of the entire 8GB heap and push 
garbage collection to its limits.


I have a question of my own for someone familiar with the code. Does 
Solr extensively use weak references?  If so, ParallelRefProcEnabled 
might be a win.


Thanks,
Shawn

Re: SOLR 4.1 Out Of Memory error After commit of a few thousand Solr Docs

[Rahul Bishnoi]

Thanks for quick reply and addressing each point queried.

Additional asked information is mentioned below:

OS = Ubuntu 12.04 (64 bit)
Sun Java 7 (64 bit)
Total RAM = 8GB

SolrConfig.xml is available at http://pastebin.com/SEFxkw2R

Re: Help with sort on dynamic field and out of memory error

[Toke Eskildsen]

On Wed, 2012-11-28 at 03:25 +0100, Arun Rangarajan wrote:

[Sorting on 14M docs, 250 fields]

 From what I have read, I understand that restricting the number of distinct
 values on sortable Solr fields will bring down the fieldCache space. The
 values in these sortable fields can be any integer from 0 to 33000 and
 quite widely distributed. We have a few scaling solutions in mind, but what
 is the best way to handle this whole issue?

Since the number of documents exceeds to maximum value in your fields,
the lowest memory consumption for a fast implementation I can come up
with is #docs * #fields * maxbits/value: 14M * 250 * 16bit ~= 7GB.

So it's time to get creative and hack Solr.

First off, is the number of unique values/field significantly lower than
your maximum of 33000 for a non-trivial amount of your fields? If so,
they can be mapped to a contiguous range when storing the data (this
could be done dynamically when creating the field cache). If an average
field holds  1024 unique values, the total memory consumption would be
about 14M * 250 * 10bits ~= 4.4GB.

Secondly, if you normally use only a few fields for sorting, which I
suspect is the case, you could compress the values as a single block and
uncompress it when requested from field cache. Having a fixed size cache
of uncompressed values in the field cache should ensure that there is no
slowdown for most requests.

It is very hard to estimate the memory savings on this, but I would not
be surprised if you could reduce memory consumption to 1/10 of the worst
case 7GB, if the values are fairly uniform. Of course, if the values are
all over the place, this gains you nothing at all.

Regards,
Toke Eskildsen

Re: Help with sort on dynamic field and out of memory error

[Erick Erickson]

I sure don't see how this can work given the constraints. Just to hold the
values, assuming that each doc holds a value in 150 fields, you have 150 *
4 * 14,000,000 or 8.4G of memory required, and you just don't have that
much memory to play around with.

Sharding seems silly for 14M docs, but that might be what's necessary. Or
get hardware with lots of memory.

Or redefine the problem so you don't have to sort so many fields. Not quite
sure how do do that off the top of my head, but.

Best
Erick


On Tue, Nov 27, 2012 at 9:25 PM, Arun Rangarajan
arunrangara...@gmail.comwrote:

 We have a Solr 3.6 core that has about 250 TrieIntFields (declared using
 dynamicField). There are about 14M docs in our Solr index and many
 documents have some value in many of these fields. We have a need to sort
 on all of these 250 fields over a period of time.

 The issue we are facing is that the underlying lucene fieldCache gets
 filled up very quickly. We have a 4 GB box and the index size is 18 GB.
 After a sort on 40 or 45 of these dynamic fields, the memory consumption is
 about 90% (tomcat set up to get max heap size of 3.6 GB) and we start
 getting OutOfMemory errors.

 For now, we have a cron job running every minute restarting tomcat if the
 total memory consumed is more than 80%.

 We thought instead of sorting, if we did boosting it won't go to
 fieldCache. So instead of issuing a query like

 select?q=name:albasort=relevance_11 desc

 we tried

 select?q={!boost relevance_11}name:alba

 but unfortunately boosting also populates the field cache.

 From what I have read, I understand that restricting the number of distinct
 values on sortable Solr fields will bring down the fieldCache space. The
 values in these sortable fields can be any integer from 0 to 33000 and
 quite widely distributed. We have a few scaling solutions in mind, but what
 is the best way to handle this whole issue?

 thanks.

Re: Help with sort on dynamic field and out of memory error

[Arun Rangarajan]

Erick,

Thanks for your reply. So there is no easy way to get around this problem.

We have a way to rework the schema by keeping a single sort field. The
dynamic fields we have are like relevance_CLASSID. The current schema has a
unique key NODEID and a multi-valued field CLASSID - the relevance scores
are for these class Ids. If we instead keep one document per classId per
nodeId i.e. the new schema will have DOCID:CLASSID as unique key and store
some redundant information across documents with the same NODEID, then we
can sort on a single field relevance and do a filter query on classId.


On Tue, Nov 27, 2012 at 7:07 PM, Erick Erickson erickerick...@gmail.comwrote:

 I sure don't see how this can work given the constraints. Just to hold the
 values, assuming that each doc holds a value in 150 fields, you have 150 *
 4 * 14,000,000 or 8.4G of memory required, and you just don't have that
 much memory to play around with.

 Sharding seems silly for 14M docs, but that might be what's necessary. Or
 get hardware with lots of memory.

 Or redefine the problem so you don't have to sort so many fields. Not quite
 sure how do do that off the top of my head, but.

 Best
 Erick


 On Tue, Nov 27, 2012 at 9:25 PM, Arun Rangarajan
 arunrangara...@gmail.comwrote:

  We have a Solr 3.6 core that has about 250 TrieIntFields (declared using
  dynamicField). There are about 14M docs in our Solr index and many
  documents have some value in many of these fields. We have a need to sort
  on all of these 250 fields over a period of time.
 
  The issue we are facing is that the underlying lucene fieldCache gets
  filled up very quickly. We have a 4 GB box and the index size is 18 GB.
  After a sort on 40 or 45 of these dynamic fields, the memory consumption
 is
  about 90% (tomcat set up to get max heap size of 3.6 GB) and we start
  getting OutOfMemory errors.
 
  For now, we have a cron job running every minute restarting tomcat if the
  total memory consumed is more than 80%.
 
  We thought instead of sorting, if we did boosting it won't go to
  fieldCache. So instead of issuing a query like
 
  select?q=name:albasort=relevance_11 desc
 
  we tried
 
  select?q={!boost relevance_11}name:alba
 
  but unfortunately boosting also populates the field cache.
 
  From what I have read, I understand that restricting the number of
 distinct
  values on sortable Solr fields will bring down the fieldCache space. The
  values in these sortable fields can be any integer from 0 to 33000 and
  quite widely distributed. We have a few scaling solutions in mind, but
 what
  is the best way to handle this whole issue?
 
  thanks.
 

Re: ExtractingRequestHandler causes Out of Memory Error

[Jan Høydahl]

Hi,

If you like, you can open a JIRA issue on this and provide as much info as 
possible. Someone can then look into (potential) memory optimization of this 
part of the code.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com

28. sep. 2012 kl. 03:42 skrev Shigeki Kobayashi 
shigeki.kobayas...@g.softbank.co.jp:

 Hi Jan.
 
 Thank you very much for your advice.
 
 So I understand Solr needs more memory to parse the files.
 To parse a file of size x,  it needs double memory (2x). Then how much
 memory allocation should be taken to heap size? 8x? 16x?
 
 Regards,
 
 
 Shigeki
 
 2012/9/28 Jan Høydahl jan@cominvent.com
 
 Please try to increase -Xmx and see how much RAM you need for it to
 succeed.
 
 I believe it is simply a case where this particular file needs double
 memory (480Mb) to parse and you have only allocated 1Gb (which is not
 particularly much). Perhaps the code could be optimized to avoid the
 Arrays.copyOf() call..
 
 --
 Jan Høydahl, search solution architect
 Cominvent AS - www.cominvent.com
 Solr Training - www.solrtraining.com
 
 27. sep. 2012 kl. 11:22 skrev Shigeki Kobayashi 
 shigeki.kobayas...@g.softbank.co.jp:
 
 Hi guys,
 
 
 I use Manifold CF to crawl files in Windows file server and index them to
 Solr using Extracting Request Handler.
 Most of the documents are succesfully indexed but some are failed and Out
 of Memory Error occurs in Solr, so I need some advice.
 
 Those failed files are not so big and they are a csv file of 240MB and a
 text file of 170MB.
 
 Here is environment and machine spec:
 Solr 3.6 (also Solr4.0Beta)
 Tomcat 6.0
 CentOS 5.6
 java version 1.6.0_23
 HDD 60GB
 MEM 2GB
 JVM Heap: -Xmx1024m -Xms1024m
 
 I feel there is enough memory that Solr should be able to extract and
 index
 file content.
 
 Here is a Solr log below:
 --
 
 [solr.servlet.SolrDispatchFilter]-[http-8080-8]-:java.lang.OutOfMemoryError:
 Java heap space
   at java.util.Arrays.copyOf(Arrays.java:2882)
   at
 
 java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
   at
 java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
   at java.lang.StringBuilder.append(StringBuilder.java:189)
   at
 
 org.apache.solr.handler.extraction.SolrContentHandler.characters(SolrContentHandler.java:293)
   at
 
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
   at
 
 org.apache.tika.sax.SecureContentHandler.characters(SecureContentHandler.java:270)
   at
 
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
   at
 
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
   at
 
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
   at
 
 org.apache.tika.sax.SafeContentHandler.access$001(SafeContentHandler.java:46)
   at
 
 org.apache.tika.sax.SafeContentHandler$1.write(SafeContentHandler.java:82)
   at
 
 org.apache.tika.sax.SafeContentHandler.filter(SafeContentHandler.java:140)
   at
 
 org.apache.tika.sax.SafeContentHandler.characters(SafeContentHandler.java:287)
   at
 
 org.apache.tika.sax.XHTMLContentHandler.characters(XHTMLContentHandler.java:268)
   at org.apache.tika.parser.txt.TXTParser.parse(TXTParser.java:134)
   at
 org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
   at
 org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
   at
 org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
   at
 
 org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:227)
   at
 
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
   at
 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
   at
 
 org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:244)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
   at
 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
   at
 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
   at
 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at
 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at
 
 filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:122)
   at
 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at
 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at
 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
 
 -
 
 Anyone has any ideas

ExtractingRequestHandler causes Out of Memory Error

[Shigeki Kobayashi]

Hi guys,


I use Manifold CF to crawl files in Windows file server and index them to
Solr using Extracting Request Handler.
Most of the documents are succesfully indexed but some are failed and Out
of Memory Error occurs in Solr, so I need some advice.

Those failed files are not so big and they are a csv file of 240MB and a
text file of 170MB.

Here is environment and machine spec:
Solr 3.6 (also Solr4.0Beta)
Tomcat 6.0
CentOS 5.6
java version 1.6.0_23
HDD 60GB
MEM 2GB
JVM Heap: -Xmx1024m -Xms1024m

I feel there is enough memory that Solr should be able to extract and index
file content.

Here is a Solr log below:
--
[solr.servlet.SolrDispatchFilter]-[http-8080-8]-:java.lang.OutOfMemoryError:
Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
at java.lang.StringBuilder.append(StringBuilder.java:189)
at
org.apache.solr.handler.extraction.SolrContentHandler.characters(SolrContentHandler.java:293)
at
org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
at
org.apache.tika.sax.SecureContentHandler.characters(SecureContentHandler.java:270)
at
org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
at
org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
at
org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
at
org.apache.tika.sax.SafeContentHandler.access$001(SafeContentHandler.java:46)
at
org.apache.tika.sax.SafeContentHandler$1.write(SafeContentHandler.java:82)
at
org.apache.tika.sax.SafeContentHandler.filter(SafeContentHandler.java:140)
at
org.apache.tika.sax.SafeContentHandler.characters(SafeContentHandler.java:287)
at
org.apache.tika.sax.XHTMLContentHandler.characters(XHTMLContentHandler.java:268)
at org.apache.tika.parser.txt.TXTParser.parse(TXTParser.java:134)
at
org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
at
org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
at
org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
at
org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:227)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:244)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:122)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

-

Anyone has any ideas?

Regards,

Shigeki

Re: ExtractingRequestHandler causes Out of Memory Error

[Lance Norskog]

These are very large files and this is not enough memory. Do you upload these 
as files? 

If the CSV file is one document per line, you can split it up. Unix has a 
'split' command which does this very nicely. 

- Original Message -
| From: Shigeki Kobayashi shigeki.kobayas...@g.softbank.co.jp
| To: solr-user@lucene.apache.org
| Sent: Thursday, September 27, 2012 2:22:06 AM
| Subject: ExtractingRequestHandler causes Out of Memory Error
| 
| Hi guys,
| 
| 
| I use Manifold CF to crawl files in Windows file server and index
| them to
| Solr using Extracting Request Handler.
| Most of the documents are succesfully indexed but some are failed and
| Out
| of Memory Error occurs in Solr, so I need some advice.
| 
| Those failed files are not so big and they are a csv file of 240MB
| and a
| text file of 170MB.
| 
| Here is environment and machine spec:
| Solr 3.6 (also Solr4.0Beta)
| Tomcat 6.0
| CentOS 5.6
| java version 1.6.0_23
| HDD 60GB
| MEM 2GB
| JVM Heap: -Xmx1024m -Xms1024m
| 
| I feel there is enough memory that Solr should be able to extract and
| index
| file content.
| 
| Here is a Solr log below:
| --
| [solr.servlet.SolrDispatchFilter]-[http-8080-8]-:java.lang.OutOfMemoryError:
| Java heap space
| at java.util.Arrays.copyOf(Arrays.java:2882)
| at
| java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
| at
| java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
| at java.lang.StringBuilder.append(StringBuilder.java:189)
| at
| 
org.apache.solr.handler.extraction.SolrContentHandler.characters(SolrContentHandler.java:293)
| at
| 
org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
| at
| 
org.apache.tika.sax.SecureContentHandler.characters(SecureContentHandler.java:270)
| at
| 
org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
| at
| 
org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
| at
| 
org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
| at
| org.apache.tika.sax.SafeContentHandler.access$001(SafeContentHandler.java:46)
| at
| org.apache.tika.sax.SafeContentHandler$1.write(SafeContentHandler.java:82)
| at
| org.apache.tika.sax.SafeContentHandler.filter(SafeContentHandler.java:140)
| at
| org.apache.tika.sax.SafeContentHandler.characters(SafeContentHandler.java:287)
| at
| 
org.apache.tika.sax.XHTMLContentHandler.characters(XHTMLContentHandler.java:268)
| at
| org.apache.tika.parser.txt.TXTParser.parse(TXTParser.java:134)
| at
| org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
| at
| org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
| at
| org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
| at
| 
org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:227)
| at
| 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
| at
| 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
| at
| 
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:244)
| at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
| at
| 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
| at
| 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
| at
| 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
| at
| 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at
| 
filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:122)
| at
| 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
| at
| 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
| at
| 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
| 
| -
| 
| Anyone has any ideas?
| 
| Regards,
| 
| Shigeki
| 

Re: ExtractingRequestHandler causes Out of Memory Error

[Jan Høydahl]

Please try to increase -Xmx and see how much RAM you need for it to succeed.

I believe it is simply a case where this particular file needs double memory 
(480Mb) to parse and you have only allocated 1Gb (which is not particularly 
much). Perhaps the code could be optimized to avoid the Arrays.copyOf() call..

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com

27. sep. 2012 kl. 11:22 skrev Shigeki Kobayashi 
shigeki.kobayas...@g.softbank.co.jp:

 Hi guys,
 
 
 I use Manifold CF to crawl files in Windows file server and index them to
 Solr using Extracting Request Handler.
 Most of the documents are succesfully indexed but some are failed and Out
 of Memory Error occurs in Solr, so I need some advice.
 
 Those failed files are not so big and they are a csv file of 240MB and a
 text file of 170MB.
 
 Here is environment and machine spec:
 Solr 3.6 (also Solr4.0Beta)
 Tomcat 6.0
 CentOS 5.6
 java version 1.6.0_23
 HDD 60GB
 MEM 2GB
 JVM Heap: -Xmx1024m -Xms1024m
 
 I feel there is enough memory that Solr should be able to extract and index
 file content.
 
 Here is a Solr log below:
 --
 [solr.servlet.SolrDispatchFilter]-[http-8080-8]-:java.lang.OutOfMemoryError:
 Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at
 java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at
 java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
at java.lang.StringBuilder.append(StringBuilder.java:189)
at
 org.apache.solr.handler.extraction.SolrContentHandler.characters(SolrContentHandler.java:293)
at
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
at
 org.apache.tika.sax.SecureContentHandler.characters(SecureContentHandler.java:270)
at
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
at
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
at
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
at
 org.apache.tika.sax.SafeContentHandler.access$001(SafeContentHandler.java:46)
at
 org.apache.tika.sax.SafeContentHandler$1.write(SafeContentHandler.java:82)
at
 org.apache.tika.sax.SafeContentHandler.filter(SafeContentHandler.java:140)
at
 org.apache.tika.sax.SafeContentHandler.characters(SafeContentHandler.java:287)
at
 org.apache.tika.sax.XHTMLContentHandler.characters(XHTMLContentHandler.java:268)
at org.apache.tika.parser.txt.TXTParser.parse(TXTParser.java:134)
at
 org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
at
 org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
at
 org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
at
 org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:227)
at
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
at
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at
 org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:244)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
at
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
at
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
at
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
 filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:122)
at
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
 
 -
 
 Anyone has any ideas?
 
 Regards,
 
 Shigeki

Re: ExtractingRequestHandler causes Out of Memory Error

[Shigeki Kobayashi]

Hi Jan.

Thank you very much for your advice.

So I understand Solr needs more memory to parse the files.
To parse a file of size x,  it needs double memory (2x). Then how much
memory allocation should be taken to heap size? 8x? 16x?

Regards,


Shigeki

2012/9/28 Jan Høydahl jan@cominvent.com

 Please try to increase -Xmx and see how much RAM you need for it to
 succeed.

 I believe it is simply a case where this particular file needs double
 memory (480Mb) to parse and you have only allocated 1Gb (which is not
 particularly much). Perhaps the code could be optimized to avoid the
 Arrays.copyOf() call..

 --
 Jan Høydahl, search solution architect
 Cominvent AS - www.cominvent.com
 Solr Training - www.solrtraining.com

 27. sep. 2012 kl. 11:22 skrev Shigeki Kobayashi 
 shigeki.kobayas...@g.softbank.co.jp:

  Hi guys,
 
 
  I use Manifold CF to crawl files in Windows file server and index them to
  Solr using Extracting Request Handler.
  Most of the documents are succesfully indexed but some are failed and Out
  of Memory Error occurs in Solr, so I need some advice.
 
  Those failed files are not so big and they are a csv file of 240MB and a
  text file of 170MB.
 
  Here is environment and machine spec:
  Solr 3.6 (also Solr4.0Beta)
  Tomcat 6.0
  CentOS 5.6
  java version 1.6.0_23
  HDD 60GB
  MEM 2GB
  JVM Heap: -Xmx1024m -Xms1024m
 
  I feel there is enough memory that Solr should be able to extract and
 index
  file content.
 
  Here is a Solr log below:
  --
 
 [solr.servlet.SolrDispatchFilter]-[http-8080-8]-:java.lang.OutOfMemoryError:
  Java heap space
 at java.util.Arrays.copyOf(Arrays.java:2882)
 at
 
 java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
 at
  java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
 at java.lang.StringBuilder.append(StringBuilder.java:189)
 at
 
 org.apache.solr.handler.extraction.SolrContentHandler.characters(SolrContentHandler.java:293)
 at
 
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
 at
 
 org.apache.tika.sax.SecureContentHandler.characters(SecureContentHandler.java:270)
 at
 
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
 at
 
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
 at
 
 org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecorator.java:146)
 at
 
 org.apache.tika.sax.SafeContentHandler.access$001(SafeContentHandler.java:46)
 at
 
 org.apache.tika.sax.SafeContentHandler$1.write(SafeContentHandler.java:82)
 at
 
 org.apache.tika.sax.SafeContentHandler.filter(SafeContentHandler.java:140)
 at
 
 org.apache.tika.sax.SafeContentHandler.characters(SafeContentHandler.java:287)
 at
 
 org.apache.tika.sax.XHTMLContentHandler.characters(XHTMLContentHandler.java:268)
 at org.apache.tika.parser.txt.TXTParser.parse(TXTParser.java:134)
 at
  org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
 at
  org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
 at
  org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
 at
 
 org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:227)
 at
 
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
 at
 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
 at
 
 org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:244)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
 at
 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
 at
 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
 at
 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at
 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at
 
 filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:122)
 at
 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at
 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at
 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
 
  -
 
  Anyone has any ideas?
 
  Regards,
 
  Shigeki

Re: Solr Out of Memory Error

[Bing Li]

Dear Adam,

I also got the OutOfMemory exception. I changed the JAVA_OPTS in catalina.sh
as follows.

   ...
   if [ -z $LOGGING_MANAGER ]; then
 JAVA_OPTS=$JAVA_OPTS
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
   else
JAVA_OPTS=$JAVA_OPTS -server -Xms8096m -Xmx8096m
   fi
   ...

Is this change correct? After that, I still got the same exception. The
index is updated and searched frequently. I am trying to change the code to
avoid the frequent updates. I guess only changing JAVA_OPTS does not work.

Could you give me some help?

Thanks,
LB


On Wed, Jan 19, 2011 at 10:05 PM, Adam Estrada 
estrada.adam.gro...@gmail.com wrote:

 Is anyone familiar with the environment variable, JAVA_OPTS? I set
 mine to a much larger heap size and never had any of these issues
 again.

 JAVA_OPTS = -server -Xms4048m -Xmx4048m

 Adam

 On Wed, Jan 19, 2011 at 3:29 AM, Isan Fulia isan.fu...@germinait.com
 wrote:
  Hi all,
  By adding more servers do u mean sharding of index.And after sharding ,
 how
  my query performance will be affected .
  Will the query execution time increase.
 
  Thanks,
  Isan Fulia.
 
  On 19 January 2011 12:52, Grijesh pintu.grij...@gmail.com wrote:
 
 
  Hi Isan,
 
  It seems your index size 25GB si much more compared to you have total
 Ram
  size is 4GB.
  You have to do 2 things to avoid Out Of Memory Problem.
  1-Buy more Ram ,add at least 12 GB of more ram.
  2-Increase the Memory allocated to solr by setting XMX values.at least
 12
  GB
  allocate to solr.
 
  But if your all index will fit into the Cache memory it will give you
 the
  better result.
 
  Also add more servers to load balance as your QPS is high.
  Your 7 Laks data makes 25 GB of index its looking quite high.Try to
 lower
  the index size
  What are you indexing in your 25GB of index?
 
  -
  Thanx:
  Grijesh
  --
  View this message in context:
 
 http://lucene.472066.n3.nabble.com/Solr-Out-of-Memory-Error-tp2280037p2285779.html
  Sent from the Solr - User mailing list archive at Nabble.com.
 
 
 
 
  --
  Thanks  Regards,
  Isan Fulia.
 

Re: Solr Out of Memory Error

[Markus Jelsma]

Bing Li,

One should be conservative when setting Xmx. Also, just setting Xmx might not 
do the trick at all because the garbage collector might also be the issue 
here. Configure the JVM to output debug logs of the garbage collector and 
monitor the heap usage (especially the tenured generation) with a good tool 
like JConsole.

You might also want to take a look at your cache settings and autowarm 
parameters. In some scenario's with very frequent updates, a large corpus and 
a high load of heterogenous queries you might want to dump the documentCache 
and queryResultCache, the cache hitratio tends to be very low and the caches 
will just consume a lot of memory and CPU time.

One of my projects i finally decided to only use the filterCache. Using the 
other caches took too much RAM and CPU while running and had a lot of 
evictions and still a lot hitratio. I could, of course, make the caches a lot 
bigger and increase autowarming but that would take a lot of time before a 
cache is autowarmed and a very, very, large amount of RAM. I choose to rely on 
the OS-cache instead.

Cheers,

 Dear Adam,
 
 I also got the OutOfMemory exception. I changed the JAVA_OPTS in
 catalina.sh as follows.
 
...
if [ -z $LOGGING_MANAGER ]; then
  JAVA_OPTS=$JAVA_OPTS
 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
else
 JAVA_OPTS=$JAVA_OPTS -server -Xms8096m -Xmx8096m
fi
...
 
 Is this change correct? After that, I still got the same exception. The
 index is updated and searched frequently. I am trying to change the code to
 avoid the frequent updates. I guess only changing JAVA_OPTS does not work.
 
 Could you give me some help?
 
 Thanks,
 LB
 
 
 On Wed, Jan 19, 2011 at 10:05 PM, Adam Estrada 
 
 estrada.adam.gro...@gmail.com wrote:
  Is anyone familiar with the environment variable, JAVA_OPTS? I set
  mine to a much larger heap size and never had any of these issues
  again.
  
  JAVA_OPTS = -server -Xms4048m -Xmx4048m
  
  Adam
  
  On Wed, Jan 19, 2011 at 3:29 AM, Isan Fulia isan.fu...@germinait.com
  
  wrote:
   Hi all,
   By adding more servers do u mean sharding of index.And after sharding ,
  
  how
  
   my query performance will be affected .
   Will the query execution time increase.
   
   Thanks,
   Isan Fulia.
   
   On 19 January 2011 12:52, Grijesh pintu.grij...@gmail.com wrote:
   Hi Isan,
   
   It seems your index size 25GB si much more compared to you have total
  
  Ram
  
   size is 4GB.
   You have to do 2 things to avoid Out Of Memory Problem.
   1-Buy more Ram ,add at least 12 GB of more ram.
   2-Increase the Memory allocated to solr by setting XMX values.at least
  
  12
  
   GB
   allocate to solr.
   
   But if your all index will fit into the Cache memory it will give you
  
  the
  
   better result.
   
   Also add more servers to load balance as your QPS is high.
   Your 7 Laks data makes 25 GB of index its looking quite high.Try to
  
  lower
  
   the index size
   What are you indexing in your 25GB of index?
   
   -
   Thanx:
   Grijesh
   --
  
   View this message in context:
  http://lucene.472066.n3.nabble.com/Solr-Out-of-Memory-Error-tp2280037p228
  5779.html
  
   Sent from the Solr - User mailing list archive at Nabble.com.
   
   --
   Thanks  Regards,
   Isan Fulia.

Re: Solr Out of Memory Error

[Markus Jelsma]

I should also add that reducing the caches and autowarm sizes (or not using 
them at all) drastically reduces memory consumption when a new searcher is 
being prepares after a commit. The memory usage will spike at these events. 
Again, use a monitoring tool to get more information on your specific scenario.

 Bing Li,
 
 One should be conservative when setting Xmx. Also, just setting Xmx might
 not do the trick at all because the garbage collector might also be the
 issue here. Configure the JVM to output debug logs of the garbage
 collector and monitor the heap usage (especially the tenured generation)
 with a good tool like JConsole.
 
 You might also want to take a look at your cache settings and autowarm
 parameters. In some scenario's with very frequent updates, a large corpus
 and a high load of heterogenous queries you might want to dump the
 documentCache and queryResultCache, the cache hitratio tends to be very
 low and the caches will just consume a lot of memory and CPU time.
 
 One of my projects i finally decided to only use the filterCache. Using the
 other caches took too much RAM and CPU while running and had a lot of
 evictions and still a lot hitratio. I could, of course, make the caches a
 lot bigger and increase autowarming but that would take a lot of time
 before a cache is autowarmed and a very, very, large amount of RAM. I
 choose to rely on the OS-cache instead.
 
 Cheers,
 
  Dear Adam,
  
  I also got the OutOfMemory exception. I changed the JAVA_OPTS in
  catalina.sh as follows.
  
 ...
 if [ -z $LOGGING_MANAGER ]; then
 
   JAVA_OPTS=$JAVA_OPTS
  
  -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
  
 else
 
  JAVA_OPTS=$JAVA_OPTS -server -Xms8096m -Xmx8096m
 
 fi
 ...
  
  Is this change correct? After that, I still got the same exception. The
  index is updated and searched frequently. I am trying to change the code
  to avoid the frequent updates. I guess only changing JAVA_OPTS does not
  work.
  
  Could you give me some help?
  
  Thanks,
  LB
  
  
  On Wed, Jan 19, 2011 at 10:05 PM, Adam Estrada 
  
  estrada.adam.gro...@gmail.com wrote:
   Is anyone familiar with the environment variable, JAVA_OPTS? I set
   mine to a much larger heap size and never had any of these issues
   again.
   
   JAVA_OPTS = -server -Xms4048m -Xmx4048m
   
   Adam
   
   On Wed, Jan 19, 2011 at 3:29 AM, Isan Fulia isan.fu...@germinait.com
   
   wrote:
Hi all,
By adding more servers do u mean sharding of index.And after sharding
,
   
   how
   
my query performance will be affected .
Will the query execution time increase.

Thanks,
Isan Fulia.

On 19 January 2011 12:52, Grijesh pintu.grij...@gmail.com wrote:
Hi Isan,

It seems your index size 25GB si much more compared to you have
total
   
   Ram
   
size is 4GB.
You have to do 2 things to avoid Out Of Memory Problem.
1-Buy more Ram ,add at least 12 GB of more ram.
2-Increase the Memory allocated to solr by setting XMX values.at
least
   
   12
   
GB
allocate to solr.

But if your all index will fit into the Cache memory it will give
you
   
   the
   
better result.

Also add more servers to load balance as your QPS is high.
Your 7 Laks data makes 25 GB of index its looking quite high.Try to
   
   lower
   
the index size
What are you indexing in your 25GB of index?

-
Thanx:
Grijesh
--
   
View this message in context:
   http://lucene.472066.n3.nabble.com/Solr-Out-of-Memory-Error-tp2280037p2
   28 5779.html
   
Sent from the Solr - User mailing list archive at Nabble.com.

--
Thanks  Regards,
Isan Fulia.

Re: Solr Out of Memory Error

[Isan Fulia]

Hi all,
By adding more servers do u mean sharding of index.And after sharding , how
my query performance will be affected .
Will the query execution time increase.

Thanks,
Isan Fulia.

On 19 January 2011 12:52, Grijesh pintu.grij...@gmail.com wrote:


 Hi Isan,

 It seems your index size 25GB si much more compared to you have total Ram
 size is 4GB.
 You have to do 2 things to avoid Out Of Memory Problem.
 1-Buy more Ram ,add at least 12 GB of more ram.
 2-Increase the Memory allocated to solr by setting XMX values.at least 12
 GB
 allocate to solr.

 But if your all index will fit into the Cache memory it will give you the
 better result.

 Also add more servers to load balance as your QPS is high.
 Your 7 Laks data makes 25 GB of index its looking quite high.Try to lower
 the index size
 What are you indexing in your 25GB of index?

 -
 Thanx:
 Grijesh
 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Solr-Out-of-Memory-Error-tp2280037p2285779.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
Thanks  Regards,
Isan Fulia.

Re: Solr Out of Memory Error

[Adam Estrada]

Is anyone familiar with the environment variable, JAVA_OPTS? I set
mine to a much larger heap size and never had any of these issues
again.

JAVA_OPTS = -server -Xms4048m -Xmx4048m

Adam

On Wed, Jan 19, 2011 at 3:29 AM, Isan Fulia isan.fu...@germinait.com wrote:
 Hi all,
 By adding more servers do u mean sharding of index.And after sharding , how
 my query performance will be affected .
 Will the query execution time increase.

 Thanks,
 Isan Fulia.

 On 19 January 2011 12:52, Grijesh pintu.grij...@gmail.com wrote:


 Hi Isan,

 It seems your index size 25GB si much more compared to you have total Ram
 size is 4GB.
 You have to do 2 things to avoid Out Of Memory Problem.
 1-Buy more Ram ,add at least 12 GB of more ram.
 2-Increase the Memory allocated to solr by setting XMX values.at least 12
 GB
 allocate to solr.

 But if your all index will fit into the Cache memory it will give you the
 better result.

 Also add more servers to load balance as your QPS is high.
 Your 7 Laks data makes 25 GB of index its looking quite high.Try to lower
 the index size
 What are you indexing in your 25GB of index?

 -
 Thanx:
 Grijesh
 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Solr-Out-of-Memory-Error-tp2280037p2285779.html
 Sent from the Solr - User mailing list archive at Nabble.com.




 --
 Thanks  Regards,
 Isan Fulia.

Re: Solr Out of Memory Error

[Grijesh]

By adding more server means add more searchers (slaves) on Load balancer not
talking about sharding.

Sharding is required when your index size will increase the size of about
50GB.

-
Thanx:
Grijesh
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Out-of-Memory-Error-tp2280037p2292944.html
Sent from the Solr - User mailing list archive at Nabble.com.

Solr Out of Memory Error

[Isan Fulia]

Hi all,
I got the following error on solr with m/c configuration 4GB RAM   and Intel
Dual Core Processor.Can you please  help me out.

java.lang.OutOfMemoryError: Java heap space
2011-01-18 18:00:27.655:WARN::Committed before 500 OutOfMemoryError likely
caused by the Sun VM Bug described in
https://issues.apache.org/jira/browse/LUCENE-1566; try calling
FSDirectory.setReadChunkSize with a a value smaller than the current chunk
size (2147483647)||java.lang.
OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug described
in https://issues.apache.org/jira/browse/LUCENE-1566; try calling
FSDirectory.setReadChunkSize with a a value smaller than the current chunk
size (2147483647)|?at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:161)|?at
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:139)|?at
org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:285)|?at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:160)|?at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)|?at
org.apache.lucene.store.DataInput.readVInt(DataInput.java:86)|?at
org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:201)|?at
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:828)|?at
org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:579)|?at
org.apache.lucene.index.IndexReader.document(IndexReader.java:755)|?at
org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:454)|?at
org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:431)|?at
org.apache.solr.response.BinaryResponseWriter$Resolver.writeDocList(BinaryResponseWriter.java:120)|?at
org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:86)|?at
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:143)|?at
org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:133)|?at
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:221)|?at
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:138)|?at
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:87)|?at
org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:46)|?at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:321)|?at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:253)|?at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)|?at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)|?at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)|?at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)|?at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)|?at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)|?at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)|?at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)|?at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)|?at
org.mortbay.jetty.Server.handle(Server.java:326)|?at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)|?at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)|?at
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)|?at
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)|?at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)|?at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)|?at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)|Caused
by: java.lang.OutOfMemoryError: GC overhead limit exceeded|
2011-01-18 18:00:27.656:WARN::/solr/ProdContentIndex/select
java.lang.IllegalStateException: Committed
at org.mortbay.jetty.Response.resetBuffer(Response.java:1024)
at org.mortbay.jetty.Response.sendError(Response.java:240)
at
org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:361)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:271)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at

Re: Solr Out of Memory Error

[Markus Jelsma]

Hi

I haven't seen one like this before. Please provide JVM settings and Solr 
version.

Cheers

On Tuesday 18 January 2011 15:08:35 Isan Fulia wrote:
 Hi all,
 I got the following error on solr with m/c configuration 4GB RAM   and
 Intel Dual Core Processor.Can you please  help me out.
 
 java.lang.OutOfMemoryError: Java heap space
 2011-01-18 18:00:27.655:WARN::Committed before 500 OutOfMemoryError likely
 caused by the Sun VM Bug described in
 https://issues.apache.org/jira/browse/LUCENE-1566; try calling
 FSDirectory.setReadChunkSize with a a value smaller than the current chunk
 size (2147483647)||java.lang.
 OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug
 described in https://issues.apache.org/jira/browse/LUCENE-1566; try
 calling
 FSDirectory.setReadChunkSize with a a value smaller than the current chunk
 size (2147483647)|?at
 org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDi
 rectory.java:161)|?at
 org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.ja
 va:139)|?at
 org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(Compo
 undFileReader.java:285)|?at
 org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
 160)|?at
 org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.jav
 a:39)|?at org.apache.lucene.store.DataInput.readVInt(DataInput.java:86)|?at
 org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:201)|?at
 org.apache.lucene.index.SegmentReader.document(SegmentReader.java:828)|?at
 org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:579)|
 ?at org.apache.lucene.index.IndexReader.document(IndexReader.java:755)|?at
 org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:454)|
 ?at
 org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:431)|?
 at
 org.apache.solr.response.BinaryResponseWriter$Resolver.writeDocList(Binary
 ResponseWriter.java:120)|?at
 org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryRespo
 nseWriter.java:86)|?at
 org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:143)|?
 at
 org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:
 133)|?at
 org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:
 221)|?at
 org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:138)|?
 at
 org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:87)|?at
 org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.j
 ava:46)|?at
 org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilte
 r.java:321)|?at
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.jav
 a:253)|?at
 org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandl
 er.java:1157)|?at
 org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)|?
 at
 org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216
 )|?at
 org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)|?
 at
 org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)|?
 at
 org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)|?at
 org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCo
 llection.java:230)|?at
 org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:
 114)|?at
 org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)|?
 at org.mortbay.jetty.Server.handle(Server.java:326)|?at
 org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)|?at
 org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java
 :938)|?at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)|?at
 org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)|?at
 org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)|?at
 org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:2
 28)|?at
 org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:5
 82)|Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded|
 2011-01-18 18:00:27.656:WARN::/solr/ProdContentIndex/select
 java.lang.IllegalStateException: Committed
 at org.mortbay.jetty.Response.resetBuffer(Response.java:1024)
 at org.mortbay.jetty.Response.sendError(Response.java:240)
 at
 org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.jav
 a:361) at
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java
 :271) at
 org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle
 r.java:1157) at
 org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
 at
 org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
 at
 org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
 at
 

Re: Solr Out of Memory Error

[Isan Fulia]

Hi markus,
We dont have any XMX memory settings as such .Our java version is 1.6.0_19
and solr version is 1.4 developer version. Can u plz help us out.

Thanks,
Isan.

On 18 January 2011 19:54, Markus Jelsma markus.jel...@openindex.io wrote:

 Hi

 I haven't seen one like this before. Please provide JVM settings and Solr
 version.

 Cheers

 On Tuesday 18 January 2011 15:08:35 Isan Fulia wrote:
  Hi all,
  I got the following error on solr with m/c configuration 4GB RAM   and
  Intel Dual Core Processor.Can you please  help me out.
 
  java.lang.OutOfMemoryError: Java heap space
  2011-01-18 18:00:27.655:WARN::Committed before 500 OutOfMemoryError
 likely
  caused by the Sun VM Bug described in
  https://issues.apache.org/jira/browse/LUCENE-1566; try calling
  FSDirectory.setReadChunkSize with a a value smaller than the current
 chunk
  size (2147483647)||java.lang.
  OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug
  described in https://issues.apache.org/jira/browse/LUCENE-1566; try
  calling
  FSDirectory.setReadChunkSize with a a value smaller than the current
 chunk
  size (2147483647)|?at
 
 org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDi
  rectory.java:161)|?at
 
 org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.ja
  va:139)|?at
 
 org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(Compo
  undFileReader.java:285)|?at
 
 org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
  160)|?at
 
 org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.jav
  a:39)|?at
 org.apache.lucene.store.DataInput.readVInt(DataInput.java:86)|?at
  org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:201)|?at
 
 org.apache.lucene.index.SegmentReader.document(SegmentReader.java:828)|?at
 
 org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:579)|
  ?at
 org.apache.lucene.index.IndexReader.document(IndexReader.java:755)|?at
 
 org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:454)|
  ?at
 
 org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:431)|?
  at
 
 org.apache.solr.response.BinaryResponseWriter$Resolver.writeDocList(Binary
  ResponseWriter.java:120)|?at
 
 org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryRespo
  nseWriter.java:86)|?at
 
 org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:143)|?
  at
 
 org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:
  133)|?at
 
 org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:
  221)|?at
 
 org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:138)|?
  at
 
 org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:87)|?at
 
 org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.j
  ava:46)|?at
 
 org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilte
  r.java:321)|?at
 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.jav
  a:253)|?at
 
 org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandl
  er.java:1157)|?at
 
 org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)|?
  at
 
 org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216
  )|?at
 
 org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)|?
  at
 
 org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)|?
  at
  org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)|?at
 
 org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCo
  llection.java:230)|?at
 
 org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:
  114)|?at
 
 org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)|?
  at org.mortbay.jetty.Server.handle(Server.java:326)|?at
 
 org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)|?at
 
 org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java
  :938)|?at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)|?at
  org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)|?at
  org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)|?at
 
 org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:2
  28)|?at
 
 org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:5
  82)|Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded|
  2011-01-18 18:00:27.656:WARN::/solr/ProdContentIndex/select
  java.lang.IllegalStateException: Committed
  at org.mortbay.jetty.Response.resetBuffer(Response.java:1024)
  at org.mortbay.jetty.Response.sendError(Response.java:240)
  at
 
 org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.jav
  a:361) at
 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java
  :271) at
 
 

Re: Solr Out of Memory Error

[Zach]

I've run into that GC overhead limit exceeded messagebefore.  Try: 
-XX:-UseGCOverheadLimit to avoid that one; but it basically means you're 
running 
low on memory and turning off GCOverheadLimit may just lead to a real heap 
space 
a few seconds later.

Also, if you have free memory on the host, try using the -Xmx command line 
parameter to raise the max amount of memory solr can use. (-Xmx2g for example)



- Original Message 
From: Isan Fulia isan.fu...@germinait.com
To: markus.jel...@openindex.io
Cc: solr-user@lucene.apache.org
Sent: Tue, January 18, 2011 9:04:31 PM
Subject: Re: Solr Out of Memory Error

Hi markus,
We dont have any XMX memory settings as such .Our java version is 1.6.0_19
and solr version is 1.4 developer version. Can u plz help us out.

Thanks,
Isan.

On 18 January 2011 19:54, Markus Jelsma markus.jel...@openindex.io wrote:

 Hi

 I haven't seen one like this before. Please provide JVM settings and Solr
 version.

 Cheers

 On Tuesday 18 January 2011 15:08:35 Isan Fulia wrote:
  Hi all,
  I got the following error on solr with m/c configuration 4GB RAM   and
  Intel Dual Core Processor.Can you please  help me out.
 
  java.lang.OutOfMemoryError: Java heap space
  2011-01-18 18:00:27.655:WARN::Committed before 500 OutOfMemoryError
 likely
  caused by the Sun VM Bug described in
  https://issues.apache.org/jira/browse/LUCENE-1566; try calling
  FSDirectory.setReadChunkSize with a a value smaller than the current
 chunk
  size (2147483647)||java.lang.
  OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug
  described in https://issues.apache.org/jira/browse/LUCENE-1566; try
  calling
  FSDirectory.setReadChunkSize with a a value smaller than the current
 chunk
  size (2147483647)|?at
 
 org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDi
  rectory.java:161)|?at
 
 org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.ja
  va:139)|?at
 
 org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(Compo
  undFileReader.java:285)|?at
 
 org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
  160)|?at
 
 org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.jav
  a:39)|?at
 org.apache.lucene.store.DataInput.readVInt(DataInput.java:86)|?at
  org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:201)|?at
 
 org.apache.lucene.index.SegmentReader.document(SegmentReader.java:828)|?at
 
 org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:579)|
  ?at
 org.apache.lucene.index.IndexReader.document(IndexReader.java:755)|?at
 
 org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:454)|
  ?at
 
 org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:431)|?
  at
 
 org.apache.solr.response.BinaryResponseWriter$Resolver.writeDocList(Binary
  ResponseWriter.java:120)|?at
 
 org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryRespo
  nseWriter.java:86)|?at
 
 org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:143)|?
  at
 
 org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:
  133)|?at
 
 org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:
  221)|?at
 
 org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:138)|?
  at
 
 org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:87)|?at
 
 org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.j
  ava:46)|?at
 
 org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilte
  r.java:321)|?at
 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.jav
  a:253)|?at
 
 org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandl
  er.java:1157)|?at
 
 org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)|?
  at
 
 org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216
  )|?at
 
 org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)|?
  at
 
 org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)|?
  at
  org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)|?at
 
 org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCo
  llection.java:230)|?at
 
 org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:
  114)|?at
 
 org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)|?
  at org.mortbay.jetty.Server.handle(Server.java:326)|?at
 
 org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)|?at
 
 org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java
  :938)|?at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)|?at
  org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)|?at
  org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)|?at
 
 org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:2
  28

Re: Solr Out of Memory Error

[Grijesh]

On which server [master/slave] Out of Memory ocuur
What is your index in size[GB]?
How many documents you have?
What is query per second?
How you are indexing?
What is you ramBufferSize?

-
Thanx:
Grijesh
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Out-of-Memory-Error-tp2280037p2285392.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Out of Memory Error

[Isan Fulia]

Hi Grijesh,all,
We are having only single master and are using multicore environment with
size of various indexes  as 675MB ,516 MB , 3GB , 25GB.
Number of documents with 3GB index are roughly around 14 lakhs
and with 25 GB are roughly around 7 lakh
Queries are fired very frequently.
ramBufferSize and indexing  are all default settings.

Thanks ,
Isan.


On 19 January 2011 10:41, Grijesh pintu.grij...@gmail.com wrote:


 On which server [master/slave] Out of Memory ocuur
 What is your index in size[GB]?
 How many documents you have?
 What is query per second?
 How you are indexing?
 What is you ramBufferSize?

 -
 Thanx:
 Grijesh
 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Solr-Out-of-Memory-Error-tp2280037p2285392.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
Thanks  Regards,
Isan Fulia.

Re: Solr Out of Memory Error

[Grijesh]

Hi Isan,

It seems your index size 25GB si much more compared to you have total Ram
size is 4GB.
You have to do 2 things to avoid Out Of Memory Problem.
1-Buy more Ram ,add at least 12 GB of more ram.
2-Increase the Memory allocated to solr by setting XMX values.at least 12 GB
allocate to solr.

But if your all index will fit into the Cache memory it will give you the
better result.

Also add more servers to load balance as your QPS is high.
Your 7 Laks data makes 25 GB of index its looking quite high.Try to lower
the index size
What are you indexing in your 25GB of index?

-
Thanx:
Grijesh
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Out-of-Memory-Error-tp2280037p2285779.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Out of memory error

[Erick Erickson]

Have you seen this page? http://wiki.apache.org/solr/DataImportHandlerFaq
http://wiki.apache.org/solr/DataImportHandlerFaqSee especially batchsize,
but it looks like you're already on to that.

Do you have any idea how big the records are in the database? You might
try adjusting the rambuffersize down, what is it at now?

In general, what are our Solr commit options?

Does anything get to Solr or is the OOM when the SQL is executed?
The first question to answer is whether you index anything at all...

There's a little-know DIH debug page you can access at:
.../solr/admin/dataimport.jsp that might help, and progress can be monitored
at:
.../solr/dataimport

DIH can be interesting, you get finer control with SolrJ and a direct
JDBC connection. If you don't get anywhere with DIH.

Scattergun response, but things to try...

Best
Erick

On Tue, Dec 7, 2010 at 12:03 AM, sivaprasad sivaprasa...@echidnainc.comwrote:


 Hi,

 When i am trying to import the data using DIH, iam getting Out of memory
 error.The below are the configurations which i have.

 Database:Mysql
 Os:windows
 No Of documents:15525532
 In Db-config.xml i made batch size as -1

 The solr server is running on Linux machine with tomcat.
 i set tomcat arguments as ./startup.sh -Xms1024M -Xmx2048M

 Can anybody has idea, where the things are going wrong?

 Regards,
 JS


 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Out-of-memory-error-tp2031761p2031761.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Out of memory error

[Fuad Efendi]

Related: SOLR-846

Sent on the TELUS Mobility network with BlackBerry

-Original Message-
From: Erick Erickson erickerick...@gmail.com
Date: Tue, 7 Dec 2010 08:11:41 
To: solr-user@lucene.apache.org
Reply-To: solr-user@lucene.apache.org
Subject: Re: Out of memory error

Have you seen this page? http://wiki.apache.org/solr/DataImportHandlerFaq
http://wiki.apache.org/solr/DataImportHandlerFaqSee especially batchsize,
but it looks like you're already on to that.

Do you have any idea how big the records are in the database? You might
try adjusting the rambuffersize down, what is it at now?

In general, what are our Solr commit options?

Does anything get to Solr or is the OOM when the SQL is executed?
The first question to answer is whether you index anything at all...

There's a little-know DIH debug page you can access at:
.../solr/admin/dataimport.jsp that might help, and progress can be monitored
at:
.../solr/dataimport

DIH can be interesting, you get finer control with SolrJ and a direct
JDBC connection. If you don't get anywhere with DIH.

Scattergun response, but things to try...

Best
Erick

On Tue, Dec 7, 2010 at 12:03 AM, sivaprasad sivaprasa...@echidnainc.comwrote:


 Hi,

 When i am trying to import the data using DIH, iam getting Out of memory
 error.The below are the configurations which i have.

 Database:Mysql
 Os:windows
 No Of documents:15525532
 In Db-config.xml i made batch size as -1

 The solr server is running on Linux machine with tomcat.
 i set tomcat arguments as ./startup.sh -Xms1024M -Xmx2048M

 Can anybody has idea, where the things are going wrong?

 Regards,
 JS


 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Out-of-memory-error-tp2031761p2031761.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Out of memory error

[sivaprasad]

Hi,

When i am trying to import the data using DIH, iam getting Out of memory
error.The below are the configurations which i have.

Database:Mysql
Os:windows
No Of documents:15525532
In Db-config.xml i made batch size as -1

The solr server is running on Linux machine with tomcat.
i set tomcat arguments as ./startup.sh -Xms1024M -Xmx2048M

Can anybody has idea, where the things are going wrong?

Regards,
JS


-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/Out-of-memory-error-tp2031761p2031761.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Out of memory error

[Fuad Efendi]

Batch size -1??? Strange but could be a problem. 

Note also you can't provide parameters to default startup.sh command; you 
should modify setenv.sh instead

--Original Message--
From: sivaprasad
To: solr-user@lucene.apache.org
ReplyTo: solr-user@lucene.apache.org
Subject: Out of memory error
Sent: Dec 7, 2010 12:03 AM


Hi,

When i am trying to import the data using DIH, iam getting Out of memory
error.The below are the configurations which i have.

Database:Mysql
Os:windows
No Of documents:15525532
In Db-config.xml i made batch size as -1

The solr server is running on Linux machine with tomcat.
i set tomcat arguments as ./startup.sh -Xms1024M -Xmx2048M

Can anybody has idea, where the things are going wrong?

Regards,
JS


-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/Out-of-memory-error-tp2031761p2031761.html
Sent from the Solr - User mailing list archive at Nabble.com.


Sent on the TELUS Mobility network with BlackBerry

Re: Need help with field collapsing and out of memory error

[Moazzam Khan]

Oh, I don't know if this matters but I store text fields in Solr but I
never get them from the index (I only get the ID field from the index
and everything else is pulled from DB cache). I store all the fields
just in case I need to debug search queries, etc and want to see the
data.

Regards,

Moazzam



On Wed, Sep 1, 2010 at 5:13 PM, Moazzam Khan moazz...@gmail.com wrote:
 Hi,


 If this is how you configure the field collapsing cache, then I don't
 have it setup:


  fieldCollapsing

        fieldCollapseCache
      class=solr.FastLRUCache
      size=512
      initialSize=512
      autowarmCount=128/

  /fieldCollapsing


 I didnt add that part to solrconfig.xml.

 The way I setup field collapsing is I added this tag:

 searchComponent name=collapse
 class=org.apache.solr.handler.component.CollapseComponent /

 Then I modified the default request handler (for standard queries) with this:

  requestHandler name=standard class=solr.SearchHandler default=true
    !-- default values for query parameters --
     lst name=defaults
       str name=echoParamsexplicit/str

     /lst
     arr name=components
        strcollapse/str
        strfacet/str
        strhighlight/str
        strdebug/str
     /arr
  /requestHandler




 On Wed, Sep 1, 2010 at 4:11 PM, Jean-Sebastien Vachon
 js.vac...@videotron.ca wrote:
 can you tell us what are your current settings regarding the 
 fieldCollapseCache?

 I had similar issues with field collapsing and I found out that this cache 
 was responsible for
 most of the OOM exceptions.

 Reduce or even remove this cache from your configuration and it should help.


 On 2010-09-01, at 1:10 PM, Moazzam Khan wrote:

 Hi guys,

 I have about 20k documents in the Solr index (and there's a lot of
 text in each of them). I have field collapsing enabled on a specific
 field (AdvisorID).

 The thing is if I have field collapsing enabled in the search request
 I don't get correct count for the total number of records that
 matched. It always says that the number of rows I asked to get back
 is the number of total records it found.

 And, when I run a query with search criteria *:* (to get the number of
 total advisors in the index) solr runs of out memory and gives me an
 error saying

 SEVERE: java.lang.OutOfMemoryError: Java heap space
        at java.nio.CharBuffer.wrap(CharBuffer.java:350)
        at java.nio.CharBuffer.wrap(CharBuffer.java:373)
        at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:138)
        at java.lang.StringCoding.decode(StringCoding.java:173)


 This is going to be a huge problem later on when we index 50k
 documents later on.

 These are the options I am running Solr with :

 java  -Xms2048M -Xmx2048M -XX:+UseConcMarkSweepGC -XX:PermSize=1024m
 MaxPermSize=1024m    -jar  start.jar


 Is there any way I can get the counts and not run out of memory?

 Thanks in advance,
 Moazzam

Need help with field collapsing and out of memory error

[Moazzam Khan]

Hi guys,

I have about 20k documents in the Solr index (and there's a lot of
text in each of them). I have field collapsing enabled on a specific
field (AdvisorID).

The thing is if I have field collapsing enabled in the search request
I don't get correct count for the total number of records that
matched. It always says that the number of rows I asked to get back
is the number of total records it found.

And, when I run a query with search criteria *:* (to get the number of
total advisors in the index) solr runs of out memory and gives me an
error saying

SEVERE: java.lang.OutOfMemoryError: Java heap space
at java.nio.CharBuffer.wrap(CharBuffer.java:350)
at java.nio.CharBuffer.wrap(CharBuffer.java:373)
at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:138)
at java.lang.StringCoding.decode(StringCoding.java:173)


This is going to be a huge problem later on when we index 50k
documents later on.

These are the options I am running Solr with :

java  -Xms2048M -Xmx2048M -XX:+UseConcMarkSweepGC -XX:PermSize=1024m
MaxPermSize=1024m-jar  start.jar


Is there any way I can get the counts and not run out of memory?

Thanks in advance,
Moazzam

Re: Need help with field collapsing and out of memory error

[Jean-Sebastien Vachon]

can you tell us what are your current settings regarding the fieldCollapseCache?

I had similar issues with field collapsing and I found out that this cache was 
responsible for 
most of the OOM exceptions.

Reduce or even remove this cache from your configuration and it should help.


On 2010-09-01, at 1:10 PM, Moazzam Khan wrote:

 Hi guys,
 
 I have about 20k documents in the Solr index (and there's a lot of
 text in each of them). I have field collapsing enabled on a specific
 field (AdvisorID).
 
 The thing is if I have field collapsing enabled in the search request
 I don't get correct count for the total number of records that
 matched. It always says that the number of rows I asked to get back
 is the number of total records it found.
 
 And, when I run a query with search criteria *:* (to get the number of
 total advisors in the index) solr runs of out memory and gives me an
 error saying
 
 SEVERE: java.lang.OutOfMemoryError: Java heap space
at java.nio.CharBuffer.wrap(CharBuffer.java:350)
at java.nio.CharBuffer.wrap(CharBuffer.java:373)
at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:138)
at java.lang.StringCoding.decode(StringCoding.java:173)
 
 
 This is going to be a huge problem later on when we index 50k
 documents later on.
 
 These are the options I am running Solr with :
 
 java  -Xms2048M -Xmx2048M -XX:+UseConcMarkSweepGC -XX:PermSize=1024m
 MaxPermSize=1024m-jar  start.jar
 
 
 Is there any way I can get the counts and not run out of memory?
 
 Thanks in advance,
 Moazzam

Re: Need help with field collapsing and out of memory error

[Moazzam Khan]

Hi,


If this is how you configure the field collapsing cache, then I don't
have it setup:


 fieldCollapsing

fieldCollapseCache
  class=solr.FastLRUCache
  size=512
  initialSize=512
  autowarmCount=128/

  /fieldCollapsing


I didnt add that part to solrconfig.xml.

The way I setup field collapsing is I added this tag:

searchComponent name=collapse
class=org.apache.solr.handler.component.CollapseComponent /

Then I modified the default request handler (for standard queries) with this:

 requestHandler name=standard class=solr.SearchHandler default=true
!-- default values for query parameters --
 lst name=defaults
   str name=echoParamsexplicit/str

 /lst
 arr name=components
strcollapse/str
strfacet/str
strhighlight/str
strdebug/str
 /arr
  /requestHandler




On Wed, Sep 1, 2010 at 4:11 PM, Jean-Sebastien Vachon
js.vac...@videotron.ca wrote:
 can you tell us what are your current settings regarding the 
 fieldCollapseCache?

 I had similar issues with field collapsing and I found out that this cache 
 was responsible for
 most of the OOM exceptions.

 Reduce or even remove this cache from your configuration and it should help.


 On 2010-09-01, at 1:10 PM, Moazzam Khan wrote:

 Hi guys,

 I have about 20k documents in the Solr index (and there's a lot of
 text in each of them). I have field collapsing enabled on a specific
 field (AdvisorID).

 The thing is if I have field collapsing enabled in the search request
 I don't get correct count for the total number of records that
 matched. It always says that the number of rows I asked to get back
 is the number of total records it found.

 And, when I run a query with search criteria *:* (to get the number of
 total advisors in the index) solr runs of out memory and gives me an
 error saying

 SEVERE: java.lang.OutOfMemoryError: Java heap space
        at java.nio.CharBuffer.wrap(CharBuffer.java:350)
        at java.nio.CharBuffer.wrap(CharBuffer.java:373)
        at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:138)
        at java.lang.StringCoding.decode(StringCoding.java:173)


 This is going to be a huge problem later on when we index 50k
 documents later on.

 These are the options I am running Solr with :

 java  -Xms2048M -Xmx2048M -XX:+UseConcMarkSweepGC -XX:PermSize=1024m
 MaxPermSize=1024m    -jar  start.jar


 Is there any way I can get the counts and not run out of memory?

 Thanks in advance,
 Moazzam

DIH - Out of Memory error when using CachedsqlEntityProcessor

[JavaGuy84]

Hi,

I am using CachedsqlEntityProcessor in my DIH dataconfig to reduce the
number of queries executed against the database ,

Entity1 query=select * from x processor= CachedsqlEntityProcessor/
Entity2 query=select * from y processor= CachedsqlEntityProcessor
cachekey=id cachelookup=x.id/


I having more than 2 million rows returned for Entity 2 and around 30
rows returned for entity1.

I am have set the heap size to 1 GB but even then I am always getting heap
out of size error. I am not sure how to flush the documents in buffer at
certain condition. I tried to enable Autocommit / reduced the
maxdocbuffersize but of no use.. Can someone let me know what is the best
way to overcome this issue?

Thanks,
Barani
-- 
View this message in context: 
http://old.nabble.com/DIH---Out-of-Memory-error-when-using-CachedsqlEntityProcessor-tp27889623p27889623.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: DIH - Out of Memory error when using CachedsqlEntityProcessor

[JavaGuy84]

 Source)
at
com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.log(P
rintStreamSocketLog.java:169)
at
com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.log(P
rintStreamSocketLog.java:175)
at
com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.logEr
ror(PrintStreamSocketLog.java:71)
at
com.metamatrix.common.comm.platform.socket.client.SocketServerInstanc
eImpl$1.run(SocketServerInstanceImpl.java:578)
at java.lang.Thread.run(Unknown Source)
Mar 13, 2010 3:52:09 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback

Thanks,
Barani

JavaGuy84 wrote:
 
 Hi,
 
 I am using CachedsqlEntityProcessor in my DIH dataconfig to reduce the
 number of queries executed against the database ,
 
 Entity1 query=select * from x processor= CachedsqlEntityProcessor/
 Entity2 query=select * from y processor= CachedsqlEntityProcessor
 cachekey=id cachelookup=x.id/
 
 
 I having more than 2 million rows returned for Entity 2 and around 30
 rows returned for entity1.
 
 I am have set the heap size to 1 GB but even then I am always getting heap
 out of size error. I am not sure how to flush the documents in buffer at
 certain condition. I tried to enable Autocommit / reduced the
 maxdocbuffersize but of no use.. Can someone let me know what is the best
 way to overcome this issue?
 
 Thanks,
 Barani
 

-- 
View this message in context: 
http://old.nabble.com/DIH---Out-of-Memory-error-when-using-CachedsqlEntityProcessor-tp27889623p27890751.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: DIH - Out of Memory error when using CachedsqlEntityProcessor

[Erick Erickson]

 Source)
at sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
at sun.nio.cs.StreamEncoder.write(Unknown Source)
at java.io.OutputStreamWriter.write(Unknown Source)
at java.io.BufferedWriter.flushBuffer(Unknown Source)
at java.io.PrintStream.write(Unknown Source)
at java.io.PrintStream.print(Unknown Source)
at
 com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.log(P
 rintStreamSocketLog.java:169)
at
 com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.log(P
 rintStreamSocketLog.java:175)
at
 com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.logEr
 ror(PrintStreamSocketLog.java:71)
at
 com.metamatrix.common.comm.platform.socket.client.SocketServerInstanc
 eImpl$1.run(SocketServerInstanceImpl.java:578)
at java.lang.Thread.run(Unknown Source)
 Mar 13, 2010 3:52:09 PM org.apache.solr.update.DirectUpdateHandler2
 rollback
 INFO: end_rollback

 Thanks,
 Barani

 JavaGuy84 wrote:
 
  Hi,
 
  I am using CachedsqlEntityProcessor in my DIH dataconfig to reduce the
  number of queries executed against the database ,
 
  Entity1 query=select * from x processor= CachedsqlEntityProcessor/
  Entity2 query=select * from y processor= CachedsqlEntityProcessor
  cachekey=id cachelookup=x.id/
 
 
  I having more than 2 million rows returned for Entity 2 and around 30
  rows returned for entity1.
 
  I am have set the heap size to 1 GB but even then I am always getting
 heap
  out of size error. I am not sure how to flush the documents in buffer at
  certain condition. I tried to enable Autocommit / reduced the
  maxdocbuffersize but of no use.. Can someone let me know what is the best
  way to overcome this issue?
 
  Thanks,
  Barani
 

 --
 View this message in context:
 http://old.nabble.com/DIH---Out-of-Memory-error-when-using-CachedsqlEntityProcessor-tp27889623p27890751.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: DIH - Out of Memory error when using CachedsqlEntityProcessor

[JavaGuy84]

Erik,

I have seen many posts regarding out of memory error but I am not sure
whether they are using cachesqlEntityProcessor.. 

I want to know if there is a way to flush out the buffer of cache instead of
storing everything in cache.

I can clearly see the heapsize growing like anything if I use the
cachesqlentity processor, trying to figure out if there is a way to resolve
this by using any other way other than using this processor.

Thanks,
Barani


Erick Erickson wrote:
 
 Have you searched the users' list? This question has come up multiple
 times
 and you'll find your question has probably already been answered. Let us
 know if you come up blank...
 
 Best
 Erick
 
 On Sat, Mar 13, 2010 at 3:56 PM, JavaGuy84 bbar...@gmail.com wrote:
 

 Sorry forgot to attach the error log,

 Error Log:
 -
 org.apache.solr.handler.dataimport.DataImportHandlerException:
 java.lang.OutOfMe
 moryError: Java heap space
at
 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
 r.java:650)
at
 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
 r.java:605)
at
 org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j
 ava:261)
at
 org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java
 :185)
at
 org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo
 rter.java:333)
at
 org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j
 ava:391)
at
 org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja
 va:372)
 Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.init(Unknown Source)
at
 org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.g
 etARow(JdbcDataSource.java:281)
at
 org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.a
 ccess$800(JdbcDataSource.java:228)
at
 org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
 .next(JdbcDataSource.java:266)
at
 org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
 .next(JdbcDataSource.java:269)
at
 org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.getAllNon
 CachedRows(CachedSqlEntityProcessor.java:70)
at
 org.apache.solr.handler.dataimport.EntityProcessorBase.getIdCacheData
 (EntityProcessorBase.java:194)
at
 org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.nextRow(C
 achedSqlEntityProcessor.java:58)
at
 org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(Ent
 ityProcessorWrapper.java:233)
at
 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
 r.java:579)
... 6 more
 Mar 13, 2010 3:52:09 PM org.apache.solr.handler.dataimport.DataImporter
 doFullIm
 port
 SEVERE: Full Import failed
 org.apache.solr.handler.dataimport.DataImportHandlerException:
 java.lang.OutOfMe
 moryError: Java heap space
at
 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
 r.java:650)
at
 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
 r.java:605)
at
 org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j
 ava:261)
at
 org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java
 :185)
at
 org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo
 rter.java:333)
at
 org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j
 ava:391)
at
 org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja
 va:372)
 Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.init(Unknown Source)
at
 org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.g
 etARow(JdbcDataSource.java:281)
at
 org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.a
 ccess$800(JdbcDataSource.java:228)
at
 org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
 .next(JdbcDataSource.java:266)
at
 org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
 .next(JdbcDataSource.java:269)
at
 org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.getAllNon
 CachedRows(CachedSqlEntityProcessor.java:70)
at
 org.apache.solr.handler.dataimport.EntityProcessorBase.getIdCacheData
 (EntityProcessorBase.java:194)
at
 org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.nextRow(C
 achedSqlEntityProcessor.java:58)
at
 org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(Ent
 ityProcessorWrapper.java:233)
at
 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
 r.java:579)
... 6 more
 Mar 13, 2010 3:52:09 PM org.apache.solr.update.DirectUpdateHandler2
 rollback
 INFO: start rollback
 Exception in thread Timer-0 java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.init(Unknown Source

Re: DIH - Out of Memory error when using CachedsqlEntityProcessor

[Mark Miller]

I don't really follow DataImportHandler, but it looks like its using an 
unbounded cache (simple HashMap).


Perhaps we should make the cache size configurable?

The impl seems a little odd - the caching occurs in the base class - so 
caching impls that extends it don't really have full control - they just 
kind of turn on the caching in the base class? Kind of an odd approach 
- to cache you have to turn on the cache support in the base class and 
impl a couple custom methods as well?


--
- Mark

http://www.lucidimagination.com



On 03/13/2010 06:11 PM, JavaGuy84 wrote:

Erik,

I have seen many posts regarding out of memory error but I am not sure
whether they are using cachesqlEntityProcessor..

I want to know if there is a way to flush out the buffer of cache instead of
storing everything in cache.

I can clearly see the heapsize growing like anything if I use the
cachesqlentity processor, trying to figure out if there is a way to resolve
this by using any other way other than using this processor.

Thanks,
Barani


Erick Erickson wrote:
   

Have you searched the users' list? This question has come up multiple
times
and you'll find your question has probably already been answered. Let us
know if you come up blank...

Best
Erick

On Sat, Mar 13, 2010 at 3:56 PM, JavaGuy84bbar...@gmail.com  wrote:

 

Sorry forgot to attach the error log,

Error Log:
-
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMe
moryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:650)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:605)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j
ava:261)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java
:185)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo
rter.java:333)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j
ava:391)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja
va:372)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.init(Unknown Source)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.g
etARow(JdbcDataSource.java:281)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.a
ccess$800(JdbcDataSource.java:228)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:266)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:269)
at
org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.getAllNon
CachedRows(CachedSqlEntityProcessor.java:70)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getIdCacheData
(EntityProcessorBase.java:194)
at
org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.nextRow(C
achedSqlEntityProcessor.java:58)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(Ent
ityProcessorWrapper.java:233)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:579)
... 6 more
Mar 13, 2010 3:52:09 PM org.apache.solr.handler.dataimport.DataImporter
doFullIm
port
SEVERE: Full Import failed
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMe
moryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:650)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:605)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j
ava:261)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java
:185)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo
rter.java:333)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j
ava:391)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja
va:372)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.init(Unknown Source)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.g
etARow(JdbcDataSource.java:281)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.a
ccess$800(JdbcDataSource.java:228)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:266)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:269)
at
org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.getAllNon
CachedRows(CachedSqlEntityProcessor.java:70)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getIdCacheData
(EntityProcessorBase.java:194

Heap Out of Memory Error

[[EMAIL PROTECTED]]

Hi,

I am running Solr within the Jetty using start.jar. I am indexing about 200,000 
documents. Sometimes out of the blue, the Solr instance cannot process any more 
requests and returns heap out of memory error.

This happens more often when I issue queries against the index that is being 
updated.

Is there some configuration setting I need to change?

Also, the server itself has plenty of RAM even when this error appears. So it 
appears Java is running out of its heap memory while there is still enough of 
RAM available for other processes.

Thanks,
Av


  

Park yourself in front of a world of choices in alternative vehicles. Visit the 
Yahoo! Auto Green Center.
http://autos.yahoo.com/green_center/ 

RE: Heap Out of Memory Error

[Daniel Pitts]

 -Original Message-
 From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] 
 Sent: Tuesday, May 22, 2007 12:31 PM

 Hi,
 
 I am running Solr within the Jetty using start.jar. I am 
 indexing about 200,000 documents. Sometimes out of the blue, 
 the Solr instance cannot process any more requests and 
 returns heap out of memory error.
 
 This happens more often when I issue queries against the 
 index that is being updated.
 
 Is there some configuration setting I need to change?
 
 Also, the server itself has plenty of RAM even when this 
 error appears. So it appears Java is running out of its heap 
 memory while there is still enough of RAM available for other 
 processes.
 
 Thanks,
 Av


This seems like the java heap size is set too low.  Wherever you start
the JVM, you can pass in -Xmx512m -Xms256m. Adjusting the values as
necessary.