Re: DataImporter : Java heap space

On Thu, Apr 16, 2009 at 10:31 AM, Mani Kumar manikumarchau...@gmail.comwrote: Aah, Bryan you got it ... Thanks! Noble: so i can hope that it'll be fixed soon :) thank you for fixing it ... please lemme know when its done.. This is fixed in trunk. The next nightly build should have this fix.

Re: DataImporter : Java heap space

I think there is a bug in the 1.4 daily builds of data import handler which is causing the batchSize parameter to be ignored. This was probably introduced with more recent patches to resolve variables. The affected code is in JdbcDataSource.java String bsz =

Re: DataImporter : Java heap space

Hi Bryan, Thanks a lot. It is invoking the wrong method it should have been bsz = context.getVariableResolver().replaceTokens(bsz); it was a silly mistake --Noble On Thu, Apr 16, 2009 at 2:13 AM, Bryan Talbot btal...@aeriagames.com wrote: I think there is a bug in the 1.4 daily builds of data

Re: DataImporter : Java heap space

Aah, Bryan you got it ... Thanks! Noble: so i can hope that it'll be fixed soon :) thank you for fixing it ... please lemme know when its done.. Thanks! Mani Kumar 2009/4/16 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@gmail.com Hi Bryan, Thanks a lot. It is invoking the wrong method it should have

Re: DataImporter : Java heap space

Hi Shalin: yes i tried with batchSize=-1 parameter as well here the config i tried with dataConfig dataSource type=JdbcDataSource batchSize=-1 name=sp driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost/mydb_development user=root password=** / document name=items

Re: DataImporter : Java heap space

On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar manikumarchau...@gmail.comwrote: Hi Shalin: yes i tried with batchSize=-1 parameter as well here the config i tried with dataConfig dataSource type=JdbcDataSource batchSize=-1 name=sp driver=com.mysql.jdbc.Driver

Re: DataImporter : Java heap space

Yes its throwing the same OOM error and from same place... yes i will try increasing the size ... just curious : how this dataimport works? Does it loads the whole table into memory? Is there any estimate about how much memory it needs to create index for 1GB of data. thx mani On Tue, Apr 14,

Re: DataImporter : Java heap space

DIH streams 1 row at a time. DIH is just a component in Solr. Solr indexing also takes a lot of memory On Tue, Apr 14, 2009 at 12:02 PM, Mani Kumar manikumarchau...@gmail.com wrote: Yes its throwing the same OOM error and from same place... yes i will try increasing the size ... just curious :

Re: DataImporter : Java heap space

I am using Tomcat ... On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar manikumarchau...@gmail.comwrote: Hi All, I am trying to setup a Solr instance on my macbook. I get following errors when m trying to do a full db import ... please help me on this Apr 13, 2009 11:53:28 PM

Re: DataImporter : Java heap space

On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar manikumarchau...@gmail.comwrote: Hi All, I am trying to setup a Solr instance on my macbook. I get following errors when m trying to do a full db import ... please help me on this java.lang.OutOfMemoryError: Java heap space at

Re: DataImporter : Java heap space

Hi Shalin: Thanks for quick response! By defaults it was set to 1.93 MB. But i also tried it with following command: $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M I also tried tricks given on http://wiki.apache.org/solr/DataImportHandlerFaq page. what should i try next ?

Re: DataImporter : Java heap space

Depending on your dataset and how your queries look you may very likely need to increase to a larger heap size. How many queries and rows are required for each of your documents to be generated? Ilan On 4/13/09 12:21 PM, Mani Kumar wrote: Hi Shalin: Thanks for quick response! By defaults

Re: DataImporter : Java heap space

Hi ILAN: Only one query is required to generate a document ... Here is my data-config.xml dataConfig dataSource type=JdbcDataSource name=sp driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost/mydb_development user=root password=** / document name=items entity name=item

Re: DataImporter : Java heap space

DIH itself may not be consuming so much memory. It also includes the memory used by Solr. Do you have a hard limit on 400MB , is it not possible to increase it? On Tue, Apr 14, 2009 at 11:09 AM, Mani Kumar manikumarchau...@gmail.com wrote: Hi ILAN: Only one query is required to generate a

Re: DataImporter : Java heap space

Here is the stack trace: notice in stack trace * at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749)* It looks like that its trying to read whole table into memory at a time. n thts y getting OOM. Apr 14, 2009 11:15:01 AM org.apache.solr.handler.dataimport.DataImporter

Re: DataImporter : Java heap space

Hi Noble: But the question is how much memory? is there any rules or something like that? so that i can estimate the how much memory it requires? Yeah i can increase it upto 800MB max will try it and let you know Thanks! Mani 2009/4/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@gmail.com DIH

Re: DataImporter : Java heap space

On Tue, Apr 14, 2009 at 11:18 AM, Mani Kumar manikumarchau...@gmail.comwrote: Here is the stack trace: notice in stack trace * at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749)* It looks like that its trying to read whole table into memory at a time. n thts y getting OOM.