You're adding on the order of 750 rows (docs)/second, which isn't bad...
have you profiled the machine as this runs? Even just with top (assuming
unix)...
because the very first question is always "what takes the time, getting
the data from MySQL or indexing or I/O?".
If you aren't maxing out your CPU, then you probably want to explore the
other
questions (db query speed, network latency) to get a sense whether you're
going as fast as you can or not...
Best
Erick
2010/12/15 Robert Gründler <rob...@dubture.com>
> Hi,
>
> we're looking for some comparison-benchmarks for importing large tables
> from a mysql database (full import).
>
> Currently, a full-import of ~ 8 Million rows from a MySQL database takes
> around 3 hours, on a QuadCore Machine with 16 GB of
> ram and a Raid 10 storage setup. Solr is running on a apache tomcat
> instance, where it is the only app. The tomcat instance
> has the following memory-related java_opts:
>
> -Xms4096M -Xmx5120M
>
>
> The data-config.xml looks like this (only 1 entity):
>
> <entity name="track" query="select t.id as id, t.title as title,
> l.title as label from track t left join label l on (l.id = t.label_id)
> where t.deleted = 0" transformer="TemplateTransformer">
> <field column="title" name="title_t" />
> <field column="label" name="label_t" />
> <field column="id" name="sf_meta_id" />
> <field column="metaclass" template="Track" name="sf_meta_class"/>
> <field column="metaid" template="${track.id}" name="sf_meta_id"/>
> <field column="uniqueid" template="Track_${track.id}"
> name="sf_unique_id"/>
>
> <entity name="artists" query="select a.name as artist from artist a
> left join track_artist ta on (ta.artist_id = a.id) where ta.track_id=${
> track.id}">
> <field column="artist" name="artists_t" />
> </entity>
>
> </entity>
>
>
> We have the feeling that 3 hours for this import is quite long - regarding
> the performance of the server running solr/mysql.
>
> Are we wrong with that assumption, or do people experience similar import
> times with this amount of data to be imported?
>
>
> thanks!
>
>
> -robert
>
>
>
>
