Note that the search-cmd is for search pre 2.0, which does not use solr. If you're planning on using the new Solr based search, you'll need to run Riak 2.0, and write an import script as Dmitri pointed out.
Eric Redmond, Engineer @ Basho
Hi Mark,Spin up a bunch of worker threads (and have them share a pool of connections) and have them issue parallel concurrent puts to all of the nodes in a cluster (you can either use something like HAProxy to load balance, or a riak client's internal load balancing capabilities).
The best way to bulk load objects into Riak (and into Solr) is to take advantage of Riak's parallelism.
This is what https://github.com/basho-labs/riak-data-migrator does, for example, to restore filed out objects quickly.On Thu, Aug 14, 2014 at 5:26 AM, Mark Richard Thomas <mark.tho...@equifax.com> wrote:
Hello
What’s the fastest way (best practice) to insert 20 million documents into a Riak Search index?
search-cmd solr my_bucket /insert.xml
For a proof-of-concept I’ve create a file (index.html) containing 100,000 documents:
<add>
<doc></doc>
<doc></doc>
:
</add>
Thanks
Mark Thomas | Software Engineer | Equifax UK
Equifax Ltd, Capital House, 25 Chapel Street, London, NW1 5DS
Equifax Limited is registered in England with Registered No. 2425920. Registered Office: Capital House, 25 Chapel Street, London NW1 5DS. Equifax Limited is authorised and regulated by the Financial Conduct Authority.
Equifax Touchstone Limited is registered in Scotland with Registered No. SC113401. Registered Office: 54 Deerdykes View, Westfield Park, Cumbernauld G68 9HN.
Equifax Commercial Services Limited is registered in the Republic of Ireland with Registered No. 215393. Registered Office: IDA Business & Technology Park, Rosslare Road, Drinagh, Wexford.
This message contains information from Equifax which may be confidential and privileged. If you are not an intended recipient, please refrain from any disclosure, copying, distribution or use of this information and note that such actions are prohibited. If you have received this transmission in error, please notify by e-mail postmas...@equifax.com.
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
