Thank you for answering. Some notes: 1. The Java engine I´ve developed that wrappers SolrJ 4.1 with some business logic only executes search queries, not index/update operations, so the problem is not related with concurrent updates, or something similar.
2. As far as I know the better SolrJ interface to index with SolrCloud is CloudSolrServer, not ConcurrentUpdateSolrServer. If you have many instances of CloudSolrServer and you correctly balance them with a Round Robin or something similar you´ll get a better performance in SolrCloud scenarios. At least is what I´ve read in the documentation, and also I asked to Mark Miller some months ago when I started dealing with Solr 4.0-BETA. 3. I´m almost convinced that the problem is related with: - Zookeeper ensemble configuration. - Zookeeper version (3.4.5) is not compatible with Solr 4.1. expected one. - SolrJ Zookeeper driver. In short, all my architecture works perfectly with search operations. Also I´ve got another NRT Indexer module that deals with CloudSolrServer and works perfectly. But after two, three days, something happens with Zookeeper - CloudSolrServer connection, and tries to update cluster status forever with no success. Only after Zookeeper + SolrCloud leader&replica shards restart the problem is solved. 2013/3/19 Michael Della Bitta <michael.della.bi...@appinions.com> > Don't use CloudSolrServer for writes. Instead, use > ConcurrentUpdateSolrServer, something like: > > SolrServer solrServer = new ConcurrentUpdateSolrServer(solrUrl, 100, 4); > > The 100 corresponds to how many docs to send in a batch. The higher > this is, the better performance is (to a point, don't set that to 50k > or anything). > > The 4 corresponds to the number of threads that will be sending batches. > > Note that this class doesn't report errors, so if you want to see > exceptions when bad things happen, you'll have to override > handleError(Throwable ex) method. > > Here's the javadoc for the class: > > http://lucene.apache.org/solr/4_2_0/solr-solrj/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.html > > It'd be best if you can use a load balancer in front of your Solr > Cloud and use that as the solrUrl parameter. > > ***Either way, though, Mark is right in that you need to diagnose why > you're only able to do a few documents per second first.*** Adding > more threads at this point is probably not going to help. > > Michael Della Bitta > > ------------------------------------------------ > Appinions > 18 East 41st Street, 2nd Floor > New York, NY 10017-6271 > > www.appinions.com > > Where Influence Isn’t a Game > > > On Tue, Mar 19, 2013 at 3:57 PM, Luis Cappa Banda <luisca...@gmail.com> > wrote: > > Anyone can help me? Each response may save a little kitten from a > horrible > > and dramatic death somewhere in the world :-P > > El 15/03/2013 21:06, "Jack Park" <jackp...@topicquests.org> escribió: > > > >> Is there a document that tells how to create multiple threads? Search > >> returns many hits which orbit this idea, but I haven't spotted one > >> which tells how. > >> > >> Thanks > >> Jack > >> > >> On Fri, Mar 15, 2013 at 1:01 PM, Mark Miller <markrmil...@gmail.com> > >> wrote: > >> > You def have to use multiple threads with it for it to be fast, but 3 > or > >> 4 docs a second still sounds absurdly slow. > >> > > >> > - Mark > >> > > >> > On Mar 15, 2013, at 2:58 PM, Luis Cappa Banda <luisca...@gmail.com> > >> wrote: > >> > > >> >> And up! :-) > >> >> > >> >> I´ve been wondering if using CloudSolrServer has something to do > here. > >> Does > >> >> it have a bad performance when a CloudSolrServer singletong receives > >> >> multiple queries? Is it recommended to have a CloudSolrServer > instances > >> >> list and select one of them with a Round Robin criteria? > >> >> > >> >> > >> >> > >> >> 2013/3/14 Luis Cappa Banda <luisca...@gmail.com> > >> >> > >> >>> Hello! > >> >>> > >> >>> Thanks a lot, Erick! I've attached some stack traces during a normal > >> >>> 'engine' running. > >> >>> > >> >>> Cheers, > >> >>> > >> >>> - Luis Cappa > >> >>> > >> >>> > >> >>> 2013/3/13 Erick Erickson <erickerick...@gmail.com> > >> >>> > >> >>>> Stack traces.. > >> >>>> > >> >>>> First, > >> >>>> jps -l > >> >>>> > >> >>>> that will give you a the process IDs of your running Java > processes. > >> Then: > >> >>>> > >> >>>> jstack <pid from above> > >> >>>> > >> >>>> Usually I pipe the output from jstack into a text file... > >> >>>> > >> >>>> Best > >> >>>> Erick > >> >>>> > >> >>>> > >> >>>> On Wed, Mar 13, 2013 at 1:48 PM, Luis Cappa Banda < > >> luisca...@gmail.com > >> >>>>> wrote: > >> >>>> > >> >>>>> Uhm, how can I do that... 'cleanly'? I know that with JConsole > it´s > >> >>>> posible > >> >>>>> to output this traces, but with a .war application built on top of > >> >>>> Spring I > >> >>>>> don´t know how can I do that. In any case, here is my > CloudSolrServer > >> >>>>> wrapper that is used by other classes. There is no sync method or > >> piece > >> >>>> of > >> >>>>> code: > >> >>>>> > >> >>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - > >> >>>> - - > >> >>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - > >> >>>>> > >> >>>>> *public class BinaryLBHttpSolrServer extends LBHttpSolrServer {* > >> >>>>> > >> >>>>> private static final long serialVersionUID = 3905956120804659445L; > >> >>>>> public BinaryLBHttpSolrServer(String[] endpoints) throws > >> >>>>> MalformedURLException { > >> >>>>> super(endpoints); > >> >>>>> } > >> >>>>> > >> >>>>> @Override > >> >>>>> protected HttpSolrServer makeServer(String server) throws > >> >>>>> MalformedURLException { > >> >>>>> HttpSolrServer solrServer = super.makeServer(server); > >> >>>>> solrServer.setRequestWriter(new BinaryRequestWriter()); > >> >>>>> return solrServer; > >> >>>>> } > >> >>>>> } > >> >>>>> > >> >>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - > >> >>>> - - > >> >>>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - > >> >>>>> > >> >>>>> *public class CloudSolrHttpServerImpl implements > CloudSolrHttpServer > >> {* > >> >>>>> private CloudSolrServer cloudSolrServer; > >> >>>>> > >> >>>>> private Logger log = > Logger.getLogger(CloudSolrHttpServerImpl.class); > >> >>>>> > >> >>>>> public CloudSolrHttpServerImpl(String zookeeperEndpoints, String[] > >> >>>>> endpoints, int clientTimeout, > >> >>>>> int connectTimeout, String cloudCollection) { > >> >>>>> try { > >> >>>>> BinaryLBHttpSolrServer lbSolrServer = new *BinaryLBHttpSolrServer* > >> >>>>> (endpoints); > >> >>>>> this.cloudSolrServer = new CloudSolrServer(zookeeperEndpoints, > >> >>>>> lbSolrServer); > >> >>>>> this.cloudSolrServer.setZkConnectTimeout(connectTimeout); > >> >>>>> this.cloudSolrServer.setZkClientTimeout(clientTimeout); > >> >>>>> this.cloudSolrServer.setDefaultCollection(cloudCollection); > >> >>>>> } catch (MalformedURLException e) { > >> >>>>> log.error(e); > >> >>>>> } > >> >>>>> } > >> >>>>> > >> >>>>> @Override > >> >>>>> public QueryResponse *search*(SolrQuery query) throws > >> >>>> SolrServerException { > >> >>>>> return cloudSolrServer.query(query, METHOD.POST); > >> >>>>> } > >> >>>>> > >> >>>>> @Override > >> >>>>> public boolean *index*(DocumentBean user) { > >> >>>>> boolean indexed = false; > >> >>>>> int retries = 0; > >> >>>>> do { > >> >>>>> indexed = addBean(user); > >> >>>>> retries++; > >> >>>>> } while(!indexed && retries<4); > >> >>>>> return indexed; > >> >>>>> } > >> >>>>> @Override > >> >>>>> public boolean *update*(SolrInputDocument updateDoc) { > >> >>>>> boolean update = false; > >> >>>>> int retries = 0; > >> >>>>> > >> >>>>> do { > >> >>>>> update = addSolrInputDocument(updateDoc); > >> >>>>> retries++; > >> >>>>> } while(!update && retries<4); > >> >>>>> return update; > >> >>>>> } > >> >>>>> @Override > >> >>>>> public void commit() { > >> >>>>> try { > >> >>>>> cloudSolrServer.commit(); > >> >>>>> } catch (SolrServerException e) { > >> >>>>> log.error(e); > >> >>>>> } catch (IOException e) { > >> >>>>> log.error(e); > >> >>>>> } > >> >>>>> } > >> >>>>> > >> >>>>> @Override > >> >>>>> public boolean *delete*(String ... ids) { > >> >>>>> boolean deleted = false; > >> >>>>> List<String> idList = Arrays.asList(ids); > >> >>>>> try { > >> >>>>> this.cloudSolrServer.deleteById(idList); > >> >>>>> this.cloudSolrServer.commit(true, true); > >> >>>>> deleted = true; > >> >>>>> > >> >>>>> } catch (SolrServerException e) { > >> >>>>> log.error(e); > >> >>>>> > >> >>>>> } catch (IOException e) { > >> >>>>> log.error(e); > >> >>>>> } > >> >>>>> return deleted; > >> >>>>> } > >> >>>>> > >> >>>>> @Override > >> >>>>> public void *optimize*() { > >> >>>>> try { > >> >>>>> this.cloudSolrServer.optimize(); > >> >>>>> } catch (SolrServerException e) { > >> >>>>> log.error(e); > >> >>>>> } catch (IOException e) { > >> >>>>> log.error(e); > >> >>>>> } > >> >>>>> } > >> >>>>> /* > >> >>>>> * ******************** > >> >>>>> * Getters & setters * > >> >>>>> * ******************** > >> >>>>> * */ > >> >>>>> public CloudSolrServer getSolrServer() { > >> >>>>> return cloudSolrServer; > >> >>>>> } > >> >>>>> > >> >>>>> public void setSolrServer(CloudSolrServer solrServer) { > >> >>>>> this.cloudSolrServer = solrServer; > >> >>>>> } > >> >>>>> > >> >>>>> private boolean addBean(DocumentBean user) { > >> >>>>> boolean added = false; > >> >>>>> try { > >> >>>>> this.cloudSolrServer.addBean(user, 100); > >> >>>>> this.commit(); > >> >>>>> > >> >>>>> } catch (IOException e) { > >> >>>>> log.error(e); > >> >>>>> > >> >>>>> } catch (SolrServerException e) { > >> >>>>> log.error(e); > >> >>>>> }catch(SolrException e) { > >> >>>>> log.error(e); > >> >>>>> } > >> >>>>> return added; > >> >>>>> } > >> >>>>> private boolean addSolrInputDocument(SolrInputDocument updateDoc) > { > >> >>>>> boolean added = false; > >> >>>>> try { > >> >>>>> this.cloudSolrServer.add(updateDoc, 100); > >> >>>>> this.commit(); > >> >>>>> added = true; > >> >>>>> } catch (IOException e) { > >> >>>>> log.error(e); > >> >>>>> > >> >>>>> } catch (SolrServerException e) { > >> >>>>> log.error(e); > >> >>>>> }catch(SolrException e) { > >> >>>>> log.error(e); > >> >>>>> } > >> >>>>> return added; > >> >>>>> } > >> >>>>> } > >> >>>>> > >> >>>>> Thank you very much, Mark. > >> >>>>> > >> >>>>> > >> >>>>> - Luis Cappa > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> And > >> >>>>> 2013/3/13 Mark Miller <markrmil...@gmail.com> > >> >>>>> > >> >>>>>> > >> >>>>>> Could you capture some thread stack traces in the 'engine' and > see > >> if > >> >>>>>> there are any blocking methods? > >> >>>>>> > >> >>>>>> - Mark > >> >>>>>> > >> >>>>>> On Mar 13, 2013, at 1:34 PM, Luis Cappa Banda < > luisca...@gmail.com> > >> >>>>> wrote: > >> >>>>>> > >> >>>>>>> Just one correction: > >> >>>>>>> > >> >>>>>>> When I said: > >> >>>>>>> > >> >>>>>>> - I´ve checked SolrCloud via Solr Admin interface and it´s OK: > >> >>>>>>> everything is green, and I cant execute queries directly into > >> >>>> Solr. > >> >>>>>>> > >> >>>>>>> I mean: > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - I´ve checked SolrCloud via Solr Admin interface and it´s OK: > >> >>>>>>> everything is green, and *I can* execute queries directly into > >> >>>> Solr. > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> Thanks! > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - Luis Cappa > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> 2013/3/13 Luis Cappa Banda <luisca...@gmail.com> > >> >>>>>>> > >> >>>>>>>> Hello, guys! > >> >>>>>>>> > >> >>>>>>>> I´ve been experiencing some annoying behavior with my current > >> >>>>> production > >> >>>>>>>> scenario. Here is the snapshot: > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> - SolrCloud: 2 shards > >> >>>>>>>> - Zookeeper ensemble: 3 nodes in *different machines *(most of > >> >>>> the > >> >>>>>>>> tutorials installs 3 Zookeeper nodes in the same machine). > >> >>>>>>>> - This is the zoo.cfg from every > >> >>>>>>>> > >> >>>>>>>> tickTime=2000 // I´ve also tried with 60000 > >> >>>>>>>> > >> >>>>>>>> initLimit=10 > >> >>>>>>>> > >> >>>>>>>> syncLimit=5 > >> >>>>>>>> > >> >>>>>>>> dataDir=/var/lib/zookeeper > >> >>>>>>>> > >> >>>>>>>> clientPort=9000 > >> >>>>>>>> > >> >>>>>>>> server.1=zoohost1:2888:3888 > >> >>>>>>>> > >> >>>>>>>> server.2=zoohost1:2888:3888 > >> >>>>>>>> > >> >>>>>>>> server.3=zoohost1:2888:3888 > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> - I´ve developed a Java Application with a REST API (let´s > call > >> >>>> it * > >> >>>>>>>> engine*) that dispatches queries into SolrCloud. It´s a > wrapper > >> >>>>> around > >> >>>>>>>> CloudSolrServer, so it´s mandatory to specify some Zookeeper > >> >>>>>> configuration > >> >>>>>>>> params too. They are loaded dynamically when the application > is > >> >>>>>> deployed in > >> >>>>>>>> a Tomcat server, but the current values that I´m using are as > >> >>>>> follows: > >> >>>>>>>> > >> >>>>>>>> cloudSolrServer.*setZkConnectTimeout(60000)* > >> >>>>>>>> > >> >>>>>>>> cloudSolrServer.*setZkClientTimeout(60000)* > >> >>>>>>>> * > >> >>>>>>>> * > >> >>>>>>>> * > >> >>>>>>>> * > >> >>>>>>>> > >> >>>>>>>> *THE PROBLEM* > >> >>>>>>>> * > >> >>>>>>>> * > >> >>>>>>>> Everything goes OK, but after two days more or less (yes, I´ve > >> >>>> checked > >> >>>>>>>> that this behavior occurrs periodically, more or less) the > *engine > >> >>>>>> blocks > >> >>>>>>>> * and cannot dispatch any query to SolrCloud. > >> >>>>>>>> > >> >>>>>>>> - The *engine *log only outputs "updating Zookeeper..." one > last > >> >>>>> time, > >> >>>>>>>> but never updates. > >> >>>>>>>> - I´ve checked SolrCloud via Solr Admin interface and it´s OK: > >> >>>>>>>> everything is green, and I cant execute queries directly into > >> >>>> Solr. > >> >>>>>>>> - So then Solr appears to be OK, so the next step is to > restart > >> >>>>>> *engine > >> >>>>>>>> but *it again appears "updating Zookeeper...". Unfortunately > >> >>>> switch > >> >>>>>>>> off + switch on doesn´t work here, :-( > >> >>>>>>>> - I´ve checked too Zookeeper logs and it appears some > connection > >> >>>> log > >> >>>>>>>> outs, but the ensemble appears to be OK too. > >> >>>>>>>> - *The end: *If I restart Zookeeper one by one, and I restart > >> >>>>>>>> SolrCloud, plus I restart the engine, the problem is solved. > I´m > >> >>>>> using > >> >>>>>>>> Amazon AWS as hostage, so I discard connection problems > between > >> >>>>>> instances. > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> Does anyone experienced something similar? Can anybody shed > some > >> >>>> light > >> >>>>>> on > >> >>>>>>>> this problem? > >> >>>>>>>> > >> >>>>>>>> Thank you very much. > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> Regards, > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> - Luis Cappa > >> >>>>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>> > >> >>>> > >> >>> > >> >>> > >> > > >> > -- Luis Cappa Banda *Phone*: (0034) 686 200 375 *Skype*: luiscappabanda
