Re: EofException with Solr in Jetty
There is nothing between the client app and the solr server, its on the same machine and on the same app server, only going through the loopback interface. Unfortunatly, I cannot reproduce it, but I see it in the server log. Thanks Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business - Ursprüngliche Mail - Von: "Jay Jaeger - DOT" An: solr-user@lucene.apache.org Gesendet: Mittwoch, 14. September 2011 16:23:45 Betreff: RE: EofException with Solr in Jetty I have not used SolrJ, but it probably is worth considering as a possible suspect. Also, do you have anything in between the client and the Solr server (a firewall, load balancer, etc.?) that might play games with HTTP connections? You might want to start up a network trace on the server or network to see if you can catch one to see what is going on. I looked at our Solr 3.1 prototype log (which has been running continuously without interruption since July 10!), and did not see any of these errors. We do not use SolrJ -- we use a combination of plain old HTTP/javascript/xslt and requests coming from another system as a (plain old XML) web service to get to Solr. However, that is under Jetty 6. JRJ -Original Message----- From: Michael Szalay [mailto:michael.sza...@basis06.ch] Sent: Wednesday, September 14, 2011 8:27 AM To: solr-user@lucene.apache.org Subject: Re: EofException with Solr in Jetty We are using SolrJ 3.1 as our http client... So it may be a bug in there? Regards Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business - Ursprüngliche Mail - Von: "Jay Jaeger - DOT" An: solr-user@lucene.apache.org, "JETTY user mailing list" Gesendet: Mittwoch, 14. September 2011 15:21:19 Betreff: RE: EofException with Solr in Jetty Looking at the source for Jetty, line 149 in Jetty's HttpOutput java file looks like this: if (_closed) throw new IOException("Closed"); <<<<< [http://www.jarvana.com/jarvana/view/org/eclipse/jetty/aggregate/jetty-all/7.1.0.RC0/jetty-all-7.1.0.RC0-sources.jar!/org/eclipse/jetty/server/HttpOutput.java?format=ok -- which may or may not match exactly, but I doubt that this code changes all that often.] I would read this as Jetty thinking that this HTTP connection is closed. It this perhaps a case of your HTTP client disconnecting (or crashing) before Jetty can get the entire message (HTTP response) sent? (The other alternative that occurs to me would be that Solr told Jetty the response was all done, but then turned around and tried to send more in the response). -Original Message- From: Michael Szalay [mailto:michael.sza...@basis06.ch] Sent: Wednesday, September 14, 2011 1:47 AM To: solr-user@lucene.apache.org; JETTY user mailing list Subject: EofException with Solr in Jetty Hi all sometimes we have this error in our system. We are running Solr 3.1.0 running on Jetty 7.2.2 Anyone an idea how to tune this? 14:41:05,693 | ERROR | qtp283504850-36 | SolrDispatchFilter | apache.solr.common.SolrException 151 | 154 - mvn_ch.basis06.eld.indexer_ch.basis06.eld.indexer.solrserver_0.1-SNAPSHOT_war - 0 | org.eclipse.jetty.io.EofException at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:149) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:96) at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:184) at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89) at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:46) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:336) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:169) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473) at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:516) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:929) at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:116) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403) at org.eclipse.jetty.serve
Re: EofException with Solr in Jetty
We are using SolrJ 3.1 as our http client... So it may be a bug in there? Regards Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business - Ursprüngliche Mail - Von: "Jay Jaeger - DOT" An: solr-user@lucene.apache.org, "JETTY user mailing list" Gesendet: Mittwoch, 14. September 2011 15:21:19 Betreff: RE: EofException with Solr in Jetty Looking at the source for Jetty, line 149 in Jetty's HttpOutput java file looks like this: if (_closed) throw new IOException("Closed"); <<<<< [http://www.jarvana.com/jarvana/view/org/eclipse/jetty/aggregate/jetty-all/7.1.0.RC0/jetty-all-7.1.0.RC0-sources.jar!/org/eclipse/jetty/server/HttpOutput.java?format=ok -- which may or may not match exactly, but I doubt that this code changes all that often.] I would read this as Jetty thinking that this HTTP connection is closed. It this perhaps a case of your HTTP client disconnecting (or crashing) before Jetty can get the entire message (HTTP response) sent? (The other alternative that occurs to me would be that Solr told Jetty the response was all done, but then turned around and tried to send more in the response). -Original Message- From: Michael Szalay [mailto:michael.sza...@basis06.ch] Sent: Wednesday, September 14, 2011 1:47 AM To: solr-user@lucene.apache.org; JETTY user mailing list Subject: EofException with Solr in Jetty Hi all sometimes we have this error in our system. We are running Solr 3.1.0 running on Jetty 7.2.2 Anyone an idea how to tune this? 14:41:05,693 | ERROR | qtp283504850-36 | SolrDispatchFilter | apache.solr.common.SolrException 151 | 154 - mvn_ch.basis06.eld.indexer_ch.basis06.eld.indexer.solrserver_0.1-SNAPSHOT_war - 0 | org.eclipse.jetty.io.EofException at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:149) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:96) at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:184) at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89) at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:46) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:336) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:169) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473) at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:516) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:929) at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:116) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:184) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:864) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:72) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114) at org.eclipse.jetty.server.Server.handle(Server.java:352) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1051) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:590) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:212) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:508) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) at java.lang.Thread.run(Thread.java:662) -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business
EofException with Solr in Jetty
Hi all sometimes we have this error in our system. We are running Solr 3.1.0 running on Jetty 7.2.2 Anyone an idea how to tune this? 14:41:05,693 | ERROR | qtp283504850-36 | SolrDispatchFilter | apache.solr.common.SolrException 151 | 154 - mvn_ch.basis06.eld.indexer_ch.basis06.eld.indexer.solrserver_0.1-SNAPSHOT_war - 0 | org.eclipse.jetty.io.EofException at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:149) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:96) at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:184) at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89) at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:46) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:336) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:169) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473) at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:516) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:929) at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:116) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:184) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:864) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:72) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114) at org.eclipse.jetty.server.Server.handle(Server.java:352) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1051) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:590) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:212) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:508) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) at java.lang.Thread.run(Thread.java:662) -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business
Re: How to list all dynamic fields of a document using solrj?
You are right, they are not stored... But I is possible to see them, as the schema browser in the admin application does? Regards Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business - Ursprüngliche Mail - Von: "Erick Erickson" An: solr-user@lucene.apache.org Gesendet: Dienstag, 30. August 2011 16:31:05 Betreff: Re: How to list all dynamic fields of a document using solrj? This works for me, admittedly with the 3.3 code base: Hmmm, did you *store* the dynamic fields? only stored fields are returned CommonsHttpSolrServer server = new CommonsHttpSolrServer("http://localhost:8983/solr";); SolrQuery query = new SolrQuery(); query.setQuery("*"); query.setRows(1000); QueryResponse qr = server.query(query, SolrRequest.METHOD.POST); for (Object obj : qr.getHeader()) log(obj.toString()); SolrDocumentList sdl = qr.getResults(); for (SolrDocument d : sdl) { // Print out all the fields in the record. for (String key : d.getFieldNames()) { log(key + " : " + d.get(key).toString()); } // try a specific dynamic field Object val = d.getFieldValue("e1_t"); if (val != null) log("getting specific value: " + val.toString()); } On Tue, Aug 30, 2011 at 2:11 AM, Michael Szalay wrote: > Hi Juan > > I tried with the following code first: > > final SolrQuery allDocumentsQuery = new SolrQuery(); > allDocumentsQuery.setQuery("id:" + myId); > allDocumentsQuery.setFields("*"); > allDocumentsQuery.setRows(1); > QueryResponse response = solr.query(allDocumentsQuery, METHOD.POST); > > > With this, only non-dynamic fields are returned. > Then I wrote the following helper method: > > private Set getDynamicFields() throws SolrServerException, > IOException { > final LukeRequest luke = new LukeRequest(); > luke.setShowSchema(false); > final LukeResponse process = luke.process(solr); > final Map fieldInfo = process.getFieldInfo(); > final Set dynamicFields = new HashSet(); > for (final String key : fieldInfo.keySet()) { > if (key.endsWith("_string") || (key.endsWith("_dateTime"))) { > dynamicFields.add(key); > } > } > return dynamicFields; > } > > where as _string and _dateTime are the suffixes of my dynamic fields. > This one returns really all stored fields of the document: > > final Set dynamicFields = getDynamicFields(); > final SolrQuery allDocumentsQuery = new SolrQuery(); > allDocumentsQuery.setQuery("uri:" + myId); > allDocumentsQuery.setFields("*"); > for (final String df : dynamicFields) { > allDocumentsQuery.addField(df); > } > > allDocumentsQuery.setRows(1); > QueryResponse response = solr.query(allDocumentsQuery, METHOD.POST); > > Is there a more elegant way to do this? We are using solrj 3.1.0 and solr > 3.1.0. > > Regards > Michael > -- > Michael Szalay > Senior Software Engineer > > basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 > http://www.basis06.ch - source of smart business > > - Ursprüngliche Mail - > Von: "Juan Grande" > An: solr-user@lucene.apache.org > Gesendet: Montag, 29. August 2011 18:19:05 > Betreff: Re: How to list all dynamic fields of a document using solrj? > > Hi Michael, > > It's supposed to work. Can we see a snippet of the code you're using to > retrieve the fields? > > *Juan* > > > > On Mon, Aug 29, 2011 at 8:33 AM, Michael Szalay > wrote: > >> Hi all >> >> how can I list all dynamic fields and their values of a document using >> solrj? >> The dynamic fields are never returned when I use setFields(*). >> >> Thanks >> >> Michael >> >> -- >> Michael Szalay >> Senior Software Engineer >> >> basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 >> http://www.basis06.ch - source of smart business >> >> >
Re: How to list all dynamic fields of a document using solrj?
Hi Juan I tried with the following code first: final SolrQuery allDocumentsQuery = new SolrQuery(); allDocumentsQuery.setQuery("id:" + myId); allDocumentsQuery.setFields("*"); allDocumentsQuery.setRows(1); QueryResponse response = solr.query(allDocumentsQuery, METHOD.POST); With this, only non-dynamic fields are returned. Then I wrote the following helper method: private Set getDynamicFields() throws SolrServerException, IOException { final LukeRequest luke = new LukeRequest(); luke.setShowSchema(false); final LukeResponse process = luke.process(solr); final Map fieldInfo = process.getFieldInfo(); final Set dynamicFields = new HashSet(); for (final String key : fieldInfo.keySet()) { if (key.endsWith("_string") || (key.endsWith("_dateTime"))) { dynamicFields.add(key); } } return dynamicFields; } where as _string and _dateTime are the suffixes of my dynamic fields. This one returns really all stored fields of the document: final Set dynamicFields = getDynamicFields(); final SolrQuery allDocumentsQuery = new SolrQuery(); allDocumentsQuery.setQuery("uri:" + myId); allDocumentsQuery.setFields("*"); for (final String df : dynamicFields) { allDocumentsQuery.addField(df); } allDocumentsQuery.setRows(1); QueryResponse response = solr.query(allDocumentsQuery, METHOD.POST); Is there a more elegant way to do this? We are using solrj 3.1.0 and solr 3.1.0. Regards Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business - Ursprüngliche Mail - Von: "Juan Grande" An: solr-user@lucene.apache.org Gesendet: Montag, 29. August 2011 18:19:05 Betreff: Re: How to list all dynamic fields of a document using solrj? Hi Michael, It's supposed to work. Can we see a snippet of the code you're using to retrieve the fields? *Juan* On Mon, Aug 29, 2011 at 8:33 AM, Michael Szalay wrote: > Hi all > > how can I list all dynamic fields and their values of a document using > solrj? > The dynamic fields are never returned when I use setFields(*). > > Thanks > > Michael > > -- > Michael Szalay > Senior Software Engineer > > basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 > http://www.basis06.ch - source of smart business > >
Copying cores with solrj?
Hi all We have a setup with two cores, a "current" and a "work". The current is used by the application and the work is used by the updating program. I'm implementing a partial index updater. To do that, I want to copy the "current" core to the "work" so that I can have the current state to start with. I'm missing the "COPY"-Core admin request. How can I copy the index of the first core to the second one in a efficient manner? Regards Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business
Reusing SolrServer instances when swapping cores
Hi all We work with two cores ("active" and "passive") and swap them when the reindexing was finished. Is it allowed to reuse the same instance of the SolrServer (both Embedded and Common)? I.E. do they point to the "other" core after the swapping? Regards Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business