EofException with Solr in Jetty
Hi all sometimes we have this error in our system. We are running Solr 3.1.0 running on Jetty 7.2.2 Anyone an idea how to tune this? 14:41:05,693 | ERROR | qtp283504850-36 | SolrDispatchFilter | apache.solr.common.SolrException 151 | 154 - mvn_ch.basis06.eld.indexer_ch.basis06.eld.indexer.solrserver_0.1-SNAPSHOT_war - 0 | org.eclipse.jetty.io.EofException at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:149) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:96) at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:184) at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89) at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:46) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:336) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:169) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473) at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:516) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:929) at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:116) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:184) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:864) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:72) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114) at org.eclipse.jetty.server.Server.handle(Server.java:352) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1051) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:590) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:212) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:508) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) at java.lang.Thread.run(Thread.java:662) -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business
Re: EofException with Solr in Jetty
We are using SolrJ 3.1 as our http client... So it may be a bug in there? Regards Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business - Ursprüngliche Mail - Von: Jay Jaeger - DOT jay.jae...@dot.wi.gov An: solr-user@lucene.apache.org, JETTY user mailing list jetty-us...@eclipse.org Gesendet: Mittwoch, 14. September 2011 15:21:19 Betreff: RE: EofException with Solr in Jetty Looking at the source for Jetty, line 149 in Jetty's HttpOutput java file looks like this: if (_closed) throw new IOException(Closed); [http://www.jarvana.com/jarvana/view/org/eclipse/jetty/aggregate/jetty-all/7.1.0.RC0/jetty-all-7.1.0.RC0-sources.jar!/org/eclipse/jetty/server/HttpOutput.java?format=ok -- which may or may not match exactly, but I doubt that this code changes all that often.] I would read this as Jetty thinking that this HTTP connection is closed. It this perhaps a case of your HTTP client disconnecting (or crashing) before Jetty can get the entire message (HTTP response) sent? (The other alternative that occurs to me would be that Solr told Jetty the response was all done, but then turned around and tried to send more in the response). -Original Message- From: Michael Szalay [mailto:michael.sza...@basis06.ch] Sent: Wednesday, September 14, 2011 1:47 AM To: solr-user@lucene.apache.org; JETTY user mailing list Subject: EofException with Solr in Jetty Hi all sometimes we have this error in our system. We are running Solr 3.1.0 running on Jetty 7.2.2 Anyone an idea how to tune this? 14:41:05,693 | ERROR | qtp283504850-36 | SolrDispatchFilter | apache.solr.common.SolrException 151 | 154 - mvn_ch.basis06.eld.indexer_ch.basis06.eld.indexer.solrserver_0.1-SNAPSHOT_war - 0 | org.eclipse.jetty.io.EofException at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:149) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:96) at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:184) at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89) at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:46) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:336) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:169) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473) at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:516) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:929) at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:116) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:184) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:864) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:72) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114) at org.eclipse.jetty.server.Server.handle(Server.java:352) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1051) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:590) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:212) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:508) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) at java.lang.Thread.run(Thread.java:662) -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business
Re: EofException with Solr in Jetty
There is nothing between the client app and the solr server, its on the same machine and on the same app server, only going through the loopback interface. Unfortunatly, I cannot reproduce it, but I see it in the server log. Thanks Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business - Ursprüngliche Mail - Von: Jay Jaeger - DOT jay.jae...@dot.wi.gov An: solr-user@lucene.apache.org Gesendet: Mittwoch, 14. September 2011 16:23:45 Betreff: RE: EofException with Solr in Jetty I have not used SolrJ, but it probably is worth considering as a possible suspect. Also, do you have anything in between the client and the Solr server (a firewall, load balancer, etc.?) that might play games with HTTP connections? You might want to start up a network trace on the server or network to see if you can catch one to see what is going on. I looked at our Solr 3.1 prototype log (which has been running continuously without interruption since July 10!), and did not see any of these errors. We do not use SolrJ -- we use a combination of plain old HTTP/javascript/xslt and requests coming from another system as a (plain old XML) web service to get to Solr. However, that is under Jetty 6. JRJ -Original Message- From: Michael Szalay [mailto:michael.sza...@basis06.ch] Sent: Wednesday, September 14, 2011 8:27 AM To: solr-user@lucene.apache.org Subject: Re: EofException with Solr in Jetty We are using SolrJ 3.1 as our http client... So it may be a bug in there? Regards Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business - Ursprüngliche Mail - Von: Jay Jaeger - DOT jay.jae...@dot.wi.gov An: solr-user@lucene.apache.org, JETTY user mailing list jetty-us...@eclipse.org Gesendet: Mittwoch, 14. September 2011 15:21:19 Betreff: RE: EofException with Solr in Jetty Looking at the source for Jetty, line 149 in Jetty's HttpOutput java file looks like this: if (_closed) throw new IOException(Closed); [http://www.jarvana.com/jarvana/view/org/eclipse/jetty/aggregate/jetty-all/7.1.0.RC0/jetty-all-7.1.0.RC0-sources.jar!/org/eclipse/jetty/server/HttpOutput.java?format=ok -- which may or may not match exactly, but I doubt that this code changes all that often.] I would read this as Jetty thinking that this HTTP connection is closed. It this perhaps a case of your HTTP client disconnecting (or crashing) before Jetty can get the entire message (HTTP response) sent? (The other alternative that occurs to me would be that Solr told Jetty the response was all done, but then turned around and tried to send more in the response). -Original Message- From: Michael Szalay [mailto:michael.sza...@basis06.ch] Sent: Wednesday, September 14, 2011 1:47 AM To: solr-user@lucene.apache.org; JETTY user mailing list Subject: EofException with Solr in Jetty Hi all sometimes we have this error in our system. We are running Solr 3.1.0 running on Jetty 7.2.2 Anyone an idea how to tune this? 14:41:05,693 | ERROR | qtp283504850-36 | SolrDispatchFilter | apache.solr.common.SolrException 151 | 154 - mvn_ch.basis06.eld.indexer_ch.basis06.eld.indexer.solrserver_0.1-SNAPSHOT_war - 0 | org.eclipse.jetty.io.EofException at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:149) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:96) at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:184) at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89) at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:46) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:336) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:169) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473) at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:516) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:929) at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:116) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403
Re: How to list all dynamic fields of a document using solrj?
You are right, they are not stored...
But I is possible to see them, as the schema browser in the admin application
does?
Regards Michael
--
Michael Szalay
Senior Software Engineer
basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22
http://www.basis06.ch - source of smart business
- Ursprüngliche Mail -
Von: Erick Erickson erickerick...@gmail.com
An: solr-user@lucene.apache.org
Gesendet: Dienstag, 30. August 2011 16:31:05
Betreff: Re: How to list all dynamic fields of a document using solrj?
This works for me, admittedly with the 3.3 code base:
Hmmm, did you *store* the dynamic fields? only stored fields are returned
CommonsHttpSolrServer server = new
CommonsHttpSolrServer(http://localhost:8983/solr;);
SolrQuery query = new SolrQuery();
query.setQuery(*);
query.setRows(1000);
QueryResponse qr = server.query(query, SolrRequest.METHOD.POST);
for (Object obj : qr.getHeader()) log(obj.toString());
SolrDocumentList sdl = qr.getResults();
for (SolrDocument d : sdl) {
// Print out all the fields in the record.
for (String key : d.getFieldNames()) {
log(key + : + d.get(key).toString());
}
// try a specific dynamic field
Object val = d.getFieldValue(e1_t);
if (val != null)
log(getting specific value: + val.toString());
}
On Tue, Aug 30, 2011 at 2:11 AM, Michael Szalay
michael.sza...@basis06.ch wrote:
Hi Juan
I tried with the following code first:
final SolrQuery allDocumentsQuery = new SolrQuery();
allDocumentsQuery.setQuery(id: + myId);
allDocumentsQuery.setFields(*);
allDocumentsQuery.setRows(1);
QueryResponse response = solr.query(allDocumentsQuery, METHOD.POST);
With this, only non-dynamic fields are returned.
Then I wrote the following helper method:
private SetString getDynamicFields() throws SolrServerException,
IOException {
final LukeRequest luke = new LukeRequest();
luke.setShowSchema(false);
final LukeResponse process = luke.process(solr);
final MapString, FieldInfo fieldInfo = process.getFieldInfo();
final SetString dynamicFields = new HashSetString();
for (final String key : fieldInfo.keySet()) {
if (key.endsWith(_string) || (key.endsWith(_dateTime))) {
dynamicFields.add(key);
}
}
return dynamicFields;
}
where as _string and _dateTime are the suffixes of my dynamic fields.
This one returns really all stored fields of the document:
final SetString dynamicFields = getDynamicFields();
final SolrQuery allDocumentsQuery = new SolrQuery();
allDocumentsQuery.setQuery(uri: + myId);
allDocumentsQuery.setFields(*);
for (final String df : dynamicFields) {
allDocumentsQuery.addField(df);
}
allDocumentsQuery.setRows(1);
QueryResponse response = solr.query(allDocumentsQuery, METHOD.POST);
Is there a more elegant way to do this? We are using solrj 3.1.0 and solr
3.1.0.
Regards
Michael
--
Michael Szalay
Senior Software Engineer
basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22
http://www.basis06.ch - source of smart business
- Ursprüngliche Mail -
Von: Juan Grande juan.gra...@gmail.com
An: solr-user@lucene.apache.org
Gesendet: Montag, 29. August 2011 18:19:05
Betreff: Re: How to list all dynamic fields of a document using solrj?
Hi Michael,
It's supposed to work. Can we see a snippet of the code you're using to
retrieve the fields?
*Juan*
On Mon, Aug 29, 2011 at 8:33 AM, Michael Szalay
michael.sza...@basis06.chwrote:
Hi all
how can I list all dynamic fields and their values of a document using
solrj?
The dynamic fields are never returned when I use setFields(*).
Thanks
Michael
--
Michael Szalay
Senior Software Engineer
basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22
http://www.basis06.ch - source of smart business
Re: How to list all dynamic fields of a document using solrj?
Hi Juan
I tried with the following code first:
final SolrQuery allDocumentsQuery = new SolrQuery();
allDocumentsQuery.setQuery(id: + myId);
allDocumentsQuery.setFields(*);
allDocumentsQuery.setRows(1);
QueryResponse response = solr.query(allDocumentsQuery, METHOD.POST);
With this, only non-dynamic fields are returned.
Then I wrote the following helper method:
private SetString getDynamicFields() throws SolrServerException, IOException
{
final LukeRequest luke = new LukeRequest();
luke.setShowSchema(false);
final LukeResponse process = luke.process(solr);
final MapString, FieldInfo fieldInfo = process.getFieldInfo();
final SetString dynamicFields = new HashSetString();
for (final String key : fieldInfo.keySet()) {
if (key.endsWith(_string) || (key.endsWith(_dateTime))) {
dynamicFields.add(key);
}
}
return dynamicFields;
}
where as _string and _dateTime are the suffixes of my dynamic fields.
This one returns really all stored fields of the document:
final SetString dynamicFields = getDynamicFields();
final SolrQuery allDocumentsQuery = new SolrQuery();
allDocumentsQuery.setQuery(uri: + myId);
allDocumentsQuery.setFields(*);
for (final String df : dynamicFields) {
allDocumentsQuery.addField(df);
}
allDocumentsQuery.setRows(1);
QueryResponse response = solr.query(allDocumentsQuery, METHOD.POST);
Is there a more elegant way to do this? We are using solrj 3.1.0 and solr 3.1.0.
Regards
Michael
--
Michael Szalay
Senior Software Engineer
basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22
http://www.basis06.ch - source of smart business
- Ursprüngliche Mail -
Von: Juan Grande juan.gra...@gmail.com
An: solr-user@lucene.apache.org
Gesendet: Montag, 29. August 2011 18:19:05
Betreff: Re: How to list all dynamic fields of a document using solrj?
Hi Michael,
It's supposed to work. Can we see a snippet of the code you're using to
retrieve the fields?
*Juan*
On Mon, Aug 29, 2011 at 8:33 AM, Michael Szalay
michael.sza...@basis06.chwrote:
Hi all
how can I list all dynamic fields and their values of a document using
solrj?
The dynamic fields are never returned when I use setFields(*).
Thanks
Michael
--
Michael Szalay
Senior Software Engineer
basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22
http://www.basis06.ch - source of smart business
Copying cores with solrj?
Hi all We have a setup with two cores, a current and a work. The current is used by the application and the work is used by the updating program. I'm implementing a partial index updater. To do that, I want to copy the current core to the work so that I can have the current state to start with. I'm missing the COPY-Core admin request. How can I copy the index of the first core to the second one in a efficient manner? Regards Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business
Reusing SolrServer instances when swapping cores
Hi all We work with two cores (active and passive) and swap them when the reindexing was finished. Is it allowed to reuse the same instance of the SolrServer (both Embedded and Common)? I.E. do they point to the other core after the swapping? Regards Michael -- Michael Szalay Senior Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business
