Hey,
Thank you i used this method in the meantime for me and it worked fine.
Is there a general way to do the encoding to utf8 to this field in
Nutchg as well?
On 24.09.2012 12:04, Markus Jelsma wrote:
Hi Stefan,
You can take the stripNonCharCodepoints() method and pass your content through
it. It should fix the problem.
Cheers,
-----Original message-----
From:Stefan Scheffler <[email protected]>
Sent: Mon 24-Sep-2012 11:23
To: [email protected]
Subject: Re: Indexing Exception
Hey Markus. you gave me the right hint.
Additionally to the normally content field i added a field fullcontent,
which simply holds the html document of the relevant content field,
because we need this in a later proccessing step. This field is not
encoded like the content field. I realised this with an own
ParsingFilter, which stores it in to the ParseResult and then an
Indexingfilter merges it into the NutchDocument.
Is there a way to do this better or just do the encoding to the
fullcontent like to the content?
Regards
Stefan
On 24.09.2012 10:41, Markus Jelsma wrote:
It was fixed for the content field with 1016. Can you pinpoint the problematic
field?
https://issues.apache.org/jira/browse/NUTCH-1016
-----Original message-----
From:Stefan Scheffler <[email protected]>
Sent: Mon 24-Sep-2012 10:37
To: [email protected]
Subject: Re: Indexing Exception
nutch 1.5, solr 3.6
On 24.09.2012 10:34, Markus Jelsma wrote:
Hi - What version?
-----Original message-----
From:Stefan Scheffler <[email protected]>
Sent: Mon 24-Sep-2012 10:29
To: [email protected]
Subject: Indexing Exception
Hello,
I have a strange Problem. While indexing a crawl to solr i got the
following exception
java.lang.RuntimeException: [was class java.io.CharConversionException]
Invalid UTF-8 character 0xfffe at char #6886708, byte #11578429)
at
com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
at
com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
at
com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
at
com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:315)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:156)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:843)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.io.CharConversionException: Invalid UTF-8 character
0xfffe at char #6886708, byte #11578429)
...
It seems to be an encoding exception. Is there a way to avoid this?
Regards
Stefan
--
Stefan Scheffler
Avantgarde Labs GmbH
Löbauer Straße 19, 01099 Dresden
Telefon: + 49 (0) 351 21590834
Email: [email protected]
--
Stefan Scheffler
Avantgarde Labs GmbH
Löbauer Straße 19, 01099 Dresden
Telefon: + 49 (0) 351 21590834
Email: [email protected]
--
Stefan Scheffler
Avantgarde Labs GmbH
Löbauer Straße 19, 01099 Dresden
Telefon: + 49 (0) 351 21590834
Email: [email protected]
--
Stefan Scheffler
Avantgarde Labs GmbH
Löbauer Straße 19, 01099 Dresden
Telefon: + 49 (0) 351 21590834
Email: [email protected]
