Thanks Timothy,
Will give the DIH a try. I have submitted a bug report.
Regards,
Simon
On 31/05/16 13:22, Allison, Timothy B. wrote:
From the same page, extractFormat=text only applies when extractOnly
is true, which just shows the output from tika without indexing the document.
Y, sorry. I just looked through the source code. You're right. If you use DIH
(TikaEntityProcessor) instead of Solr Cell (ExtractingDocumentLoader), you should be able to set
the handler type by setting the "format" attribute, and "text" is one option
there.
I just want to make sure I'm not missing something really obvious before
submitting a bug report.
I don't think you are.
From the same page, extractFormat=text only applies when extractOnly
is true, which just shows the output from tika without indexing the document.
Running it in "extractOnly" mode resulting in a XML output. The
difference between selecting "text" or "xml" format is that the
escaped document in the <response> tag is either the original HTML
(xml mode) or stripped HTML (text mode). It seems some Javascript
creeps into the text version. (See below)
Regards,
Simon
HTML mode sample:
<?xml version="1.0" encoding="UTF-8"?> <response> <lst
name="responseHeader"><int name="status">0</int><int
name="QTime">51</int></lst><str
name="UsingMailingLists.html"><?xml
version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<link
rel="stylesheet" type="text/css" charset="utf-8" media="all"
href="/wiki/modernized/css/common.css"/>
<link rel="stylesheet" type="text/css" charset="utf-8"
media="screen" href="/wiki/modernized/css/screen.css"/>
<link rel="stylesheet" type="text/css" charset="utf-8"
media="print" href="/wiki/modernized/css/print.css"/>.......
TEXT mode (Blank lines stripped):
<response>
<lst name="responseHeader"><int name="status">0</int><int
name="QTime">47</int></lst><str name="UsingMailingLists.html">
UsingMailingLists - Solr Wiki
Search:
<!--// Initialize search form
var f = document.getElementById('searchform');
f.getElementsByTagName('label')[0].style.display = 'none'; var e =
document.getElementById('searchinput');
searchChange(e);
searchBlur(e);
//-->
Solr Wiki
Login
On 27/05/16 13:31, Allison, Timothy B. wrote:
I'm only minimally familiar with Solr Cell, but...
1) It looks like you aren't setting extractFormat=text. According
to [0]...the default is xhtml which will include a bunch of the metadata.
2) is there an attr_* dynamic field in your index with type="ignored"?
This would strip out the attr_ fields so they wouldn't even be
indexed...if you don't want them.
As for the HTML file, it looks like Tika is failing to strip out the
style section. Try running the file alone with tika-app: java -jar
tika-app.jar -t inputfile.html. If you are finding the noise there.
Please open an issue on our JIRA:
https://issues.apache.org/jira/browse/tika
[0]
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with
+Solr+Cell+using+Apache+Tika
-----Original Message-----
From: Simon Blandford [mailto:simon.blandf...@bkconnect.net]
Sent: Thursday, May 26, 2016 9:49 AM
To: solr-user@lucene.apache.org
Subject: Metadata and HTML ending up in searchable text
Hi,
I am using Solr 6.0 on Ubuntu 14.04.
I am ending up with loads of junk in the text body. It starts like,
The JSON entry output of a search result shows the indexed text
starting with...
body_txt_en: " stream_size 36499 X-Parsed-By
org.apache.tika.parser.DefaultParser X-Parsed-By...."
And then once it gets to the actual text I get CSS class names
appearing that were in <p> or <div> tags etc.
e.g. "....the power of calibre3 silence calibre2 and....", where
"calibre3" etc are the CSS class names.
All this junk is searchable and is polluting the index.
I would like to index _only_ the actual content I am interested in
searching for.
Steps to reproduce:
1) Solr installed by untaring solr tgz in /opt.
2) Core created by typing "bin/solr create -c mycore"
3) Solr started with bin/solr start
4) TXT document index using the following command curl
"http://localhost:8983/solr/mycore/update/extract?literal.id=doc1&uprefix=attr_&fmap.content=body_txt_en&commit=true";
-F
"content/UsingMailingLists.txt=@/home/user/Documents/library/UsingMailingLists.txt"
5) HTML document index using following command curl
"http://localhost:8983/solr/mycore/update/extract?literal.id=doc2&uprefix=attr_&fmap.content=body_txt_en&commit=true";
-F
"content/UsingMailingLists.html=@/home/user/Documents/library/UsingMailingLists.html"
6) Query using URL:
http://localhost:8983/solr/mycore/select?q=especially&wt=json
Result:
For the txt file, I get the following JSON for the document...
{
id: "doc1",
attr_stream_size: [
"8107"
],
attr_x_parsed_by: [
"org.apache.tika.parser.DefaultParser",
"org.apache.tika.parser.txt.TXTParser"
],
attr_stream_content_type: [
"text/plain"
],
attr_stream_name: [
"UsingMailingLists.txt"
],
attr_stream_source_info: [
"content/UsingMailingLists.txt"
],
attr_content_encoding: [
"ISO-8859-1"
],
attr_content_type: [
"text/plain; charset=ISO-8859-1"
],
body_txt_en: " stream_size 8107 X-Parsed-By
org.apache.tika.parser.DefaultParser X-Parsed-By
org.apache.tika.parser.txt.TXTParser stream_content_type text/plain
stream_name UsingMailingLists.txt stream_source_info
content/UsingMailingLists.txt Content-Encoding ISO-8859-1
Content-Type text/plain; charset=ISO-8859-1 Search: [value ]
[Titles] [Text] Solr_Wiki Login ****** UsingMailingLists ****** *
FrontPage * RecentChanges...etc",
_version_: 1535398235801124900
}
For the HTML file, I get the following JSON for the document...
{
id: "doc2",
attr_stream_size: [
"20440"
],
attr_x_parsed_by: [
"org.apache.tika.parser.DefaultParser",
"org.apache.tika.parser.html.HtmlParser"
],
attr_stream_content_type: [
"text/html"
],
attr_stream_name: [
"UsingMailingLists.html"
],
attr_stream_source_info: [
"content/UsingMailingLists.html"
],
attr_dc_title: [
"UsingMailingLists - Solr Wiki"
],
attr_content_encoding: [
"UTF-8"
],
attr_robots: [
"index,nofollow"
],
attr_title: [
"UsingMailingLists - Solr Wiki"
],
attr_content_type: [
"text/html; charset=utf-8"
],
body_txt_en: " stylesheet text/css utf-8 all
/wiki/modernized/css/common.css stylesheet text/css utf-8 screen
/wiki/modernized/css/screen.css stylesheet text/css utf-8 print
/wiki/modernized/css/print.css stylesheet text/css utf-8 projection
/wiki/modernized/css/projection.css alternate Solr Wiki:
UsingMailingLists
/solr/UsingMailingLists?diffs=1&show_att=1&action=rss_rc&unique=0&pa
ge=UsingMailingLists&ddiffs=1 application/rss+xml Start
/solr/FrontPage Alternate Wiki Markup
/solr/UsingMailingLists?action=raw Alternate print Print View
/solr/UsingMailingLists?action=print Search /solr/FindPage Index
/solr/TitleIndex Glossary /solr/WordIndex Help
/solr/HelpOnFormatting stream_size 20440 X-Parsed-By
org.apache.tika.parser.DefaultParser
X-Parsed-By org.apache.tika.parser.html.HtmlParser
stream_content_type text/html stream_name UsingMailingLists.html
stream_source_info...etc",
_version_: 1535398408383103000 }
