To follow up, I've found that my queries are very fast (even
with &fq=),
until I add &hl=true. What can I do to speed up highlighting?
Should I
consider injecting a line at a time, rather than the entire
file as a field?
-Pete
On Jun 29, 2010, at 11:07 AM, Peter Spam wrote:
Thanks for everyone's help - I have this working now, but
sometimes the
queries are incredibly slow!! For example, <int
name="QTime">461360</int>.
Also, I had to bump up the min/max RAM size to 1GB/3.5GB for
things to
inject without throwing heap memory errors. However, my data
set is very
small! 36 text files, for a total of 113MB. (It will grow to
many TB, but
for now, this is a test). The largest file is 34MB.
Therefore, I'm sure I'm doing something wrong :-) Here's my
config:
-----------------------------------------------------------------------------------------------
For the schema.xml, <types> is all default. For fields, here
are the
only lines that aren't commented out:
<field name="id" type="string" indexed="true" stored="true"
required="true" />
<field name="body" type="text" indexed="true" stored="true"
multiValued="true"/>
<field name="timestamp" type="date" indexed="true" stored="true"
default="NOW" multiValued="false"/>
<field name="build" type="string" indexed="true" stored="true"
multiValued="false"/>
<field name="device" type="string" indexed="true" stored="true"
multiValued="false"/>
<dynamicField name="*" type="ignored" multiValued="true" />
... then, for the rest:
<uniqueKey>id</uniqueKey>
<!-- field for the QueryParser to use when an explicit
fieldname is
absent -->
<defaultSearchField>body</defaultSearchField>
<!-- SolrQueryParser configuration: defaultOperator="AND|OR" -->
<solrQueryParser defaultOperator="AND"/>
-----------------------------------------------------------------------------------------------
Invoking: java -Xmx3584M -Xms1024M -jar start.jar
-----------------------------------------------------------------------------------------------
Injecting:
#!/bin/sh
J=0
for i in `find . -name \*.txt`; do
(( J++ ))
curl "
http://localhost:8983/solr/update/extract?literal.id=doc$J&fmap.content=body
"
-F "myfi...@$i";
done;
echo "------------- Committing"
curl "http://localhost:8983/solr/update/extract?commit=true";
-----------------------------------------------------------------------------------------------
Searching:
http://localhost:8983/solr/select?q=testing&hl=true&fl=id,score&hl.snippets=5&hl.mergeContiguous=true
-Pete
On Jun 28, 2010, at 5:22 PM, Erick Erickson wrote:
try adding &hl.fl=text
to specify your highlight field. I don't understand why
you're only
getting the ID field back though. Do note that the highlighting
is after the docs, related by the ID.
Try a (non highlighting) query of just * to verify that you're
pointing at the index you think you are. It's possible that
you've modified a different index with SolrJ than your web
server is pointing at.
Also, SOLR has no way of knowing you're modified your index
with SolrJ, so it may not be automatically reopening an
IndexReader so your recent changes may not be visible
until you force the SOLR reader to reopen.
HTH
Erick
On Mon, Jun 28, 2010 at 6:49 PM, Peter Spam <ps...@mac.com>
wrote:
On Jun 28, 2010, at 2:00 PM, Ahmet Arslan wrote:
1) I can get my docs in the index, but when I search, it
returns the entire document. I'd love to have it only
return the line (or two) around the search term.
Solr can generate Google-like snippets as you describe.
http://wiki.apache.org/solr/HighlightingParameters
Here's how I commit my documents:
J=0;
for i in `find . -name \*.txt`; do
(( J++ ))
curl "http://localhost:8983/solr/update/extract?literal.id=doc$J
"
-F "myfi...@$i";
done;
echo "------------- Committing"
curl "http://localhost:8983/solr/update/extract?commit=true";
Then, I try to query using
http://localhost:8983/solr/select?rows=10&start=0&fl=*,score&hl=true&q=testing
but I only get back the document ID rather than the snippet:
<doc>
<float name="score">0.05030759</float>
<arr name="content_type">
<str>text/plain</str>
</arr>
<str name="id">doc16</str>
</doc>
I'm using the schema.xml from the "lucid imagination:
Indexing text and
html files" tutorial.
-Pete
