Yes, the ExtractingRequestHandler uses Tika to parse many file formats.
Solr 1.4.1 uses a previous version of Tika (0.6 or 0.7).
Here's the problem with Tika and extraction utilities in general: they
are not perfect. They will fail on some files. In the
ExtractingRequestHandler's case, there is no way to let it fail in
parsing but save the document's metadata anyway with a notation: "sorry
not parsed". I would rather have the unix 'strings' command parse my
documents (thanks to a co-worker for this).
K. Seshadri Iyer wrote:
Thanks for all the responses.
Govind: To answer your question, yes, all I want to search is plain text
files. They are located in NFS directories across multiple Solaris/Linux
storage boxes. The total storage is in hundreds of terabytes.
I have just got started with Solr and my understanding is that I will
somehow need Tika to help stream/upload files to Solr. I don't know anything
about Java programming, being a system admin. So far, I have read that the
autodetect parser in Tika will somehow detect the file type and I can use
the stream to populate Solr. How, that is still a mystery to me - working on
it. Any tips appreciated; thanks in advance.
Sesh
On 13 November 2010 15:24, Govind Kanshi<govind.kan...@gmail.com> wrote:
Another pov you might want to think about - what kind of search you want.
Just plain - full text search or there is something more to those text
files. Are they grouped in folders? Do the folders imply certain kind of
grouping/hierarchy/tagging?
I recently was trying to help somebody who had files across lot of places
grouped by date/subject/author - he wanted to ensure these are "fields"
which too can act as filters/navigators.
Just an input - ignore it if you just want plain full text search.
On Sat, Nov 13, 2010 at 11:25 AM, Lance Norskog<goks...@gmail.com> wrote:
About web servers: Solr is a servlet war file and needs a Java web server
"container" to run. The example/ folder in the Solr disribution uses
'Jetty', and this is fine for small production-quality projects. You can
just copy the example/ directory somewhere to set up your own running
Solr;
that's what I always do.
About indexing programs: if you know Unix scripting, it may be easiest to
walk the file system yourself with the 'find' program and create Solr
input
XML files.
But yes, you definitely want the Solr 1.4 Enterprise manual. I spent
months
learning this stuff very slowly, and the book would have been great back
then.
Lance
Erick Erickson wrote:
Think of the data import handler (DIH) as Solr pulling data to index
from some source based on configuration. So, once you set up
your DIH config to point to your file system, you issue a command
to solr like "OK, do your data import thing". See the
FileListEntityProcessor.
http://wiki.apache.org/solr/DataImportHandler
<http://wiki.apache.org/solr/DataImportHandler>SolrJ is a clent library
you'd use to push data to Solr. Basically, you
write a Java program that uses SolrJ to walk the file system, find
documents, create a Solr document and sent that to Solr. It's not
nearly as complex as it sounds<G>. See:
http://wiki.apache.org/solr/Solrj
<http://wiki.apache.org/solr/Solrj>It's probably worth your while to
get
a
copy of "Solr 1.4, Enterprise Search Server"
by Erik Pugh and David Smiley.
Best
Erick
On Fri, Nov 12, 2010 at 8:37 AM, K. Seshadri Iyer<seshadri...@gmail.com
wrote:
Hi Lance,
Thank you very much for responding (not sure how I reply to the group,
so,
writing to you).
Can you please expand on your suggestion? I am not a web guy and so,
don't
know where to start.
What is the difference between SolrJ and DataImportHandler? Do I need
to
set
up web servers on all my storage boxes?
Apologies for the basic level of questions, but hope I can get started
and
implement this before the year end (you know why :o)
Thanks,
Sesh
On 12 November 2010 13:31, Lance Norskog<goks...@gmail.com> wrote:
Using 'curl' is fine. There is a library called SolrJ for Java and
other libraries for other scripting languages that let you upload with
more control. There is a thing in Solr called the DataImportHandler
that lets you script walking a file system.
On Thu, Nov 11, 2010 at 8:38 PM, K. Seshadri Iyer<
seshadri...@gmail.com
wrote:
Hi,
Pardon me if this sounds very elementary, but I have a very basic
question
regarding Solr search. I have about 10 storage devices running
Solaris
with
hundreds of thousands of text files (there are other files, as well,
but
my
target is these text files). The directories on the Solaris boxes are
exported and are available as NFS mounts.
I have installed Solr 1.4 on a Linux box and have tested the
installation,
using curl to post documents. However, the manual says that curl is
not
the
recommended way of posting documents to Solr. Could someone please
tell
me
what is the preferred approach in such an environment? I am not a
programmer
and would appreciate some hand-holding here :o)
Thanks in advance,
Sesh
--
Lance Norskog
goks...@gmail.com
