Harald, I assume you're referring to http://www.raytion.com/products/enterprise-search-connectors.html What about scalability of this solution - would it support crawling and indexing 100s of millions of files within reasonable amount of time? Where would the index be stored? Would it allow us to define parsers for various file formats - similar to what might be possible with Tika?
Thanks, Gregory -----Original Message----- From: Harald Kirsch [mailto:[email protected]] Sent: Monday, April 28, 2014 14:15 To: [email protected] Subject: Re: Nutch for NFS crawling and data indexing The strength of web crawlers is their ability to follow links in websites to explore more and more of the landscape to index. When you want to index fileshares, like NFS, potentially with access rights included, you way want to use a different beast, (we call it a connector). The benefit is, that the connector can just follow the directory structure. There is no need to figure out new documents by parsing the documents found so far. Harald. On 28.04.2014 12:54, Touretsky, Gregory wrote: > Hi, > > I see multiple references to Web search implementation with Nutch. > Have anyone implemented large scale (many TBs of data, millions of files) NFS > search? > Are there any alternative solutions for scale-out crawling and indexing of > file systems? > > Thank you, > Gregory > --------------------------------------------------------------------- > Intel Israel (74) Limited > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > -- Harald Kirsch Raytion GmbH Kaiser-Friedrich-Ring 74 40547 Duesseldorf Fon +49-211-550266-0 Fax +49-211-550266-19 http://www.raytion.com --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
