Maybe memcahced can help http://www.danga.com/memcached/
-----Original Message----- From: Arun C Murthy [mailto:[EMAIL PROTECTED] Sent: Wed 4/30/2008 12:35 PM To: [email protected] Subject: Re: DistributedCache on Java Objects? On Apr 30, 2008, at 8:14 AM, ncardoso wrote: > > Hello, > > I'm using Hadoop for distributed text mining of large collection of > documents, and in my optimizing process, I want to speed things up > a bit, > and I want to know how can I do this step with Hadoop... > > Each Map process takes a group of documents, analyses each > sentence, and for > certain patterns it queries a database and some indexes to provide > a proper > answer. This step can take a while, so I'm caching the results in a > LinkedHashMap, which works pretty well for standalone jobs, and avoids > repeated queries for same patterns in a documet. > > I think it would be great to share this LinkedHashMap cache object > for all > Map instances, so that if the #2 Map object finds the same pattern > as the #1 > Map object previously noticed on other document, it can use the cached > result that #1 Map placed there for all Map objects, saving some time. > > Right now, the DistributedCache just shares files, archives and jar > files. > Is there any way to share such a Java object such as a LinkedHashMap, > synchronized or not? > Not directly. You could serialize your LinkedHashMap into a file via something like java.io.DataOutput, store it on HDFS and then deserialize it via java.io.DataInput in your map's configure method. Does that work? Arun > -- > View this message in context: http://www.nabble.com/ > DistributedCache-on-Java-Objects--tp16985074p16985074.html > Sent from the Hadoop lucene-users mailing list archive at Nabble.com. >
