Hi, I need to do calculations that would benefit from storing information in distributed updateable cache. What are best practices for such things in hadoop?
PS In case there is no good solution for my problem, here are details and ideas I have. I'm going to count unique visitors of a site several times per day(each 5 mins), for that I will need distributed cache that will be accessible from all mappers to store already counted visitors. My plan is: store unique visitors in a file on hdfs each time mapper jvm starts store in HashSet in each jvm (I use mapred.job.reuse.jvm.num.tasks=-1) after each map/reduce job add additional data to this file any critics and advises are welcome :-) Regards, Vitaliy S
