Am 06.03.2012 14:42, schrieb Jochen Topf:
On Tue, Mar 06, 2012 at 01:31:31PM +0100, npl wrote:
TagInfo-like systems (aggregating big data and creating statistics)
could definitely be built using hadoop/mapreduce.
Or you can do what Taginfo does and just write it cleverly, so it just uses one
host for an hour instead of 10 hosts for several hours. :-) I do agree that
there are many use-cases for Hadoop& Co. But they also create a lot of
overhead...
You're absolutely right - hadoop(java)/pig/... creates a lot of overhead.
In our case we had some join+sort operations which slowed things
considerably down (these operations don't utilize the cluster well - I
haven't figured it out yet why this is the case) and the upload to HBase
was also very slow.
- npl
_______________________________________________
dev mailing list
[email protected]
http://lists.openstreetmap.org/listinfo/dev
