Re-balancing blocks

2009-04-11 Thread Stas Oskin
Hi. Should I call the block re-balancer every time a new DataNode is added or removed? Or what is the recommended procedure of re-balancing the blocks for better fault tolerance? Regards.

Re: Can we somehow read from the HDFS without converting it to local?

2009-04-11 Thread nitesh bhatia
Hi You can perhaps mount the hdfs to linux fs and then read directly. Check this link : http://wiki.apache.org/hadoop/MountableHDFS -nitesh On Fri, Apr 10, 2009 at 5:19 PM, Stuart White stuart.whi...@gmail.comwrote: Not sure if this is what you're looking for...

NameNode resilency

2009-04-11 Thread Stas Oskin
Hi. I wonder, what Hadoop community uses in order to make NameNode resilient to failures? I mean, what High-Availability measures are taken to keep the HDFS available even in case of NameNode failure? So far I read a possible solution using DRBD, and another one using carp. Both of them had the

RE: More Replication on dfs

2009-04-11 Thread Puri, Aseem
Alex, Ouput of $ bin/hadoop fsck / command after running HBase data insert command in a table is: . . . . . /hbase/test/903188508/tags/info/4897652949308499876: Under replicated blk_-5193 695109439554521_3133. Target Replicas is 3 but found 1 replica(s). .

auto-increment in hadoop streaming using partitioning

2009-04-11 Thread Michael Klatt
We use Hadoop Streaming and needed to assign unique ids to rows/records. I wrote some code in Python to do this and thought I'd share it. This works in both the map reduce phase. This was inspired by how MySQL handles auto-increment in a master-master setup (with auto_increment_increment

Re: NameNode resilency

2009-04-11 Thread Billy Pearson
Not 100% sure but I thank they plan on using zookeeper to help with namenode fail over but that may have changed. Billy Stas Oskin stas.os...@gmail.com wrote in message news:77938bc20904110243u7a2baa6dw6d710e4e51ae0...@mail.gmail.com... Hi. I wonder, what Hadoop community uses in order to

Re: NameNode resilency

2009-04-11 Thread Stas Oskin
Hi. Any tutorial about using Zookeeper with NameNode? Thanks! 2009/4/12 Billy Pearson sa...@pearsonwholesale.com Not 100% sure but I thank they plan on using zookeeper to help with namenode fail over but that may have changed. Billy Stas Oskin stas.os...@gmail.com wrote in message