FUSE and Hadoop 0.23

[Dejan Menges]

Hi,

Does anyone has some production ready experience with latest HDFS and one
of FUSE-like products?

Tried until now fuse-dfs and hdfs-fuse, but looks like it's not ported for
newer Hadoop installations, and lot of issues with Java library
compatibility.

What's, from your experience, best way to resolve this issue?

Thanks a lot!

Dejan

P.S. Before bombing list tried to find adequate documentation, but wasn't
so successful :) Thanks in advance!

Re: Desperate!!!! Expanding,shrinking cluster or replacing failed nodes.

[Dejan Menges]

Hi,

Here is quick info in section 1.5 http://wiki.apache.org/hadoop/FAQ

So just briefly - when you add new node, and you are sure configuration on
that one is fine, before you start anything you need to issue hadoop
dfsadmin -refreshNodes after what you need to start datanode/mr services.

Hope this helps!

Regards,
Dejan

On Tue, Dec 20, 2011 at 1:42 PM, Sloot, Hans-Peter 
hans-peter.sl...@atos.net wrote:

 Hello all,

 I have asked this question a couple of days ago but no one responded.

 I built a 6 node hadoop cluster, guided Michael Noll, starting with a
 single node and expanding it one by one.
 Every time I expanded the cluster I ran into error : java.io.IOException:
 Incompatible namespaceIDs

 So now my question is what is the correct procedure for expanding,
 shrinking a cluster?
 And how to replace a failed node?

 Can someone  point me to the correct manuals.
 I have already looked at the available documents on the wiki and
 hadoop.apache.org but could not find the answers.

 Regards Hans-Peter





 Dit bericht is vertrouwelijk en kan geheime informatie bevatten enkel
 bestemd voor de geadresseerde. Indien dit bericht niet voor u is bestemd,
 verzoeken wij u dit onmiddellijk aan ons te melden en het bericht te
 vernietigen. Aangezien de integriteit van het bericht niet veilig gesteld
 is middels verzending via internet, kan Atos Nederland B.V. niet
 aansprakelijk worden gehouden voor de inhoud daarvan. Hoewel wij ons
 inspannen een virusvrij netwerk te hanteren, geven wij geen enkele garantie
 dat dit bericht virusvrij is, noch aanvaarden wij enige aansprakelijkheid
 voor de mogelijke aanwezigheid van een virus in dit bericht. Op al onze
 rechtsverhoudingen, aanbiedingen en overeenkomsten waaronder Atos Nederland
 B.V. goederen en/of diensten levert zijn met uitsluiting van alle andere
 voorwaarden de Leveringsvoorwaarden van Atos Nederland B.V. van toepassing.
 Deze worden u op aanvraag direct kosteloos toegezonden.

 This e-mail and the documents attached are confidential and intended
 solely for the addressee; it may also be privileged. If you receive this
 e-mail in error, please notify the sender immediately and destroy it. As
 its integrity cannot be secured on the Internet, the Atos Nederland B.V.
 group liability cannot be triggered for the message content. Although the
 sender endeavours to maintain a computer virus-free network, the sender
 does not warrant that this transmission is virus-free and will not be
 liable for any damages resulting from any virus transmitted. On all offers
 and agreements under which Atos Nederland B.V. supplies goods and/or
 services of whatever nature, the Terms of Delivery from Atos Nederland B.V.
 exclusively apply. The Terms of Delivery shall be promptly submitted to you
 on your request.

 Atos Nederland B.V. / Utrecht
 KvK Utrecht 30132762

Re: HDFS Explained as Comics

[Dejan Menges]

Hi Maneesh,

Thanks a lot for this! Just distributed it over the team and comments are
great :)

Best regards,
Dejan

On Wed, Nov 30, 2011 at 9:28 PM, maneesh varshney mvarsh...@gmail.comwrote:

 For your reading pleasure!

 PDF 3.3MB uploaded at (the mailing list has a cap of 1MB attachments):

 https://docs.google.com/open?id=0B-zw6KHOtbT4MmRkZWJjYzEtYjI3Ni00NTFjLWE0OGItYTU5OGMxYjc0N2M1


 Appreciate if you can spare some time to peruse this little experiment of
 mine to use Comics as a medium to explain computer science topics. This
 particular issue explains the protocols and internals of HDFS.

 I am eager to hear your opinions on the usefulness of this visual medium to
 teach complex protocols and algorithms.

 [My personal motivations: I have always found text descriptions to be too
 verbose as lot of effort is spent putting the concepts in proper time-space
 context (which can be easily avoided in a visual medium); sequence diagrams
 are unwieldy for non-trivial protocols, and they do not explain concepts;
 and finally, animations/videos happen too fast and do not offer
 self-paced learning experience.]

 All forms of criticisms, comments (and encouragements) welcome :)

 Thanks
 Maneesh

Re: HDFS and Openstack - avoiding excessive redundancy

[Dejan Menges]

Replication factor for HDFS can easily be changed to 1 if you don't need it's 
redundancy in hdfs-site.xml

Regards,
Dejo

Sent from my iPhone

On 12. 11. 2011., at 03:58, Edmon Begoli ebeg...@gmail.com wrote:

 A question related to standing up cloud infrastructure for running 
 Hadoop/HDFS.
 
 We are building up an infrastructure using Openstack which has its own
 storage management redundancy.
 
 We are planning to use Openstack to instantiate Hadoop nodes (HDFS,
 M/R tasks, Hive, HBase)
 on demand.
 
 The problem is that HDFS by design creates three copies of the data,
 so there is a 4x times redundancy
 which we would prefer to avoid.
 
 I am asking here if anyone has had a similar case and if anyone has
 had any helpful solution to recommend.
 
 Thank you in advance,
 Edmon