Hello all
I have a slightly unusual use case that I hope I can use Hadoop for (maybe
after writing a bit of code).
My setup looks as follows:
I have one machine containing somewhere from 750 GB to 2 TB of data on a
disk or some kind of RAID-0 array. I am in full control of this machine. I
have a backup of the part of the data that I haven't generated (maybe 30% of
the data, the rest is calculated from this original data).
I have another disparate collection of machines running any number of
operating systems (Windows, Linux, Solaris, etc.). I might not have
administrative access on these machines. I can probably run a JVM on them.
These machines should not be considered reliable in any sense of the word.
Hosting a HDFS on them would be a bad idea -- on a bad day all of them might
be down, they might get reformatted, you name it.
I need to process this data using legacy applications (e.g., C++ programs,
MATLAB scripts, Python scripts) and Java applications.
The legacy applications typically perform operations that would be hard to
parallelise (e.g. a program that can't easily be compiled on all the
different machines, MATLAB licenses not being available for all the
machines, etc.), and as such I would like to run these legacy apps only on
the machine that directly has access to the data. I would prefer if I didn't
have to teach these legacy apps about getting data out of HDFS.
Through the magic of the JVM, I can can consider parallelising the
processing of the data done with Java programs.
These Java programs (parallelised with Hadoop's MapReduce, or GridGain, or
whatever) need an easy way to access the data on this single machine.
Setting up a traditional shared file system (NFS and Samba come to mind)
would be a pain for various reasons. A shared file system probably wouldn't
be a bottleneck, since the amount of processing time required typically far
outweighs the time it takes to access the data over the network (at least
for the number of nodes I'm dealing with).
What I'm hoping to do is use Hadoop as my shared file system.
I would imagine that I would have to run the equivalent of a
namenode+datanode+etc. on the machine that has direct access to the data so
that it appears as a HDFS to the other machines. Making a copy of the data
into a HDFS instance isn't really an option, due to the size of the data I'm
dealing with, so I'm thinking along the lines of exposing a LocalFileSystem
on this machine as a DistributedFileSystem to the other machines.
This has the added advantage that if I ever do get a stable HDFS setup
going, my programs will be ready to deal with it.
Is something like this doable already? If not, where would one need to start
filling in code to make it possible?
Thanks for reading.
Regards,
Albert
