Hi all,
I'm trying to use Hadoop MapReduce (new api) in a particular way. What I
would like to do is to make it work with a external executable not made
for mapreduce (but able to read from hdfs), and with binary input.
The idea is to store on hdfs the binary input files, and then to run a
mapreduce job specifing these files as input. Once the mapreduce task is
landend to on the node, I would like to block it from reading the input
data, but instead I would like it to spawn the precompiled executable to
load the input data from hdfs. In this way, the mapreduce framework
should have taken care in placing the mapper as closer as possible to
the data, and consequently the binary spawned.
I do not want to run the reduce, the aggreation (very fast for my
computational problem) will be done via a simple script that will take
care of downloading from hdfs the outputs (previously uploaded to hdfs
from the spawned binary).
I made some tests:
I can obtain the file being currently "analyzed" by the mapper (to pass
it to the the spawned binary) using:
Configuration conf = context.getConfiguration();
FileSplit fileSplit = (FileSplit) context.getInputSplit();
String sFileName = fileSplit.getPath().toString();
I could avoid input binary files to be splitted using the "isSplitable"
function in a new InputFormat(about performance: files will be usually
smaller than block size)
But I don't know how to block the map task from reading his input file:
I was thinkg about something like defining a new RecordReader with
records defined by the end of file, so that in the map() function of the
mapper I can spawn the binary. But will this cause the entire file to be
loaded in the memory?
Or, is there a way to tell the MapReduce framework to do not
automatically feed the map task using the push but instead to wait for
the map task to pull? (and never pull?)
Any help is appreciated!
Thnakyou,
Stefano.
