Hello
I realize that using HTTP, you can have a file in HDFS streamed - that
is, the servlet responds to the following request with Content-
Disposition: attachment, and a download is forced (at least from a
browsers perspective) like so:
http://localhost:50075/streamFile?filename=/somewhere/image.jpg
Is there another way to get at this file more directly from HTTP 'out
of the box'?
I'm imagining something like:
http://localhost:50075/somewhere/image.jpg
Is this sort of exposure of the HDFS namespace something I need to
write into a server myself?
Thanks in advance
David
On Mar 13, 2009, at 10:12 PM, S D wrote:
I've used wget with Hadoop Streaming without any problems. Based on
the
error code you're getting, I suggest you make sure that you have the
proper
write permissions for the directory in which Hadoop will process
(e.g.,
download, convert, ...) on each of the task tracker machines. The
location
where is processed on each machine is controlled by the
"hadoop.tmp.dir"
variable. The default value set in $HADOOP_HOME/conf/hadoop-
default.xml is
"/tmp/hadoop-${user.name}". Make sure that the user running hadoop has
permission to write to whatever directory you're using.
John
On Thu, Mar 12, 2009 at 10:02 PM, Nick Cen <cenyo...@gmail.com> wrote:
Hi All,
I am trying to use the hadoop straeming with "wget" to simulate a
distributed downloader.
The command line i use is
./bin/hadoop jar -D mapred.reduce.tasks=0
contrib/streaming/hadoop-0.19.0-streaming.jar -input urli -output
urlo
-mapper /usr/bin/wget -outputformat
org.apache.hadoop.mapred.lib.MultipleTextOutputFormat
But it thrown an exception
java.lang.RuntimeException: PipeMapRed.waitOutputThreads():
subprocess
failed with code 1
at
org
.apache
.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:295)
at
org
.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:
519)
at
org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
can somebody point me a way of why this happend. thanks.
--
http://daily.appspot.com/food/
