And beside with a single file, if that were possible, how do you handle error? Let' say task 1 ran 3 times : 1 error, 1 speculative and 1 success... A per-task file has been a standard to easily solve that problem.
Bertrand Dechoux On Thu, Jul 10, 2014 at 10:00 PM, Vinod Kumar Vavilapalli < [email protected]> wrote: > Current writes to a single file in HDFS is not possible today. You may > want to write a per-task file and use that entire directory as your output. > > +Vinod > Hortonworks Inc. > http://hortonworks.com/ > > > On Wed, Jul 9, 2014 at 10:42 PM, rab ra <[email protected]> wrote: > >> >> hello >> >> >> >> I have one use-case that spans multiple map tasks in hadoop environment. >> I use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output >> into a file stored in hdfs. This file is shared across all the map tasks. >> Though, they all computes thier output but some of them are missing in the >> output file. >> >> >> >> The output file is an excel file with 8 parameters(headings). Each map >> task is supposed to compute all these 8 values, and save it as soon as it >> is computed. This means, the programming logic of a map task opens the >> file, writes the value and close, 8 times. >> >> >> >> Can someone give me a hint on whats going wrong here? >> >> >> >> Is it possible to make more than one map task to write in a shared file >> in HDFS? >> > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
