Phantom wrote:
Which would mean that if I want to have my logs to reside in HDFS I will
have to move them using copyFromLocal or some version thereof and then run
Map/Reduce process against them ? Am I right ?
Yes. HDFS is probably not currently suitable for directly storing log
output as it
Which would mean that if I want to have my logs to reside in HDFS I will
have to move them using copyFromLocal or some version thereof and then run
Map/Reduce process against them ? Am I right ?
Thanks
Avinash
On 6/13/07, Owen O'Malley <[EMAIL PROTECTED]> wrote:
On Jun 13, 2007, at 3:29 PM, P
On Jun 13, 2007, at 3:29 PM, Phantom wrote:
Hmm I was under the impression that HDFS is like GFS optimized for
appends
although GFS supports random writes.
HDFS doesn't support appends. There has been discussion of
implementing single-writer appends, but it hasn't reached the top of
any
Yeah, you are right about the google fs.
I have also heard from this list that some people are planning on
adding the append functionality to Hadoop, but it's just not there
yet. I am not sure why.
Perhaps my "inefficient" comment was premature. The term logging
stuck in my head and I have pre
Hmm I was under the impression that HDFS is like GFS optimized for appends
although GFS supports random writes. So let's say I want to process logs
using Hadoop. The only way I can do it is to move the entire log into Hadoop
from some place else and then perhaps run Map/Reduce jobs against it. It
No appending, AFAIK. Hadoop is not intended for writing in this way.
It's more of a write few read many system. Such granular writes would
be inefficient.
On 6/13/07, Phantom <[EMAIL PROTECTED]> wrote:
Hi
Can this only be done for read only and write only mode ? How do I do
appends ? Because i
Hi
Can this only be done for read only and write only mode ? How do I do
appends ? Because if I am using this for writing logs then I would want to
append to the file rather overwrite which is what the write only mode is
doing.
Thanks
A
