Thank you so much.
The problem is that I need to operate on text as is, without
modification, and I don't want the filepos to be outputted.
There's no way in hadoop to map and to output a block of text containing
newline characters?
Thank you again,
Francesco
Jingkei Ly ha scritto:
I think you need to strip out the newline characters in the value you
return, as the TextOutputFormat will treat each newline character as the
start of a new record.
-----Original Message-----
From: Francesco Tamberi [mailto:[EMAIL PROTECTED]
Sent: 09 July 2008 11:27
To: [email protected]
Subject: Custom InputFormat/OutputFormat
Hi all,
I want to use hadoop for some streaming text processing on text
documents like:
<doc id=... ... ... >
text text
text
...
</doc>
Just xml-like notation but not real xml files.
I have to work on text included between <doc> tags, so I implemented an
InputFormat (extending FileInputFormat) with a RecordReader that return
file position as Key and needed text as Value.
This is next method and I'm pretty sure that it works as expected..
/** Read a text block. */
public synchronized boolean next(LongWritable key, Text value)
throws IOException
{
if (pos >= end)
return false;
key.set(pos); // key is position
buffer.reset();
long bytesRead = readBlock(startTag, endTag); // put needed
text in buffer
if (bytesRead == 0)
return false;
pos += bytesRead;
value.set(buffer.getData(), 0, buffer.getLength());
return true;
}
But when I test it, using "cat" as mapper function and TextOutputFormat
as OutputFormat, I have one key/value per line:
For every text block, the first tuple has fileposition as key and text
as value, remaining have text as key and no value... ie:
file_pos / first_line
second_line /
third_line /
...
Where am I wrong?
Thank you in advance,
Francesco
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by
an authorised signatory. The contents of this email may relate to dealings
with other companies within the Detica Group plc group of companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.
