[
https://issues.apache.org/jira/browse/HADOOP-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671146#action_12671146
]
klbostee edited comment on HADOOP-1722 at 2/6/09 6:35 AM:
----------------------------------------------------------------
This is what changed in version 5 of the patch:
* I implemented Owen's suggestion by adding the package
{{org.apache.hadoop.streaming.io}}. This package contains an
{{IdentifierResolver}} class that resolves string identifiers like "typedbytes"
and "text" (case insensitive) into an {{InputWriter}} class, an
{{OutputReader}} class, a key output class, and a value output class (different
{{OutputReader}} classes require different key and value output classes to be
set). Since a different resolver can be used by setting the property
{{stream.io.identifier.resolver.class}}, external code can add new identifiers
by extending {{IdentifierResolver}} and pointing
{{stream.io.identifier.resolver.class}} to this extension.
* I removed the {{-typedbytes ...}} option and added {{-io <identifier>}}
instead. This latter option is less fine-grained since it triggers usage of the
classes corresponding to the given identifier for everything (e.g. {{-io
typedbytes}} means that typed bytes will be used for the map input, the map
output, the reduce input, and the reduce output), but you can still use more
fine-grained configurations by setting the relevant properties manually. As
suggested by Owen, the properties now are:
** {{stream.map.input}}={{<identifier>}}
** {{stream.map.output}}={{<identifier>}}
** {{stream.reduce.input}}={{<identifier>}}
** {{stream.reduce.output}}={{<identifier>}}
* Since it was easy to do, I also added {{RawBytesInputWriter}} and
{{RawBytesOutputReader}}, which implement Eric's original suggestion (i.e. just
a 4 byte length followed by the raw bytes). Quoting could be added as well of
course, but I would rather pass that on to someone else/another patch...
was (Author: klbostee):
This is what changed in version 5 of the patch:
* I implemented Owen's suggestion by adding the package
{{org.apache.hadoop.streaming.io}}. This package contains an
{{IdentifierResolver}} class that resolves string identifiers like "typedbytes"
and "text" (case insensitive) into an {{InputWriter}} class, an
{{OutputReader}} class, a key output class, and a value output class (different
{{OutputReader}} classes require different key and value output classes to be
set). Since a different resolver can be used by setting the property
{{stream.io.identifier.resolver.class}}, external code can add new identifiers
by extending {{IdentifierResolver}} and pointing
{{stream.io.identifier.resolver.class}} to this extension.
* I removed the {{-typedbytes ...}} option and added {{-io <identifier>}}
instead. This latter option is less fine-grained since it triggers usage of the
classes corresponding to the given identifier for everything (e.g. {{-io
typedbytes}} means that typed bytes will be used for the map input, the map
output, the reduce input, and the reduce output), but you can still use more
fine-grained configurations by setting the relevant properties manually. As
suggested by Owen, the properties now are:
** {{stream.map.input}}={{<identifier>}}
** {{stream.map.output}}={{<identifier>}}
** {{stream.reduce.input}}={{<identifier>}}
** {{stream.reduce.output}}={{<identifier>}}
* Since it was easy to do, I also added {{RawBytesInputWriter}} and
{{RawBytesOutputReader}}, which implement Eric's original suggestion (i.e. just
a 4 byte length followed by the raw bytes). Quoting could be added as well of
course, but I would rather pass that on to some else/another patch...
> Make streaming to handle non-utf8 byte array
> --------------------------------------------
>
> Key: HADOOP-1722
> URL: https://issues.apache.org/jira/browse/HADOOP-1722
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/streaming
> Reporter: Runping Qi
> Assignee: Klaas Bosteels
> Attachments: HADOOP-1722-v2.patch, HADOOP-1722-v3.patch,
> HADOOP-1722-v4.patch, HADOOP-1722-v4.patch, HADOOP-1722-v5.patch,
> HADOOP-1722.patch
>
>
> Right now, the streaming framework expects the output sof the steam process
> (mapper or reducer) are line
> oriented UTF-8 text. This limit makes it impossible to use those programs
> whose outputs may be non-UTF-8
> (international encoding, or maybe even binary data). Streaming can overcome
> this limit by introducing a simple
> encoding protocol. For example, it can allow the mapper/reducer to hexencode
> its keys/values,
> the framework decodes them in the Java side.
> This way, as long as the mapper/reducer executables follow this encoding
> protocol,
> they can output arabitary bytearray and the streaming framework can handle
> them.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
