[
https://issues.apache.org/jira/browse/CRUNCH-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16877448#comment-16877448
]
Suyash Agarwal edited comment on CRUNCH-635 at 7/3/19 3:16 AM:
---------------------------------------------------------------
Attached the patch for the implementation of path per key target for Text.
I've successfully tried multiple nested sub-directories as the keys and
compression also works.
I was getting AlreadyBeingCreatedException by another attempt while trying to
append to an existing file (across task failures). So I've used committer to
get the work path.
There were cases where a single key had too much data and reducer was taking a
lot of time and was creating a single huge file so I tried writing per key
output from mappers and it's working fine. I thought of maintaining a map of
key and record writer under TextPathPerKeyRecordWriter to avoid creating a
writer everytime a new key is encountered (especially in case of mappers) as
done in
[MultipleOutputs|[https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java#L383]].
because of the possibility of having many keys resulting in a huge map causing
OOM. Not sure if MultipleOutputs will also run into the same issue. If the
approach makes sense, I can implement that.
was (Author: sshagarwal):
Attached the patch for the implementation of path per key target for Text.
I've successfully tried multiple nested sub-directories as the keys and
compression also works.
I was getting AlreadyBeingCreatedException by another attempt while trying to
append to an existing file (across task failures). So I've used committer to
get the work path.
There were cases where a single key had too much data and reducer was taking a
lot of time and was creating a single huge file so I tried writing per key
output from mappers and it's working fine. I thought of maintaining a map of
key and record writer under TextPathPerKeyRecordWriter to avoid creating a
writer everytime a new key is encountered (especially in case of mappers) as
done in
[MultipleOutputs|[https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java#L383|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java#L383].]]
because of the possibility of having many keys resulting in a huge map causing
OOM. Not sure if MultipleOutputs will also run into the same issue. If the
approach makes sense, I can implement that.
> MultipleOutput Text Targets
> ---------------------------
>
> Key: CRUNCH-635
> URL: https://issues.apache.org/jira/browse/CRUNCH-635
> Project: Crunch
> Issue Type: Improvement
> Components: IO
> Reporter: Jim McStanton
> Priority: Minor
> Attachments: CRUNCH-635.patch
>
>
> CRUNCH-306 was logged to add the ability to write an output file per key in a
> PTable. The original issue covers just Avro output files, this Improvement
> would extend this by adding support for Text file targets.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
