[
https://issues.apache.org/jira/browse/MAPREDUCE-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-1347:
-------------------------------
Assignee: (was: Harsh J)
> Missing synchronization in MultipleOutputFormat
> -----------------------------------------------
>
> Key: MAPREDUCE-1347
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 1.0.0
> Reporter: Todd Lipcon
> Labels: concurrency
> Attachments: MAPREDUCE-1347.r10.diff, MAPREDUCE-1347.r2.diff,
> MAPREDUCE-1347.r3.diff, MAPREDUCE-1347.r4.diff, MAPREDUCE-1347.r5.diff,
> MAPREDUCE-1347.r6.diff, MAPREDUCE-1347.r7.diff, MAPREDUCE-1347.r8.diff,
> MAPREDUCE-1347.r9.diff, mapreduce.1347.r1.diff
>
>
> MultipleOutputFormat's RecordWriter implementation doesn't use
> synchronization when accessing the recordWriters member. When using
> multithreaded mappers or reducers, this can result in problems where two
> threads will both try to create the same file, causing
> AlreadyBeingCreatedException. Doing this more fine-grained than just
> synchronizing the whole method is probably a good idea, so that multithreaded
> mappers can actually achieve parallelism writing into separate output streams.
> From what I can tell, the new API's MultipleOutputs seems not to have this
> issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
