[
https://issues.apache.org/jira/browse/PIG-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205477#comment-15205477
]
Daniel Dai commented on PIG-4845:
---------------------------------
+1
> Parallel instantiation of classes in Tez cause tasks to fail
> ------------------------------------------------------------
>
> Key: PIG-4845
> URL: https://issues.apache.org/jira/browse/PIG-4845
> Project: Pig
> Issue Type: Bug
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.16.0
>
> Attachments: PIG-4845-1.patch
>
>
> OptionBuilder is not thread-safe
> https://commons.apache.org/proper/commons-cli/javadocs/api-release/org/apache/commons/cli/OptionBuilder.html
> {code}
> This class is NOT thread safe. See CLI-209
> {code}
> HBaseStorage, elephantbird SequenceFileConfig, etc use it in their
> constructor. This leads to NoSuchMethodException, UnrecognizedOptionException
> etc when processor, inputs and outputs are initialized in parallel in Tez
> making the task fail. Retry attempts mostly go through and job might succeed.
> Need to synchronize the initialization. synchronize would make it similar to
> mapreduce though at a slight performance cost. But that should be lot better
> than cost of relaunching containers after hitting failures.
> Following two kinds of exception where seen when HBaseStorage and
> SequenceFileStorage of elephantbird were used together.
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
> -loadKey
> {code}
> Caused by: java.lang.RuntimeException: Failed to create WritableConverter
> instance
> at
> com.twitter.elephantbird.pig.util.SequenceFileConfig.getWritableConverter(SequenceFileConfig.java:225)
> at
> com.twitter.elephantbird.pig.util.SequenceFileConfig.<init>(SequenceFileConfig.java:101)
> at
> com.twitter.elephantbird.pig.util.SequenceFileConfig.<init>(SequenceFileConfig.java:115)
> ... 11 more
> Caused by: java.lang.NoSuchMethodException:
> com.twitter.elephantbird.pig.util.TextConverter.<init>(java.lang.String)
> at java.lang.Class.getConstructor0(Class.java:3082)
> at java.lang.Class.getConstructor(Class.java:1825)
> at
> com.twitter.elephantbird.pig.util.SequenceFileConfig.getWritableConverter(SequenceFileConfig.java:213)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
