Rohini Palaniswamy created PIG-4845:
---------------------------------------
Summary: Parallel instantiation of classes in Tez cause tasks to
fail
Key: PIG-4845
URL: https://issues.apache.org/jira/browse/PIG-4845
Project: Pig
Issue Type: Bug
Reporter: Rohini Palaniswamy
Assignee: Rohini Palaniswamy
Fix For: 0.16.0
OptionBuilder is not thread-safe
https://commons.apache.org/proper/commons-cli/javadocs/api-release/org/apache/commons/cli/OptionBuilder.html
{code}
This class is NOT thread safe. See CLI-209
{code}
HBaseStorage, elephantbird SequenceFileConfig, etc use it in their constructor.
This leads to NoSuchMethodException, UnrecognizedOptionException etc when
processor, inputs and outputs are initialized in parallel in Tez making the
task fail. Retry attempts mostly go through and job might succeed. Need to
synchronize the initialization. synchronize would make it similar to mapreduce
though at a slight performance cost. But that should be lot better than cost of
relaunching containers after hitting failures.
Following two kinds of exception where seen when HBaseStorage and
SequenceFileStorage of elephantbird were used together.
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
-loadKey
{code}
Caused by: java.lang.RuntimeException: Failed to create WritableConverter
instance
at
com.twitter.elephantbird.pig.util.SequenceFileConfig.getWritableConverter(SequenceFileConfig.java:225)
at
com.twitter.elephantbird.pig.util.SequenceFileConfig.<init>(SequenceFileConfig.java:101)
at
com.twitter.elephantbird.pig.util.SequenceFileConfig.<init>(SequenceFileConfig.java:115)
... 11 more
Caused by: java.lang.NoSuchMethodException:
com.twitter.elephantbird.pig.util.TextConverter.<init>(java.lang.String)
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getConstructor(Class.java:1825)
at
com.twitter.elephantbird.pig.util.SequenceFileConfig.getWritableConverter(SequenceFileConfig.java:213)
... 13 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
