[
https://issues.apache.org/jira/browse/PIG-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112500#comment-16112500
]
Nandor Kollar commented on PIG-5283:
------------------------------------
This is a problem with Hadoop 3 (HADOOP-14459) and Spark with Pig. Any Pig
script will fail on Hadoop 3 with spark exec type, and like Adam mentioned in
the description, the problem might be with {{ow.setConf(new
Configuration(false))}} in Spark SerializableWritable class. It creates a new
configuration, so whatever was set before is gone. HADOOP-8588 was committed in
Hadoop 3, supposed to fix an NPE, but actually causes an other one, if the
required key is not set in Configuration (that's why it is fine on Hadoop 2).
> Configuration is not passed to SparkPigSplits on the backend
> ------------------------------------------------------------
>
> Key: PIG-5283
> URL: https://issues.apache.org/jira/browse/PIG-5283
> Project: Pig
> Issue Type: Bug
> Components: spark
> Reporter: Adam Szita
> Assignee: Adam Szita
> Attachments: PIG-5283.0.patch
>
>
> When a Hadoop ObjectWritable is created during a Spark job, the instantiated
> PigSplit (wrapped into a SparkPigSplit) is given an empty Configuration
> instance.
> This happens
> [here|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SerializableWritable.scala#L44]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
