[
https://issues.apache.org/jira/browse/PIG-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15885835#comment-15885835
]
Adam Szita commented on PIG-5140:
---------------------------------
Seems like Pig on Spark was never able to handle empty directories. If 0 splits
(partitions) were read then Spark will cut the job submission before notifying
listeners here:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L577.
Because of this the mapping of the new JobId with its job group will not happen
as it normally does at:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala#L190
Uploaded [^PIG-5140.0.patch] to fix Pig's Spark integration so it doesn't
expect new JobIds when the Store RDD had zero partitions.
> fix TestEmptyInputDir unit test failure after PIG-5132
> ------------------------------------------------------
>
> Key: PIG-5140
> URL: https://issues.apache.org/jira/browse/PIG-5140
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: Adam Szita
> Fix For: spark-branch
>
> Attachments: PIG-5140.0.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
