[
https://issues.apache.org/jira/browse/PIG-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5449:
------------------------------
Attachment: pig-5449-v01.patch
Before (in spark2 land), this used to work by checking empty list returned by
getjobIDs.
https://github.com/apache/pig/blob/branch-0.17/src/org/apache/pig/backend/hadoop/executionengine/spark/JobGraphBuilder.java#L210-L219
But with spark3, this is returning actual jobid but no metrics stored behind.
Instead of adding another logic for spark3, I think we can treat metrics
retrieval as optional like we do in mapreduce & tez. Attaching a patch.
(pig-5449-v01.patch)
> TestEmptyInputDir failing on pig-on-spark3
> ------------------------------------------
>
> Key: PIG-5449
> URL: https://issues.apache.org/jira/browse/PIG-5449
> Project: Pig
> Issue Type: Bug
> Components: spark
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Major
> Attachments: pig-5449-v01.patch
>
>
> TestEmptyInputDir failing on pig-on-spark3 with
> {noformat:title=TestEmptyInputDir.testMergeJoinFailure}
> junit.framework.AssertionFailedError
> at
> org.apache.pig.test.TestEmptyInputDir.testMergeJoin(TestEmptyInputDir.java:141)
> {noformat}
> {noformat:title=TestEmptyInputDir.testGroupByFailure}
> junit.framework.AssertionFailedError
> at
> org.apache.pig.test.TestEmptyInputDir.testGroupBy(TestEmptyInputDir.java:80)
> {noformat}
> {noformat:title=TestEmptyInputDir.testBloomJoinOuterFailure}
> junit.framework.AssertionFailedError
> at
> org.apache.pig.test.TestEmptyInputDir.testBloomJoinOuter(TestEmptyInputDir.java:297)
> {noformat}
> {noformat:title=TestEmptyInputDir.testFRJoinFailure}
> junit.framework.AssertionFailedError
> at
> org.apache.pig.test.TestEmptyInputDir.testFRJoin(TestEmptyInputDir.java:171)
> {noformat}
> {noformat:title=TestEmptyInputDir.testBloomJoinFailure}
> junit.framework.AssertionFailedError
> at
> org.apache.pig.test.TestEmptyInputDir.testBloomJoin(TestEmptyInputDir.java:267)
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
