[
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276557#comment-16276557
]
Nandor Kollar commented on PIG-5318:
------------------------------------
bq. You should just do isSpark2_x (sparkVersion.startsWith("2.")) instead of
isSpark2_2_x . If Spark 2.3 gets released, then code will have to change.
You're right, but matching for 2.x is not good enough. On Spark 2.1, abortTask
and abortJob is not called (see SPARK-7953), but looks like in Spark 2.2 this
is fixed (at least it looks like it is fixed). I'll update the patch soon, we
should match Spark 2.2+.
bq. Spark should consistently use the same OutputFormat instance in this case
Ok, so I guess this should be a new Jira for Spark, however Spark 2.2 is
already released, and creates more OutputFormat instances like said before.
Indeed, we shouldn't modify the test case, but how about modifying
PigOutputformat, like I did in the patch (making the relevant variables static)?
> Unit test failures on Pig on Spark with Spark 2.2
> -------------------------------------------------
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
> Issue Type: Bug
> Components: spark
> Reporter: Nandor Kollar
> Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
> org.apache.pig.test.TestAssert#testNegativeWithoutFetch
> org.apache.pig.test.TestAssert#testNegative
> org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
> org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
> org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
> org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
> org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed
> by asserting on the message of the exception's root cause, looks like on
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
