[jira] Subscription: PIG patch available
Issue Subscription Filter: PIG patch available (37 issues) Subscriber: pigdaily Key Summary PIG-5318Unit test failures on Pig on Spark with Spark 2.2 https://issues.apache.org/jira/browse/PIG-5318 PIG-5317Upgrade old dependencies: commons-lang, hsqldb, commons-logging https://issues.apache.org/jira/browse/PIG-5317 PIG-5312Uids not set in inner schemas after UNION ONSCHEMA https://issues.apache.org/jira/browse/PIG-5312 PIG-5300hashCode for Bag needs to be order independent https://issues.apache.org/jira/browse/PIG-5300 PIG-5273_SUCCESS file should be created at the end of the job https://issues.apache.org/jira/browse/PIG-5273 PIG-5267Review of org.apache.pig.impl.io.BufferedPositionedInputStream https://issues.apache.org/jira/browse/PIG-5267 PIG-5256Bytecode generation for POFilter and POForeach https://issues.apache.org/jira/browse/PIG-5256 PIG-5191Pig HBase 2.0.0 support https://issues.apache.org/jira/browse/PIG-5191 PIG-5160SchemaTupleFrontend.java is not thread safe, cause PigServer thrown NPE in multithread env https://issues.apache.org/jira/browse/PIG-5160 PIG-5115Builtin AvroStorage generates incorrect avro schema when the same pig field name appears in the alias https://issues.apache.org/jira/browse/PIG-5115 PIG-5106Optimize when mapreduce.input.fileinputformat.input.dir.recursive set to true https://issues.apache.org/jira/browse/PIG-5106 PIG-5081Can not run pig on spark source code distribution https://issues.apache.org/jira/browse/PIG-5081 PIG-5080Support store alias as spark table https://issues.apache.org/jira/browse/PIG-5080 PIG-5057IndexOutOfBoundsException when pig reducer processOnePackageOutput https://issues.apache.org/jira/browse/PIG-5057 PIG-5029Optimize sort case when data is skewed https://issues.apache.org/jira/browse/PIG-5029 PIG-4926Modify the content of start.xml for spark mode https://issues.apache.org/jira/browse/PIG-4926 PIG-4913Reduce jython function initiation during compilation https://issues.apache.org/jira/browse/PIG-4913 PIG-4849pig on tez will cause tez-ui to crash,because the content from timeline server is too long. https://issues.apache.org/jira/browse/PIG-4849 PIG-4750REPLACE_MULTI should compile Pattern once and reuse it https://issues.apache.org/jira/browse/PIG-4750 PIG-4684Exception should be changed to warning when job diagnostics cannot be fetched https://issues.apache.org/jira/browse/PIG-4684 PIG-4656Improve String serialization and comparator performance in BinInterSedes https://issues.apache.org/jira/browse/PIG-4656 PIG-4598Allow user defined plan optimizer rules https://issues.apache.org/jira/browse/PIG-4598 PIG-4551Partition filter is not pushed down in case of SPLIT https://issues.apache.org/jira/browse/PIG-4551 PIG-4539New PigUnit https://issues.apache.org/jira/browse/PIG-4539 PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException https://issues.apache.org/jira/browse/PIG-4515 PIG-4323PackageConverter hanging in Spark https://issues.apache.org/jira/browse/PIG-4323 PIG-4313StackOverflowError in LIMIT operation on Spark https://issues.apache.org/jira/browse/PIG-4313 PIG-4251Pig on Storm https://issues.apache.org/jira/browse/PIG-4251 PIG-4002Disable combiner when map-side aggregation is used https://issues.apache.org/jira/browse/PIG-4002 PIG-3952PigStorage accepts '-tagSplit' to return full split information https://issues.apache.org/jira/browse/PIG-3952 PIG-3911Define unique fields with @OutputSchema https://issues.apache.org/jira/browse/PIG-3911 PIG-3877Getting Geo Latitude/Longitude from Address Lines https://issues.apache.org/jira/browse/PIG-3877 PIG-3873Geo distance calculation using Haversine https://issues.apache.org/jira/browse/PIG-3873 PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange handling of Daylight Saving Time with location based timezones https://issues.apache.org/jira/browse/PIG-3864 PIG-3668COR built-in function when atleast one of the coefficient values is NaN https://issues.apache.org/jira/browse/PIG-3668 PIG-3587add functionality for rolling over dates https://issues.apache.org/jira/browse/PIG-3587 PIG-1804Alow Jython function to implement Algebraic and/or Accumulator interfaces https://issues.apache.org/jira/browse/PIG-1804 You may edit this subscription at: https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=16328=12322384
[jira] Subscription: PIG patch available
Issue Subscription Filter: PIG patch available (38 issues) Subscriber: pigdaily Key Summary PIG-5317Upgrade old dependencies: commons-lang, hsqldb, commons-logging https://issues-test.apache.org/jira/browse/PIG-5317 PIG-5316Initialize mapred.task.id property for PoS jobs https://issues-test.apache.org/jira/browse/PIG-5316 PIG-5312Uids not set in inner schemas after UNION ONSCHEMA https://issues-test.apache.org/jira/browse/PIG-5312 PIG-5310MergeJoin throwing NullPointer Exception https://issues-test.apache.org/jira/browse/PIG-5310 PIG-5300hashCode for Bag needs to be order independent https://issues-test.apache.org/jira/browse/PIG-5300 PIG-5273_SUCCESS file should be created at the end of the job https://issues-test.apache.org/jira/browse/PIG-5273 PIG-5267Review of org.apache.pig.impl.io.BufferedPositionedInputStream https://issues-test.apache.org/jira/browse/PIG-5267 PIG-5256Bytecode generation for POFilter and POForeach https://issues-test.apache.org/jira/browse/PIG-5256 PIG-5191Pig HBase 2.0.0 support https://issues-test.apache.org/jira/browse/PIG-5191 PIG-5160SchemaTupleFrontend.java is not thread safe, cause PigServer thrown NPE in multithread env https://issues-test.apache.org/jira/browse/PIG-5160 PIG-5115Builtin AvroStorage generates incorrect avro schema when the same pig field name appears in the alias https://issues-test.apache.org/jira/browse/PIG-5115 PIG-5106Optimize when mapreduce.input.fileinputformat.input.dir.recursive set to true https://issues-test.apache.org/jira/browse/PIG-5106 PIG-5081Can not run pig on spark source code distribution https://issues-test.apache.org/jira/browse/PIG-5081 PIG-5080Support store alias as spark table https://issues-test.apache.org/jira/browse/PIG-5080 PIG-5057IndexOutOfBoundsException when pig reducer processOnePackageOutput https://issues-test.apache.org/jira/browse/PIG-5057 PIG-5029Optimize sort case when data is skewed https://issues-test.apache.org/jira/browse/PIG-5029 PIG-4926Modify the content of start.xml for spark mode https://issues-test.apache.org/jira/browse/PIG-4926 PIG-4913Reduce jython function initiation during compilation https://issues-test.apache.org/jira/browse/PIG-4913 PIG-4849pig on tez will cause tez-ui to crash,because the content from timeline server is too long. https://issues-test.apache.org/jira/browse/PIG-4849 PIG-4750REPLACE_MULTI should compile Pattern once and reuse it https://issues-test.apache.org/jira/browse/PIG-4750 PIG-4684Exception should be changed to warning when job diagnostics cannot be fetched https://issues-test.apache.org/jira/browse/PIG-4684 PIG-4656Improve String serialization and comparator performance in BinInterSedes https://issues-test.apache.org/jira/browse/PIG-4656 PIG-4598Allow user defined plan optimizer rules https://issues-test.apache.org/jira/browse/PIG-4598 PIG-4551Partition filter is not pushed down in case of SPLIT https://issues-test.apache.org/jira/browse/PIG-4551 PIG-4539New PigUnit https://issues-test.apache.org/jira/browse/PIG-4539 PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException https://issues-test.apache.org/jira/browse/PIG-4515 PIG-4323PackageConverter hanging in Spark https://issues-test.apache.org/jira/browse/PIG-4323 PIG-4313StackOverflowError in LIMIT operation on Spark https://issues-test.apache.org/jira/browse/PIG-4313 PIG-4251Pig on Storm https://issues-test.apache.org/jira/browse/PIG-4251 PIG-4002Disable combiner when map-side aggregation is used https://issues-test.apache.org/jira/browse/PIG-4002 PIG-3952PigStorage accepts '-tagSplit' to return full split information https://issues-test.apache.org/jira/browse/PIG-3952 PIG-3911Define unique fields with @OutputSchema https://issues-test.apache.org/jira/browse/PIG-3911 PIG-3877Getting Geo Latitude/Longitude from Address Lines https://issues-test.apache.org/jira/browse/PIG-3877 PIG-3873Geo distance calculation using Haversine https://issues-test.apache.org/jira/browse/PIG-3873 PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange handling of Daylight Saving Time with location based timezones https://issues-test.apache.org/jira/browse/PIG-3864 PIG-3668COR built-in function when atleast one of the coefficient values is NaN https://issues-test.apache.org/jira/browse/PIG-3668 PIG-3587add functionality for rolling over dates https://issues-test.apache.org/jira/browse/PIG-3587 PIG-1804Alow Jython
[jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2
[
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283023#comment-16283023
]
liyunzhang commented on PIG-5318:
-
[~nkollar]: testKeepGoingFailed is excluded because of SPARK-7953. At that
time we used spark 1.3. And after upgrading to 1.6, not enable this test again.
> Unit test failures on Pig on Spark with Spark 2.2
> -
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
> Issue Type: Bug
> Components: spark
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch,
> PIG-5318_4.patch, PIG-5318_5.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
> org.apache.pig.test.TestAssert#testNegativeWithoutFetch
> org.apache.pig.test.TestAssert#testNegative
> org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
> org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
> org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
> org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
> org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed
> by asserting on the message of the exception's root cause, looks like on
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2
[
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281815#comment-16281815
]
Nandor Kollar commented on PIG-5318:
Attached PIG-5318_5.patch which includes fix for TestAssert, TestScalarAliases,
TestEvalPipeline2, TestStore and TestStoreLocal test cases, but doesn't fix
TestStoreInstances failure. The Spark version is determined like Rohini
suggested. I also noticed, that testKeepGoigFailed (fixed the typo in method
name, now testKeepGoingFailed) was excluded from spark exec type, I enabled
this test case, since it passed in my environment with 1.6, 2.1 and 2.2 Spark
versions. [~kellyzly] do you remember why this was excluded? Looks like the
Jira it is referring to is not yet fixed, despite this the test passes with
1.6.x Spark.
> Unit test failures on Pig on Spark with Spark 2.2
> -
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
> Issue Type: Bug
> Components: spark
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch,
> PIG-5318_4.patch, PIG-5318_5.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
> org.apache.pig.test.TestAssert#testNegativeWithoutFetch
> org.apache.pig.test.TestAssert#testNegative
> org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
> org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
> org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
> org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
> org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed
> by asserting on the message of the exception's root cause, looks like on
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2
[
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated PIG-5318:
---
Attachment: PIG-5318_5.patch
> Unit test failures on Pig on Spark with Spark 2.2
> -
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
> Issue Type: Bug
> Components: spark
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch,
> PIG-5318_4.patch, PIG-5318_5.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
> org.apache.pig.test.TestAssert#testNegativeWithoutFetch
> org.apache.pig.test.TestAssert#testNegative
> org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
> org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
> org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
> org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
> org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed
> by asserting on the message of the exception's root cause, looks like on
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (PIG-5319) Investigate why TestStoreInstances fails with Spark 2.2
Nandor Kollar created PIG-5319: -- Summary: Investigate why TestStoreInstances fails with Spark 2.2 Key: PIG-5319 URL: https://issues.apache.org/jira/browse/PIG-5319 Project: Pig Issue Type: Bug Components: spark Reporter: Nandor Kollar TestStoreInstances unit test fails with Spark 2.2.x. It seems in job and task commit logic changed a lot since Spark 2.1.x, now it looks like Spark uses a different PigOutputFormat when writing to files, and a different one when getting the OutputCommitters -- This message was sent by Atlassian JIRA (v6.4.14#64029)
