date:20171204

[jira] Subscription: PIG patch available

2017-12-04 Thread jira

Issue Subscription
Filter: PIG patch available (38 issues)

Subscriber: pigdaily

Key Summary
PIG-5317Upgrade old dependencies: commons-lang, hsqldb, commons-logging
https://issues-test.apache.org/jira/browse/PIG-5317
PIG-5316Initialize mapred.task.id property for PoS jobs
https://issues-test.apache.org/jira/browse/PIG-5316
PIG-5312Uids not set in inner schemas after UNION ONSCHEMA
https://issues-test.apache.org/jira/browse/PIG-5312
PIG-5310MergeJoin throwing NullPointer Exception
https://issues-test.apache.org/jira/browse/PIG-5310
PIG-5300hashCode for Bag needs to be order independent 
https://issues-test.apache.org/jira/browse/PIG-5300
PIG-5273_SUCCESS file should be created at the end of the job
https://issues-test.apache.org/jira/browse/PIG-5273
PIG-5267Review of org.apache.pig.impl.io.BufferedPositionedInputStream
https://issues-test.apache.org/jira/browse/PIG-5267
PIG-5256Bytecode generation for POFilter and POForeach
https://issues-test.apache.org/jira/browse/PIG-5256
PIG-5191Pig HBase 2.0.0 support
https://issues-test.apache.org/jira/browse/PIG-5191
PIG-5160SchemaTupleFrontend.java is not thread safe, cause PigServer thrown 
NPE in multithread env
https://issues-test.apache.org/jira/browse/PIG-5160
PIG-5115Builtin AvroStorage generates incorrect avro schema when the same 
pig field name appears in the alias
https://issues-test.apache.org/jira/browse/PIG-5115
PIG-5106Optimize when mapreduce.input.fileinputformat.input.dir.recursive 
set to true
https://issues-test.apache.org/jira/browse/PIG-5106
PIG-5081Can not run pig on spark source code distribution
https://issues-test.apache.org/jira/browse/PIG-5081
PIG-5080Support store alias as spark table
https://issues-test.apache.org/jira/browse/PIG-5080
PIG-5057IndexOutOfBoundsException when pig reducer processOnePackageOutput
https://issues-test.apache.org/jira/browse/PIG-5057
PIG-5029Optimize sort case when data is skewed
https://issues-test.apache.org/jira/browse/PIG-5029
PIG-4926Modify the content of start.xml for spark mode
https://issues-test.apache.org/jira/browse/PIG-4926
PIG-4913Reduce jython function initiation during compilation
https://issues-test.apache.org/jira/browse/PIG-4913
PIG-4849pig on tez will cause tez-ui to crash,because the content from 
timeline server is too long. 
https://issues-test.apache.org/jira/browse/PIG-4849
PIG-4750REPLACE_MULTI should compile Pattern once and reuse it
https://issues-test.apache.org/jira/browse/PIG-4750
PIG-4684Exception should be changed to warning when job diagnostics cannot 
be fetched
https://issues-test.apache.org/jira/browse/PIG-4684
PIG-4656Improve String serialization and comparator performance in 
BinInterSedes
https://issues-test.apache.org/jira/browse/PIG-4656
PIG-4598Allow user defined plan optimizer rules
https://issues-test.apache.org/jira/browse/PIG-4598
PIG-4551Partition filter is not pushed down in case of SPLIT
https://issues-test.apache.org/jira/browse/PIG-4551
PIG-4539New PigUnit
https://issues-test.apache.org/jira/browse/PIG-4539
PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException
https://issues-test.apache.org/jira/browse/PIG-4515
PIG-4323PackageConverter hanging in Spark
https://issues-test.apache.org/jira/browse/PIG-4323
PIG-4313StackOverflowError in LIMIT operation on Spark
https://issues-test.apache.org/jira/browse/PIG-4313
PIG-4251Pig on Storm
https://issues-test.apache.org/jira/browse/PIG-4251
PIG-4002Disable combiner when map-side aggregation is used
https://issues-test.apache.org/jira/browse/PIG-4002
PIG-3952PigStorage accepts '-tagSplit' to return full split information
https://issues-test.apache.org/jira/browse/PIG-3952
PIG-3911Define unique fields with @OutputSchema
https://issues-test.apache.org/jira/browse/PIG-3911
PIG-3877Getting Geo Latitude/Longitude from Address Lines
https://issues-test.apache.org/jira/browse/PIG-3877
PIG-3873Geo distance calculation using Haversine
https://issues-test.apache.org/jira/browse/PIG-3873
PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange 
handling of Daylight Saving Time with location based timezones
https://issues-test.apache.org/jira/browse/PIG-3864
PIG-3668COR built-in function when atleast one of the coefficient values is 
NaN
https://issues-test.apache.org/jira/browse/PIG-3668
PIG-3587add functionality for rolling over dates
https://issues-test.apache.org/jira/browse/PIG-3587
PIG-1804Alow Jython

[jira] Subscription: PIG patch available

2017-12-04 Thread jira

Issue Subscription
Filter: PIG patch available (37 issues)

Subscriber: pigdaily

Key Summary
PIG-5318Unit test failures on Pig on Spark with Spark 2.2
https://issues.apache.org/jira/browse/PIG-5318
PIG-5317Upgrade old dependencies: commons-lang, hsqldb, commons-logging
https://issues.apache.org/jira/browse/PIG-5317
PIG-5312Uids not set in inner schemas after UNION ONSCHEMA
https://issues.apache.org/jira/browse/PIG-5312
PIG-5300hashCode for Bag needs to be order independent 
https://issues.apache.org/jira/browse/PIG-5300
PIG-5273_SUCCESS file should be created at the end of the job
https://issues.apache.org/jira/browse/PIG-5273
PIG-5267Review of org.apache.pig.impl.io.BufferedPositionedInputStream
https://issues.apache.org/jira/browse/PIG-5267
PIG-5256Bytecode generation for POFilter and POForeach
https://issues.apache.org/jira/browse/PIG-5256
PIG-5191Pig HBase 2.0.0 support
https://issues.apache.org/jira/browse/PIG-5191
PIG-5160SchemaTupleFrontend.java is not thread safe, cause PigServer thrown 
NPE in multithread env
https://issues.apache.org/jira/browse/PIG-5160
PIG-5115Builtin AvroStorage generates incorrect avro schema when the same 
pig field name appears in the alias
https://issues.apache.org/jira/browse/PIG-5115
PIG-5106Optimize when mapreduce.input.fileinputformat.input.dir.recursive 
set to true
https://issues.apache.org/jira/browse/PIG-5106
PIG-5081Can not run pig on spark source code distribution
https://issues.apache.org/jira/browse/PIG-5081
PIG-5080Support store alias as spark table
https://issues.apache.org/jira/browse/PIG-5080
PIG-5057IndexOutOfBoundsException when pig reducer processOnePackageOutput
https://issues.apache.org/jira/browse/PIG-5057
PIG-5029Optimize sort case when data is skewed
https://issues.apache.org/jira/browse/PIG-5029
PIG-4926Modify the content of start.xml for spark mode
https://issues.apache.org/jira/browse/PIG-4926
PIG-4913Reduce jython function initiation during compilation
https://issues.apache.org/jira/browse/PIG-4913
PIG-4849pig on tez will cause tez-ui to crash,because the content from 
timeline server is too long. 
https://issues.apache.org/jira/browse/PIG-4849
PIG-4750REPLACE_MULTI should compile Pattern once and reuse it
https://issues.apache.org/jira/browse/PIG-4750
PIG-4684Exception should be changed to warning when job diagnostics cannot 
be fetched
https://issues.apache.org/jira/browse/PIG-4684
PIG-4656Improve String serialization and comparator performance in 
BinInterSedes
https://issues.apache.org/jira/browse/PIG-4656
PIG-4598Allow user defined plan optimizer rules
https://issues.apache.org/jira/browse/PIG-4598
PIG-4551Partition filter is not pushed down in case of SPLIT
https://issues.apache.org/jira/browse/PIG-4551
PIG-4539New PigUnit
https://issues.apache.org/jira/browse/PIG-4539
PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException
https://issues.apache.org/jira/browse/PIG-4515
PIG-4323PackageConverter hanging in Spark
https://issues.apache.org/jira/browse/PIG-4323
PIG-4313StackOverflowError in LIMIT operation on Spark
https://issues.apache.org/jira/browse/PIG-4313
PIG-4251Pig on Storm
https://issues.apache.org/jira/browse/PIG-4251
PIG-4002Disable combiner when map-side aggregation is used
https://issues.apache.org/jira/browse/PIG-4002
PIG-3952PigStorage accepts '-tagSplit' to return full split information
https://issues.apache.org/jira/browse/PIG-3952
PIG-3911Define unique fields with @OutputSchema
https://issues.apache.org/jira/browse/PIG-3911
PIG-3877Getting Geo Latitude/Longitude from Address Lines
https://issues.apache.org/jira/browse/PIG-3877
PIG-3873Geo distance calculation using Haversine
https://issues.apache.org/jira/browse/PIG-3873
PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange 
handling of Daylight Saving Time with location based timezones
https://issues.apache.org/jira/browse/PIG-3864
PIG-3668COR built-in function when atleast one of the coefficient values is 
NaN
https://issues.apache.org/jira/browse/PIG-3668
PIG-3587add functionality for rolling over dates
https://issues.apache.org/jira/browse/PIG-3587
PIG-1804Alow Jython function to implement Algebraic and/or Accumulator 
interfaces
https://issues.apache.org/jira/browse/PIG-1804

You may edit this subscription at:
https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=16328=12322384

Build failed in Jenkins: Pig-trunk #2060

2017-12-04 Thread Apache Jenkins Server

See 

Changes:

[rohini] PIG-5315: pig.script is not set for scripts run via PigServer 
(satishsaley via rohini)

--
[...truncated 19.84 KB...]
   [javacc] File "Token.java" does not exist.  Will create one.
   [javacc] File "SimpleCharStream.java" does not exist.  Will create one.
   [javacc] Parser generated successfully.
 [move] Moving 1 file to 


prepare:
[mkdir] Created dir: 


genLexer:

genParser:

genTreeParser:

gen:

compile:
 [echo] *** Building Main Sources ***
 [echo] *** To compile with all warnings enabled, supply -Dall.warnings=1 
on command line ***
 [echo] *** Else, you will only be warned about deprecations ***
 [echo] *** Hadoop version used: 2 ; HBase version used: 1 ; Spark version 
used: 1 ***
[javac] Compiling 1105 source files to 

[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
 [copy] Copying 1 file to 

 [copy] Copying 1 file to 

 [copy] Copying 2 files to 

 [copy] Copying 1 file to 


ivy-buildJar:

jar-simple:
 [echo] svnString 1817132
  [jar] Building jar: 

 [echo] svnString 1817132
  [jar] Building jar: 

  [jar] META-INF/native/linux32/libjansi.so already added, skipping
  [jar] META-INF/native/linux64/libjansi.so already added, skipping
  [jar] META-INF/native/osx/libjansi.jnilib already added, skipping
  [jar] META-INF/native/windows32/jansi.dll already added, skipping
  [jar] META-INF/native/windows64/jansi.dll already added, skipping
  [jar] org/fusesource/hawtjni/runtime/Callback.class already added, 
skipping
  [jar] org/fusesource/hawtjni/runtime/Library.class already added, skipping
  [jar] org/fusesource/hawtjni/runtime/PointerMath.class already added, 
skipping
  [jar] org/fusesource/jansi/Ansi$1.class already added, skipping
  [jar] org/fusesource/jansi/Ansi$2.class already added, skipping
  [jar] org/fusesource/jansi/Ansi$Attribute.class already added, skipping
  [jar] org/fusesource/jansi/Ansi$Color.class already added, skipping
  [jar] org/fusesource/jansi/Ansi$Erase.class already added, skipping
  [jar] org/fusesource/jansi/Ansi$NoAnsi.class already added, skipping
  [jar] org/fusesource/jansi/Ansi.class already added, skipping
  [jar] org/fusesource/jansi/AnsiConsole$1.class already added, skipping
  [jar] org/fusesource/jansi/AnsiConsole.class already added, skipping
  [jar] org/fusesource/jansi/AnsiOutputStream.class already added, skipping
  [jar] org/fusesource/jansi/AnsiRenderWriter.class already added, skipping
  [jar] org/fusesource/jansi/AnsiRenderer$Code.class already added, skipping
  [jar] org/fusesource/jansi/AnsiRenderer.class already added, skipping
  [jar] org/fusesource/jansi/AnsiString.class already added, skipping
  [jar] org/fusesource/jansi/HtmlAnsiOutputStream.class already added, 
skipping
  [jar] org/fusesource/jansi/WindowsAnsiOutputStream.class already added, 
skipping
  [jar] org/fusesource/jansi/internal/CLibrary.class already added, skipping
  [jar] 
org/fusesource/jansi/internal/Kernel32$CONSOLE_SCREEN_BUFFER_INFO.class already 
added, skipping
  [jar] org/fusesource/jansi/internal/Kernel32$COORD.class already added, 
skipping
  [jar] org/fusesource/jansi/internal/Kernel32$INPUT_RECORD.class already 
added, skipping
  [jar] org/fusesource/jansi/internal/Kernel32$KEY_EVENT_RECORD.class 
already added, skipping
  [jar] org/fusesource/jansi/internal/Kernel32$SMALL_RECT.class already 
added, skipping
  [jar] org/fusesource/jansi/internal/Kernel32.class already added, skipping
  [jar] org/fusesource/jansi/internal/WindowsSupport.class already added, 
skipping
Trying to override old definition of task propertycopy
Trying to override old definition of task propertycopy

copyCommonDependencies:
[mkdir] Created dir: 
 [copy] Copying 47 files to 
Trying to override old

[jira] [Updated] (PIG-5315) pig.script is not set for scripts run via PigServer

2017-12-04 Thread Satish Subhashrao Saley (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated PIG-5315:
-
Attachment: PIG-5315-2.patch

> pig.script is not set for scripts run via PigServer
> ---
>
> Key: PIG-5315
> URL: https://issues.apache.org/jira/browse/PIG-5315
> Project: Pig
>  Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Satish Subhashrao Saley
>Priority: Minor
>  Labels: newbie
> Fix For: 0.18.0
>
> Attachments: PIG-5315-1.patch, PIG-5315-2.patch
>
>
> ScriptState.get().setScript() is only called in Main and BoundScript and not 
> in PigServer.registerScript



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

2017-12-04 Thread Rohini Palaniswamy (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277122#comment-16277122
 ] 

Rohini Palaniswamy commented on PIG-5318:
-

bq. it looks like the way I wanted to tell Spark version doesn't work on Spark 
1.x
  Missed this earlier. If the spark-version-info.properties file is not there, 
you could just return false for isSpark2_2_plus which will be easier.

> Unit test failures on Pig on Spark with Spark 2.2
> -
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
>  Issue Type: Bug
>  Components: spark
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch, 
> PIG-5318_4.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
>  org.apache.pig.test.TestAssert#testNegativeWithoutFetch
>  org.apache.pig.test.TestAssert#testNegative
>  org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
>  org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
>  org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
>  org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
>  org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed 
> by asserting on the message of the exception's root cause, looks like on 
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks 
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

2017-12-04 Thread Rohini Palaniswamy (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277120#comment-16277120
 ] 

Rohini Palaniswamy commented on PIG-5318:
-

bq. but how about modifying PigOutputformat, like I did in the patch (making 
the relevant variables static)?
 This cannot be done. It is hacky and will break Pig local mode and Tez. In 
local mode, save jvm is used to execute the whole script which can have 
parallel STORE statements. Tez also allows storing to multiple outputs from 
same vertex in a DAG - i.e multiple PigOutputFormat in the save jvm.

bq. isSpark2_1_minus
  Can you make it  isSpark2_2_plus which is slightly more intuitive than 
2_1_minus. Also instantiating SparkContext just to get version seems overkill. 
Prefer the previous logic you had. Is there any reason that could not be used?


> Unit test failures on Pig on Spark with Spark 2.2
> -
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
>  Issue Type: Bug
>  Components: spark
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch, 
> PIG-5318_4.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
>  org.apache.pig.test.TestAssert#testNegativeWithoutFetch
>  org.apache.pig.test.TestAssert#testNegative
>  org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
>  org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
>  org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
>  org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
>  org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed 
> by asserting on the message of the exception's root cause, looks like on 
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks 
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

2017-12-04 Thread Nandor Kollar (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276747#comment-16276747
 ] 

Nandor Kollar commented on PIG-5318:


Attached PIG-5318_4.patch, it looks like the way I wanted to tell Spark version 
doesn't work on Spark 1.x, using SparkContext#version instead.

> Unit test failures on Pig on Spark with Spark 2.2
> -
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
>  Issue Type: Bug
>  Components: spark
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch, 
> PIG-5318_4.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
>  org.apache.pig.test.TestAssert#testNegativeWithoutFetch
>  org.apache.pig.test.TestAssert#testNegative
>  org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
>  org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
>  org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
>  org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
>  org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed 
> by asserting on the message of the exception's root cause, looks like on 
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks 
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

2017-12-04 Thread Nandor Kollar (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandor Kollar updated PIG-5318:
---
Attachment: PIG-5318_4.patch

> Unit test failures on Pig on Spark with Spark 2.2
> -
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
>  Issue Type: Bug
>  Components: spark
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch, 
> PIG-5318_4.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
>  org.apache.pig.test.TestAssert#testNegativeWithoutFetch
>  org.apache.pig.test.TestAssert#testNegative
>  org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
>  org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
>  org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
>  org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
>  org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed 
> by asserting on the message of the exception's root cause, looks like on 
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks 
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

2017-12-04 Thread Nandor Kollar (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandor Kollar updated PIG-5318:
---
Attachment: PIG-5318_3.patch

> Unit test failures on Pig on Spark with Spark 2.2
> -
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
>  Issue Type: Bug
>  Components: spark
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
>  org.apache.pig.test.TestAssert#testNegativeWithoutFetch
>  org.apache.pig.test.TestAssert#testNegative
>  org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
>  org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
>  org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
>  org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
>  org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed 
> by asserting on the message of the exception's root cause, looks like on 
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks 
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

2017-12-04 Thread Nandor Kollar (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276557#comment-16276557
 ] 

Nandor Kollar commented on PIG-5318:


bq. You should just do isSpark2_x (sparkVersion.startsWith("2.")) instead of 
isSpark2_2_x . If Spark 2.3 gets released, then code will have to change.

You're right, but matching for 2.x is not good enough. On Spark 2.1, abortTask 
and abortJob is not called (see SPARK-7953), but looks like in Spark 2.2 this 
is fixed (at least it looks like it is fixed). I'll update the patch soon, we 
should match Spark 2.2+.

bq. Spark should consistently use the same OutputFormat instance in this case

Ok, so I guess this should be a new Jira for Spark, however Spark 2.2 is 
already released, and creates more OutputFormat instances like said before. 
Indeed, we shouldn't modify the test case, but how about modifying 
PigOutputformat, like I did in the patch (making the relevant variables static)?

> Unit test failures on Pig on Spark with Spark 2.2
> -
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
>  Issue Type: Bug
>  Components: spark
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
>  org.apache.pig.test.TestAssert#testNegativeWithoutFetch
>  org.apache.pig.test.TestAssert#testNegative
>  org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
>  org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
>  org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
>  org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
>  org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed 
> by asserting on the message of the exception's root cause, looks like on 
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks 
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] Subscription: PIG patch available

[jira] Subscription: PIG patch available

Build failed in Jenkins: Pig-trunk #2060

[jira] [Updated] (PIG-5315) pig.script is not set for scripts run via PigServer

[jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

[jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

[jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

[jira] [Updated] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

[jira] [Updated] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

[jira] [Commented] (PIG-5318) Unit test failures on Pig on Spark with Spark 2.2

10 matches

Site Navigation

Mail list logo

Footer information