[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-11-07 Thread Barna Zsombor Klara (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15643789#comment-15643789
 ] 

Barna Zsombor Klara commented on HIVE-14910:


Flaky tests are:
https://issues.apache.org/jira/browse/HIVE-14937 - hbase_bulk
https://issues.apache.org/jira/browse/HIVE-15116 - join_acid_non_acid
https://issues.apache.org/jira/browse/HIVE-15115 - union_fast_stats
https://issues.apache.org/jira/browse/HIVE-15084 - explainanalyze_2 and 
explainanalyze_4

> Flaky test: TestSparkClient.testJobSubmission
> -
>
> Key: HIVE-14910
> URL: https://issues.apache.org/jira/browse/HIVE-14910
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14910.1.patch, HIVE-14910.2.patch, 
> HIVE-14910.3.patch, HIVE-14910.patch
>
>
> Have seen this fail in multiple runs (not consistently)
> e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-11-07 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15643769#comment-15643769
 ] 

Hive QA commented on HIVE-14910:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12837710/HIVE-14910.3.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10628 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_bulk] 
(batchId=89)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
 (batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=91)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4] 
(batchId=91)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1996/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1996/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1996/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12837710 - PreCommit-HIVE-Build

> Flaky test: TestSparkClient.testJobSubmission
> -
>
> Key: HIVE-14910
> URL: https://issues.apache.org/jira/browse/HIVE-14910
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14910.1.patch, HIVE-14910.2.patch, 
> HIVE-14910.3.patch, HIVE-14910.patch
>
>
> Have seen this fail in multiple runs (not consistently)
> e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-11-04 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635729#comment-15635729
 ] 

Hive QA commented on HIVE-14910:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12837074/HIVE-14910.3.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1961/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1961/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1961/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-11-04 09:02:49.810
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ MAVEN_OPTS='-Xmx1g -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-1961/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-11-04 09:02:49.813
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 97f7d7b HIVE-15109: Set MaxPermSize to 256M for maven tests with 
JDKs prior to 8 (Chaoyu Tang, reviewed by Siddharth Seth, Sergio Pena)
+ git clean -f -d
Removing testutils/metastore/dataload.properties
Removing testutils/metastore/dbs/derby/validate.sh
Removing testutils/metastore/dbs/mysql/validate.sh
Removing testutils/metastore/dbs/postgres/validate.sh
Removing testutils/metastore/metastore-validation-test.sh
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 97f7d7b HIVE-15109: Set MaxPermSize to 256M for maven tests with 
JDKs prior to 8 (Chaoyu Tang, reviewed by Siddharth Seth, Sergio Pena)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-11-04 09:02:50.710
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file 
spark-client/src/main/java/org/apache/hive/spark/client/JobHandle.java
patching file 
spark-client/src/main/java/org/apache/hive/spark/client/JobHandleImpl.java
patching file 
spark-client/src/main/java/org/apache/hive/spark/client/SparkClient.java
patching file 
spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java
patching file 
spark-client/src/test/java/org/apache/hive/spark/client/TestJobHandle.java
patching file 
spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q 
-Dmaven.repo.local=/data/hiveptest/working/maven
ANTLR Parser Generator  Version 3.4
org/apache/hadoop/hive/metastore/parser/Filter.g
DataNucleus Enhancer (version 4.1.6) for API "JDO"
DataNucleus Enhancer : Classpath
>>  /usr/share/maven/boot/plexus-classworlds-2.x.jar
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDatabase
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MFieldSchema
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MType
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTable
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MConstraint
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MSerDeInfo
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MOrder
ENHANCED (Persistable) : 
org.apache.hadoop.hive.metastore.model.MColumnDescriptor
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MStringList
ENHANCED (Persistable) : 
org.apache.hadoop.hive.metastore.model.MStorageDescriptor
ENHANCED (Persistable) :

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-11-03 Thread Barna Zsombor Klara (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632184#comment-15632184
 ] 

Barna Zsombor Klara commented on HIVE-14910:


I dont't think these failures are related.
I have raised flaky test jiras:
https://issues.apache.org/jira/browse/HIVE-15116
https://issues.apache.org/jira/browse/HIVE-15115

> Flaky test: TestSparkClient.testJobSubmission
> -
>
> Key: HIVE-14910
> URL: https://issues.apache.org/jira/browse/HIVE-14910
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14910.1.patch, HIVE-14910.2.patch, HIVE-14910.patch
>
>
> Have seen this fail in multiple runs (not consistently)
> e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-11-02 Thread Rui Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631077#comment-15631077
 ] 

Rui Li commented on HIVE-14910:
---

+1

> Flaky test: TestSparkClient.testJobSubmission
> -
>
> Key: HIVE-14910
> URL: https://issues.apache.org/jira/browse/HIVE-14910
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14910.1.patch, HIVE-14910.2.patch, HIVE-14910.patch
>
>
> Have seen this fail in multiple runs (not consistently)
> e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-11-02 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630232#comment-15630232
 ] 

Hive QA commented on HIVE-14910:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12836529/HIVE-14910.2.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10628 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
 (batchId=145)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1931/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1931/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1931/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12836529 - PreCommit-HIVE-Build

> Flaky test: TestSparkClient.testJobSubmission
> -
>
> Key: HIVE-14910
> URL: https://issues.apache.org/jira/browse/HIVE-14910
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14910.1.patch, HIVE-14910.2.patch, HIVE-14910.patch
>
>
> Have seen this fail in multiple runs (not consistently)
> e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-11-02 Thread Barna Zsombor Klara (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628390#comment-15628390
 ] 

Barna Zsombor Klara commented on HIVE-14910:


Yes, currently the listener is only used in unit tests.

> Flaky test: TestSparkClient.testJobSubmission
> -
>
> Key: HIVE-14910
> URL: https://issues.apache.org/jira/browse/HIVE-14910
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14910.1.patch, HIVE-14910.2.patch, HIVE-14910.patch
>
>
> Have seen this fail in multiple runs (not consistently)
> e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-10-31 Thread Rui Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622330#comment-15622330
 ] 

Rui Li commented on HIVE-14910:
---

It makes sense to me. Have a question: is the Listener only used for tests?

> Flaky test: TestSparkClient.testJobSubmission
> -
>
> Key: HIVE-14910
> URL: https://issues.apache.org/jira/browse/HIVE-14910
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14910.1.patch, HIVE-14910.patch
>
>
> Have seen this fail in multiple runs (not consistently)
> e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-10-28 Thread Xuefu Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615727#comment-15615727
 ] 

Xuefu Zhang commented on HIVE-14910:


Minor comment on RB. Changes look good to me, but I wonder if [~lirui] has any 
thought on this. Thanks.

> Flaky test: TestSparkClient.testJobSubmission
> -
>
> Key: HIVE-14910
> URL: https://issues.apache.org/jira/browse/HIVE-14910
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14910.1.patch, HIVE-14910.patch
>
>
> Have seen this fail in multiple runs (not consistently)
> e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-10-26 Thread Barna Zsombor Klara (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15607924#comment-15607924
 ] 

Barna Zsombor Klara commented on HIVE-14910:


Hi [~xuefuz],
based on the commit history you seem very knowledgeable in this part of the 
code, would you be so kind to take a quick look at this patch?
My initial plan was only to fix the flaky test, but I ended up doing a small 
refactoring.

> Flaky test: TestSparkClient.testJobSubmission
> -
>
> Key: HIVE-14910
> URL: https://issues.apache.org/jira/browse/HIVE-14910
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14910.1.patch, HIVE-14910.patch
>
>
> Have seen this fail in multiple runs (not consistently)
> e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-10-25 Thread Barna Zsombor Klara (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15605352#comment-15605352
 ] 

Barna Zsombor Klara commented on HIVE-14910:


I'll steal this one [~sseth] if you don't mind.
I ran into this one today and had a look at it. I think the failure is coming 
from this race condition: the listener can be added to the JobHandle only after 
the job has been submitted. So there is no guarantee that every method of the 
listener will be invoked, some state changes may have happened before the 
caller received the handler back.

I propose a slight change in the API (because fixing the test is I think not 
possible). We should add the listeners as an argument of the submit method, so 
we can set them on the Handler before the job itself is submitted. In my 
opinion having a listener with methods which may or may not be invoked 
depending on a race condition is a problem anyway.

> Flaky test: TestSparkClient.testJobSubmission
> -
>
> Key: HIVE-14910
> URL: https://issues.apache.org/jira/browse/HIVE-14910
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>
> Have seen this fail in multiple runs (not consistently)
> e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)