[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-09-19 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504870#comment-15504870
 ] 

Sergey Shelukhin commented on HIVE-14651:
-

lgtm +1

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-09-19 Thread Prasanth Jayachandran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504277#comment-15504277
 ] 

Prasanth Jayachandran commented on HIVE-14651:
--

+1

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-09-19 Thread Prasanth Jayachandran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504276#comment-15504276
 ] 

Prasanth Jayachandran commented on HIVE-14651:
--

That's the one. Makes sense.

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-09-19 Thread Siddharth Seth (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504272#comment-15504272
 ] 

Siddharth Seth commented on HIVE-14651:
---

Which comment are you referring to? (The one which says "Force fs to file://, 
setup staging dir?)

The staging dir is setup within the Hive directories, otherwise Tez takes care 
of creating the dirs based on configuration. Once local mode works properly, 
these comments will get resolved (I'd like to allow local mode to work with 
either file:// or hdfs. Tez does not support hdfs with local mode yet)

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-09-19 Thread Prasanth Jayachandran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504223#comment-15504223
 ] 

Prasanth Jayachandran commented on HIVE-14651:
--

[~sseth] regarding your comments in the code. how does it work without setting 
up staging dir? Is there a default that tez AM sets up?

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-09-19 Thread Siddharth Seth (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504202#comment-15504202
 ] 

Siddharth Seth commented on HIVE-14651:
---

[~prasanth_j], [~sershe] - the test results look good now. Any other comments?

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-08-30 Thread Siddharth Seth (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450931#comment-15450931
 ] 

Siddharth Seth commented on HIVE-14651:
---

Hmm. I think you're right. Somehow missed that altogether. Will investigate 
some more. Really need to get the tests to be reliable - or at least close to 
reliable.

Local cluster - run everything in a single process. For LLAP - this would mean 
avoiding AM launches. For MiniTez, avoid AM launches and container launches. 
Everything runs inline.

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch, HIVE-14651.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-08-30 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450464#comment-15450464
 ] 

Sergey Shelukhin commented on HIVE-14651:
-

Well, complete failure of TestMiniSparkOnYarnCliDriver is suspicious. Some 
small comments on RB. Would be nice to describe what is local cluster, etc.

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch, HIVE-14651.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-08-30 Thread Siddharth Seth (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450383#comment-15450383
 ] 

Siddharth Seth commented on HIVE-14651:
---


I don't think the current set of failures are related. Have seen all of these 
fail before. Fixed hybridgraceHashjoin1.q from the last run.
Anyway, RB link is here: https://reviews.apache.org/r/51543/

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch, HIVE-14651.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-08-30 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450256#comment-15450256
 ] 

Sergey Shelukhin commented on HIVE-14651:
-

Failures may be related. Is it possible to post an RB?

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch, HIVE-14651.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-08-30 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450250#comment-15450250
 ] 

Hive QA commented on HIVE-14651:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12826225/HIVE-14651.04.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10407 tests 
executed
*Failed tests:*
{noformat}
TestClearDanglingScratchDir - did not produce a TEST-*.xml file
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1045/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1045/console
Test logs: 
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1045/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12826225 - PreCommit-HIVE-MASTER-Build

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch, HIVE-14651.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-08-27 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15441764#comment-15441764
 ] 

Hive QA commented on HIVE-14651:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825728/HIVE-14651.03.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10405 tests 
executed
*Failed tests:*
{noformat}
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[hybridgrace_hashjoin_1]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1021/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1021/console
Test logs: 
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1021/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12825728 - PreCommit-HIVE-MASTER-Build

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-08-26 Thread Prasanth Jayachandran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440097#comment-15440097
 ] 

Prasanth Jayachandran commented on HIVE-14651:
--

nit: LOG.error instead?

{code}
  try {
  fs.mkdirs(userInstallPath);
   } catch (IOException e) {
  LOG.info("Failed to create path={}. Continuing. Exception message={}", 
warehousePath,
  e.getMessage());
   }
{code}

There are 2 other places as well.

Other that that lgtm, +1 


> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, 
> HIVE-14651.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-08-26 Thread Siddharth Seth (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440077#comment-15440077
 ] 

Siddharth Seth commented on HIVE-14651:
---

It should not be. Will upload a patch with this removed.

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP

2016-08-26 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439586#comment-15439586
 ] 

Hive QA commented on HIVE-14651:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825619/HIVE-14651.02.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10404 tests 
executed
*Failed tests:*
{noformat}
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_join30]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1007/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1007/console
Test logs: 
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1007/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12825619 - PreCommit-HIVE-MASTER-Build

> Add a local cluster for Tez and LLAP
> 
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)