[jira] [Created] (HIVE-16912) Improve table validator's performance against Oracle

2017-06-15 Thread Naveen Gangam (JIRA) 
Naveen Gangam created HIVE-16912:


 Summary: Improve table validator's performance against Oracle
 Key: HIVE-16912
 URL: https://issues.apache.org/jira/browse/HIVE-16912
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 3.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
Priority: Minor


Currently, this validator uses DatabaseMetaData.getTables() that takes in the 
order of minutes to return because of the number of SYSTEM tables present in 
Oracle.
Providing a schema name via a system property would limit the number of tables 
being returned and thus improve performance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16911) Upgrade groovy version to 2.4.11

2017-06-15 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-16911:
---

 Summary: Upgrade groovy version to 2.4.11
 Key: HIVE-16911
 URL: https://issues.apache.org/jira/browse/HIVE-16911
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


Hive currently uses groovy 2.4.4 which has security issue 
(https://access.redhat.com/security/cve/cve-2016-6814). Need to upgrade to 
2.4.8 or later. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16910) RpcConfiguration - Improper Cast From Long To Int

2017-06-15 Thread BELUGA BEHR (JIRA) 
BELUGA BEHR created HIVE-16910:
--

 Summary: RpcConfiguration - Improper Cast From Long To Int
 Key: HIVE-16910
 URL: https://issues.apache.org/jira/browse/HIVE-16910
 Project: Hive
  Issue Type: Bug
  Components: Spark
Affects Versions: 2.1.1, 3.0.0
Reporter: BELUGA BEHR
Priority: Minor


{code:title=org.apache.hive.spark.client.rpc.RpcConfiguration}
  long getConnectTimeoutMs() {
String value = 
config.get(HiveConf.ConfVars.SPARK_RPC_CLIENT_CONNECT_TIMEOUT.varname);
return value != null ? Integer.parseInt(value) : DEFAULT_CONF.getTimeVar(
  HiveConf.ConfVars.SPARK_RPC_CLIENT_CONNECT_TIMEOUT, 
TimeUnit.MILLISECONDS);
  }
{code}

This method returns a *long* but it is trying to parse the configuration string 
into an *int*.
 {{Integer.parseInt}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16909) Improve ndv estimates when ndvs are aggregated across partitions

2017-06-15 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-16909:
---

 Summary: Improve ndv estimates when ndvs are aggregated across 
partitions
 Key: HIVE-16909
 URL: https://issues.apache.org/jira/browse/HIVE-16909
 Project: Hive
  Issue Type: Improvement
  Components: Metastore, Statistics
Reporter: Ashutosh Chauhan


Tasks involved:
1. Switch over from FM-sketch to HLL bit vectors to compute ndvs.
2. Store these bit vectors in RDBMS metastore. This code already exists for 
HBase metastore.
3. Combine bit vectors requested for partition list to get better ndv estimate. 
This can be done initially only for CachedStore to avoid implementation 
complexity.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844

2017-06-15 Thread Sunitha Beeram (JIRA) 
Sunitha Beeram created HIVE-16908:
-

 Summary: Failures in TestHcatClient due to HIVE-16844
 Key: HIVE-16908
 URL: https://issues.apache.org/jira/browse/HIVE-16908
 Project: Hive
  Issue Type: Bug
Reporter: Sunitha Beeram
Assignee: Sunitha Beeram


Some of the tests in TestHCatClient.java, for ex:
{noformat}
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=177)
{noformat}

are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new 
configuration object is set on the ObjectStore. TestHCatClient fires up a 
second instance of metastore thread with a different conf object that results 
in the PersistenceMangaerFactory closure and hence tests fail. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 60116: HIVE-16885

2017-06-15 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60116/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16885
https://issues.apache.org/jira/browse/HIVE-16885


Repository: hive-git


Description
---

HIVE-16885


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
fce8db3df1026de8b6ee8c59567e55db40696217 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
6651900e79a5c3d4ad8329afbe3894544ce9f46e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 
07fd653dedc9a98d89b492ae6b49da70984569f7 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
737aad1b764ee6487b420f2b9ea651c42e08e9bf 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 fc6adafa0ebd0bd49d59cd0f4a82f70e9646ca6d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
9e84a29470c481d932d4f2d12e2898e05a925e5b 
  ql/src/test/queries/clientpositive/join47.q PRE-CREATION 
  ql/src/test/queries/clientpositive/mapjoin47.q PRE-CREATION 
  ql/src/test/queries/clientpositive/smb_mapjoin_47.q PRE-CREATION 
  ql/src/test/results/clientpositive/join47.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/mapjoin47.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/smb_mapjoin_47.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/60116/diff/1/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-16907) "INSERT INTO" overwrite old data when destination table encapsulated by backquote

2017-06-15 Thread Nemon Lou (JIRA) 
Nemon Lou created HIVE-16907:


 Summary:  "INSERT INTO"  overwrite old data when destination table 
encapsulated by backquote 
 Key: HIVE-16907
 URL: https://issues.apache.org/jira/browse/HIVE-16907
 Project: Hive
  Issue Type: Bug
  Components: Parser
Affects Versions: 2.1.1, 1.1.0
Reporter: Nemon Lou


A way to reproduce:
{noformat}
create database tdb;
use tdb;
create table t1(id int);
create table t2(id int);
explain insert into `tdb.t1` select * from t2;
{noformat}
{noformat}
+---+
|  Explain  
|
+---+
| STAGE DEPENDENCIES:   
|
|   Stage-1 is a root stage 
|
|   Stage-6 depends on stages: Stage-1 , consists of Stage-3, Stage-2, Stage-4  
|
|   Stage-3 
|
|   Stage-0 depends on stages: Stage-3, Stage-2, Stage-5
|
|   Stage-2 
|
|   Stage-4 
|
|   Stage-5 depends on stages: Stage-4  
|
|   
|
| STAGE PLANS:  
|
|   Stage: Stage-1  
|
| Map Reduce
|
|   Map Operator Tree:  
|
|   TableScan   
|
| alias: t2 
|
| Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column 
stats: NONE |
| Select Operator   
|
|   expressions: id (type: int) 
|
|   outputColumnNames: _col0
|
|   Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column 
stats: NONE   |
|   File Output Operator
|
| compressed: false 
|
| Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column 
stats: NONE |
| table:
|
| input format: 
org.apache.hadoop.hive.ql.io.RCFileInputFormat  
|
| output format: 
org.apache.hadoop.hive.ql.io.RCFileOutputFormat 
   |
|

Re: Review Request 60006: HIVE-14746: Remove branch and repositories from profiles by sending them from ptest-client

2017-06-15 Thread Barna Zsombor Klara 


> On June 14, 2017, 4:03 p.m., Sergio Pena wrote:
> > testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
> > Line 322 (original), 324 (patched)
> > 
> >
> > Should we check that BRANCH is not empty before starting the test? 
> > 
> > Is this a required argument or optional?
> > 
> > If it is optional, what branch will use as default? We're removing the 
> > branch option from the profiles, so this will not exist.

Based on the scripts and the properties it seems like PTest also supports SVN 
as the SCM system alongside Git, but there branch is a not supported parameter. 
If we want to keep the compatibility with SVN then we cannot make branch 
mandatory (since at this point we don't know the SCM type in the PTestClient).


> On June 14, 2017, 4:03 p.m., Sergio Pena wrote:
> > testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java
> > Lines 108 (patched)
> > 
> >
> > Why is a workingDirectoryWrapper needed?
> > 
> > Currently, Ptest works on a 'working' directory for any branch detected 
> > on the profile. Why do we want to change that for different branches?

I did this based on Siddharth Seth's comments on the Jira, where he mentioned 
that associating the working dir with the branch could prevent frequent 
rebasing when the checkout occurs. We don't need it for the original intent of 
the Jira. We can discuss if we want it or if the rebase would not be that 
problematic.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60006/#review177910
---


On June 13, 2017, 2:28 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60006/
> ---
> 
> (Updated June 13, 2017, 2:28 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Siddharth Seth.
> 
> 
> Bugs: HIVE-14746
> https://issues.apache.org/jira/browse/HIVE-14746
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14746: Remove branch and repositories from profiles by sending them from 
> ptest-client
> 
> 
> Diffs
> -
> 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
>  8e2604d372ac29b94445b269f08423b058308efe 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java
>  8deed52ae0307d4fc075654a4d75e6cb09a5d9db 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
>  b2c61f03c5bf5f170894141848c89fc26129115a 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
> 1cdfdb309acd8282e593abd7ed10c87721926c60 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/CloudExecutionContextProvider.java
>  8b82497bdaf43694e0e1552e125b5ffdce40f56c 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/ExecutionContext.java
>  b09de1d4d930cf2d4d26b500f3457cea3fffa9ce 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/FixedExecutionContextProvider.java
>  f7b50d6a61962d2727b5181215be9de2e64b05b7 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/WorkingDirWrapper.java
>  PRE-CREATION 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/api/server/TestTestExecutor.java
>  a4a789b579305d9ed573d8c1fd0b6ce75787d50f 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
>  848faf27af1ed8945d7013b6562bab544605e4bc 
> 
> 
> Diff: https://reviews.apache.org/r/60006/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested the PTestClient with and without the branch argument.
> Updated and ran the unit tests.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

[jira] [Created] (HIVE-16906) Hive ATSHook should check for yarn.timeline-service.enabled before connecting to ATS

2017-06-15 Thread Prabhu Joseph (JIRA) 
Prabhu Joseph created HIVE-16906:


 Summary: Hive ATSHook should check for 
yarn.timeline-service.enabled before connecting to ATS
 Key: HIVE-16906
 URL: https://issues.apache.org/jira/browse/HIVE-16906
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.2.2
Reporter: Prabhu Joseph


Hive ATShook has to check yarn.timeline-service.enabled (Indicate to clients 
whether timeline service is enabled or not. If enabled, clients will put 
entities and events to the timeline server.) before creating TimelineClient 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)