[jira] [Created] (HIVE-14236) CTAS with UNION ALL puts the wrong stats + count(*) = 0

2016-07-13 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-14236:
--

 Summary: CTAS with UNION ALL puts the wrong stats + count(*) = 0
 Key: HIVE-14236
 URL: https://issues.apache.org/jira/browse/HIVE-14236
 Project: Hive
  Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


to repo. create table t as select * from src union all select * from src;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14235) create shaded standalone jdbc jar

2016-07-13 Thread Thejas M Nair (JIRA) 
Thejas M Nair created HIVE-14235:


 Summary: create shaded standalone jdbc jar 
 Key: HIVE-14235
 URL: https://issues.apache.org/jira/browse/HIVE-14235
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Reporter: Thejas M Nair


The jdbc jar includes several libs including ones for http, thrift etc.
When it is used in other applications, it can conflict with the libraries used 
by the application.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 49288: HIVE-11402 HS2 - disallow parallel query execution within a single Session

2016-07-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49288/
---

(Updated July 14, 2016, 1:01 a.m.)


Review request for hive and Thejas Nair.


Repository: hive-git


Description
---

.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bae1825 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
b0fa98f 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestHiveSessionImpl.java
 d58a913 
  
service/src/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
 ff46ed8 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 021c1fe 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
e453354 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
7341635 
  
service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
 f7b3412 

Diff: https://reviews.apache.org/r/49288/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-14234) TestHiveMetaStorePartitionSpecs does not drop database created in this test causes other test failure

2016-07-13 Thread niklaus xiao (JIRA) 
niklaus xiao created HIVE-14234:
---

 Summary: TestHiveMetaStorePartitionSpecs does not drop database 
created in this test causes other test failure
 Key: HIVE-14234
 URL: https://issues.apache.org/jira/browse/HIVE-14234
 Project: Hive
  Issue Type: Bug
  Components: Tests
Affects Versions: 2.1.0, 1.3.0
Reporter: niklaus xiao
Assignee: niklaus xiao
Priority: Minor


TestHiveMetaStorePartitionSpecs creates a database named testpartitionspecs_db, 
but never drop it, sometimes causes TestObjectStore#testDatabaseOps failed:
{code}
testDatabaseOps(org.apache.hadoop.hive.metastore.TestObjectStore)  Time 
elapsed: 0.188 sec  <<< FAILURE!
java.lang.AssertionError: expected:<2> but was:<3>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps(TestObjectStore.java:120)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14233) Improve vectorization for ACID by eliminating row-by-row stitching

2016-07-13 Thread Saket Saurabh (JIRA) 
Saket Saurabh created HIVE-14233:


 Summary: Improve vectorization for ACID by eliminating row-by-row 
stitching
 Key: HIVE-14233
 URL: https://issues.apache.org/jira/browse/HIVE-14233
 Project: Hive
  Issue Type: New Feature
  Components: Transactions, Vectorization
Reporter: Saket Saurabh
Assignee: Saket Saurabh


This JIRA proposes to improve vectorization for ACID by eliminating row-by-row 
stitching when reading back ACID files. In the current implementation, a 
vectorized row batch is created by populating the batch one row at a time, 
before the vectorized batch is passed up along the operator pipeline. This 
row-by-row stitching limitation was because of the fact that the ACID 
insert/update/delete events from various delta files needed to be merged 
together before the actual version of a given row was found out. HIVE-14035 has 
enabled us to break away from that limitation by splitting ACID update events 
into a combination of delete+insert. In fact, it has now enabled us to create 
splits on delta files.
Building on top of HIVE-14035, this JIRA proposes to solve this earlier 
bottleneck in the vectorized code path for ACID by now directly reading row 
batches from the underlying ORC files and avoiding any stitching altogether. 
Once a row batch is read from the split (which may be on a base/delta file), 
the deleted rows will be found by cross-referencing them against a data 
structure that will just keep track of deleted events (found in the 
deleted_delta files). This will lead to a large performance gain when reading 
ACID files in vectorized fashion, while enabling further optimizations in 
future that can be done on top of that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14232) Kryo: Perflogger is not transient in MapJoinOperator

2016-07-13 Thread Gopal V (JIRA) 
Gopal V created HIVE-14232:
--

 Summary: Kryo: Perflogger is not transient in MapJoinOperator
 Key: HIVE-14232
 URL: https://issues.apache.org/jira/browse/HIVE-14232
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.2.0
Reporter: Gopal V
Assignee: Nita Dembla
Priority: Blocker


{code}
  private final PerfLogger perfLogger = SessionState.getPerfLogger();
{code}

This produces a new copy of HiveConf into the plan in serialized form.

{code}
classes (sun.misc.Launcher$AppClassLoader)
classloader (java.security.ProtectionDomain)
context (java.security.AccessControlContext)
acc (org.apache.hadoop.hive.ql.exec.UDFClassLoader)
classLoader (org.apache.hadoop.hive.conf.HiveConf)
conf (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics)
metrics 
(org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics$CodahaleMetricsScope)
openScopes (org.apache.hadoop.hive.ql.log.PerfLogger)
perfLogger 
(org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator)
childOperators 
(org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator)
childOperators (org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:366)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:307)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:606)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:87)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14231) timestamp support is limited to 4 digit year

2016-07-13 Thread Thejas M Nair (JIRA) 
Thejas M Nair created HIVE-14231:


 Summary: timestamp support is limited to 4 digit year
 Key: HIVE-14231
 URL: https://issues.apache.org/jira/browse/HIVE-14231
 Project: Hive
  Issue Type: Bug
  Components: Types
Reporter: Thejas M Nair


Hive doesn't handle timestamp type that have a year with more than 4 digits.
This limitation seems to be primarily around string to timestamp conversion.

{code}
Following insert query would insert NULL record -
create table ts_test (t timestamp);
insert into ts_test values ('2015-01-01 1:1:1');
insert into ts_test values ('20151-01-01 1:1:1');
select CAST(t as String)  from ts_test;
+--+--+
|  t   |
+--+--+
| 2015-01-01 01:01:01  |
| NULL |
+--+--+
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14230) Hadoop23Shims.cloneUgi() doesn't add credentials from original UGI

2016-07-13 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-14230:
-

 Summary: Hadoop23Shims.cloneUgi() doesn't add credentials from 
original UGI
 Key: HIVE-14230
 URL: https://issues.apache.org/jira/browse/HIVE-14230
 Project: Hive
  Issue Type: Bug
  Components: Shims
Reporter: Jason Dere
Assignee: Jason Dere


Hadoop23Shims.cloneUgi() creates a Subject using the default constructor, 
leaving the newly created subject with empty credentials.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 49288: HIVE-11402 HS2 - disallow parallel query execution within a single Session

2016-07-13 Thread Thejas Nair 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49288/#review142125
---




itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java (line 
332)


can you also add a comment about what it is testing



service/src/java/org/apache/hive/service/cli/operation/Operation.java (line 303)


The background thread in SQLOperation also calls these register/unregister 
methods. (there is probably some scope for cleanup there, specially with the 
new method in hivesession)

Remove the TODO ?


- Thejas Nair


On July 13, 2016, 8:57 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49288/
> ---
> 
> (Updated July 13, 2016, 8:57 p.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bae1825 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> b0fa98f 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestHiveSessionImpl.java
>  d58a913 
>   
> service/src/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
>  ff46ed8 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 021c1fe 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> e453354 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 78ff388 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 7341635 
>   
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
>  f7b3412 
> 
> Diff: https://reviews.apache.org/r/49288/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Asking for permissions to edit wiki

2016-07-13 Thread Tao Li 
Hi,

I am from Hortonworks and just started working on Hive. I would like to be 
granted permissions to edit the wiki page so that I can make some contributions 
there. Please help. Thanks.

Re: Review Request 49288: HIVE-11402 HS2 - disallow parallel query execution within a single Session

2016-07-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49288/
---

(Updated July 13, 2016, 8:57 p.m.)


Review request for hive and Thejas Nair.


Repository: hive-git


Description
---

.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bae1825 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
b0fa98f 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestHiveSessionImpl.java
 d58a913 
  
service/src/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
 ff46ed8 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 021c1fe 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
e453354 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
7341635 
  
service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
 f7b3412 

Diff: https://reviews.apache.org/r/49288/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-14229) the jars in hive.aux.jar.paths are not added to HiveServer2 classpath

2016-07-13 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-14229:
---

 Summary: the jars in hive.aux.jar.paths are not added to 
HiveServer2 classpath 
 Key: HIVE-14229
 URL: https://issues.apache.org/jira/browse/HIVE-14229
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


The jars in hive.reloadable.aux.jar.paths are being added to HiveServer2 
classpath while hive.aux.jar.paths is not. 

Then the local task like 'select udf(x) from src' will fail to find needed udf 
class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 49728: HIVE-14172 LLAP: force evict blocks by size to handle memory fragmentation

2016-07-13 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49728/#review142109
---


Ship it!





llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 (line 228)


From my paraphrashing.

The new sublist extracted from the locked list will not be accessible to 
any future traversals.

The spliced sub-list disappears for future reads, which is how this does 
not have a race condition.


- Gopal V


On July 6, 2016, 9:31 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49728/
> ---
> 
> (Updated July 6, 2016, 9:31 p.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
> 47325ad 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapCacheableBuffer.java
>  5c0b6f3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
>  4def4a1 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java
>  acbaf85 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
>  0838682 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
>  5a0b27f 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/MemoryManager.java 
> 6cc262e 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java
>  345f5b1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
>  0846db9 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelLrfuCachePolicy.java
>  616c040 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
>  40edb28 
> 
> Diff: https://reviews.apache.org/r/49728/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 49728: HIVE-14172 LLAP: force evict blocks by size to handle memory fragmentation

2016-07-13 Thread Sergey Shelukhin 


> On July 13, 2016, 4:54 p.m., Gopal V wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java,
> >  line 229
> > 
> >
> > Race condition?

hmm? :)


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49728/#review142076
---


On July 6, 2016, 9:31 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49728/
> ---
> 
> (Updated July 6, 2016, 9:31 p.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
> 47325ad 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapCacheableBuffer.java
>  5c0b6f3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
>  4def4a1 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java
>  acbaf85 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
>  0838682 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
>  5a0b27f 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/MemoryManager.java 
> 6cc262e 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java
>  345f5b1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
>  0846db9 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelLrfuCachePolicy.java
>  616c040 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
>  40edb28 
> 
> Diff: https://reviews.apache.org/r/49728/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 49728: HIVE-14172 LLAP: force evict blocks by size to handle memory fragmentation

2016-07-13 Thread Sergey Shelukhin 


> On July 13, 2016, 4:54 p.m., Gopal V wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java,
> >  line 227
> > 
> >
> > The assumption about dropping locked blocks - does it ever happen that 
> > another thread has a valid reference to that block when this is hit?

LRFU policy does not need references for locked blocks because unlocking the 
block adds it back to the list/heap. However, this (removal) is not strictly 
enforced to avoid waiting/traversing when locking ie from processing thread 
every time. So the thread that already has to traverse removes them. It's not 
an eviction (eviction calls invalidate())


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49728/#review142076
---


On July 6, 2016, 9:31 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49728/
> ---
> 
> (Updated July 6, 2016, 9:31 p.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
> 47325ad 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapCacheableBuffer.java
>  5c0b6f3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
>  4def4a1 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java
>  acbaf85 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
>  0838682 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
>  5a0b27f 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/MemoryManager.java 
> 6cc262e 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java
>  345f5b1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
>  0846db9 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelLrfuCachePolicy.java
>  616c040 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
>  40edb28 
> 
> Diff: https://reviews.apache.org/r/49728/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 49739: HIVE-14187 : JDOPersistenceManager objects remain cached if MetaStoreClient#close is not called

2016-07-13 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49739/#review142085
---


Ship it!




Ship It!

- Vaibhav Gumashta


On July 7, 2016, 6:34 p.m., Mohit Sabharwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49739/
> ---
> 
> (Updated July 7, 2016, 6:34 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-14187
> https://issues.apache.org/jira/browse/HIVE-14187
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Instead of relying the client to call close, automatically perform RawStore 
> related cleanup at the server end via 
> deleteContext() which gets called when the server detects a lost/closed 
> connection.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
>  641abab71ad51748689a78e1e9ccb41aacdb9ace 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java
>  6da516558f6cf9a643d33f55cce31def9b7abc91 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestSetUGIOnOnlyClient.java
>  2c6d56782219fba26a878497860e2b5e047beafa 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestSetUGIOnOnlyServer.java
>  6c3fbf62c927644087478c4c7cce26cc144da501 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> f45b90d2f6bc9d853b895a837938a15f7d45a1e1 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
> 44d73d482263bc7f3c210503ba9440a6cd793a72 
> 
> Diff: https://reviews.apache.org/r/49739/diff/
> 
> 
> Testing
> ---
> 
> Added unit test
> 
> 
> Thanks,
> 
> Mohit Sabharwal
> 
>

Re: Review Request 49728: HIVE-14172 LLAP: force evict blocks by size to handle memory fragmentation

2016-07-13 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49728/#review142076
---




llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 (line 226)


The assumption about dropping locked blocks - does it ever happen that 
another thread has a valid reference to that block when this is hit?



llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 (line 228)


Race condition?


- Gopal V


On July 6, 2016, 9:31 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49728/
> ---
> 
> (Updated July 6, 2016, 9:31 p.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
> 47325ad 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapCacheableBuffer.java
>  5c0b6f3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
>  4def4a1 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java
>  acbaf85 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
>  0838682 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
>  5a0b27f 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/MemoryManager.java 
> 6cc262e 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java
>  345f5b1 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
>  0846db9 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelLrfuCachePolicy.java
>  616c040 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
>  40edb28 
> 
> Diff: https://reviews.apache.org/r/49728/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-14228) Better row count estimates for outer join during physical planning

2016-07-13 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-14228:
---

 Summary: Better row count estimates for outer join during physical 
planning
 Key: HIVE-14228
 URL: https://issues.apache.org/jira/browse/HIVE-14228
 Project: Hive
  Issue Type: Improvement
  Components: Physical Optimizer
Affects Versions: 2.1.0, 2.0.0, 1.2.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Currently, row counts for all join types are estimated as if they are outer 
join. We need to update that logic to take into account different join types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 49288: HIVE-11402 HS2 - disallow parallel query execution within a single Session

2016-07-13 Thread Thejas Nair 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49288/#review142070
---


Ship it!




Ship It!

- Thejas Nair


On July 11, 2016, 8:12 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49288/
> ---
> 
> (Updated July 11, 2016, 8:12 p.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 283ef2e 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestHiveSessionImpl.java
>  d58a913 
>   
> service/src/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
>  ff46ed8 
>   
> service/src/java/org/apache/hive/service/cli/operation/MetadataOperation.java 
> 44463c9 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> ba034ab 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 28c4553 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 78ff388 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 7341635 
>   
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
>  f7b3412 
> 
> Diff: https://reviews.apache.org/r/49288/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-14227) Investigate invalid SessionHandle and invalid OperationHandle

2016-07-13 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-14227:
---

 Summary: Investigate invalid SessionHandle and invalid 
OperationHandle
 Key: HIVE-14227
 URL: https://issues.apache.org/jira/browse/HIVE-14227
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


There are the following warnings. 

{noformat}
WARN  org.apache.hive.service.cli.thrift.ThriftCLIService: 
[HiveServer2-Handler-Pool: Thread-55]: Error executing statement:
org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle: 
SessionHandle [1bc00251-64e9-4a95-acb7-a7f53f773528]
at 
org.apache.hive.service.cli.session.SessionManager.getSession(SessionManager.java:318)
at 
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
{noformat}

{noformat}
WARN  org.apache.hive.service.cli.thrift.ThriftCLIService: 
[HiveServer2-Handler-Pool: Thread-1060]: Error closing operation:
org.apache.hive.service.cli.HiveSQLException: Invalid OperationHandle: 
OperationHandle [opType=EXECUTE_STATEMENT, 
getHandleIdentifier()=13d930dd-316c-4c09-9f44-fee5f483e73d]
at 
org.apache.hive.service.cli.operation.OperationManager.getOperation(OperationManager.java:185)
at 
org.apache.hive.service.cli.CLIService.closeOperation(CLIService.java:408)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.CloseOperation(ThriftCLIService.java:664)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1513)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1498)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14226) Invalid check on an ASTNode#toStringTree in CalcitePlanner

2016-07-13 Thread Barna Zsombor Klara (JIRA) 
Barna Zsombor Klara created HIVE-14226:
--

 Summary: Invalid check on an ASTNode#toStringTree in CalcitePlanner
 Key: HIVE-14226
 URL: https://issues.apache.org/jira/browse/HIVE-14226
 Project: Hive
  Issue Type: Bug
Reporter: Barna Zsombor Klara
Priority: Minor


The patch for HIVE-12590 has modified the string tree representation of an 
ASTNode, so that everything except string literals are now lowercase.
However CalcitePlanner#genSelectLogicalPlan is still checking the occurrence of 
an uppercase TOK_FUNCTIONDI constant in the string tree.
(Line 2997 in CalcitePlanner on master)

I think this check will never evaluate to true, so we should either be using a 
lowercase String, or remove the check if it is unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 49952: HIVE-14205: Hive doesn't support union type with AVRO file format

2016-07-13 Thread Yibing Shi 


> On July 13, 2016, 11:11 a.m., Chaoyu Tang wrote:
> > Thanks [~yshi] for patch. It looks good. But I have a couple of questions:
> > It seems to me that the union in existing code is only used to support 
> > Nullable type in Avro, and has not been fully supported as a data type in 
> > general. This patch actually extends (or adds) this type support. 
> > So with the patch, how can we be able to distinguish an Avro union between 
> > nullable and non-nullable, for example, for following field schema, both 
> > might end with type uniontype
> > {code}
> >   "fields":[
> >{
> >  "name":"value",
> >  "type":[
> > "null",
> >  "int",
> >  "long"
> >   ],
> >   "default":null
> > ]
> > ---
> >   "fields":[
> >{
> >  "name":"value",
> >  "type":[
> >  "int",
> >  "long"
> >   ],
> >   "default": 0
> > ]
> > {code}
> > Will there be any problem? Also could we add some qtests using Avro union 
> > data (with or without null)?

Hi [~ctang], thanks for the review!
Your concern about that both nullable and non-nullable avro union may end with 
same union type in Hive is very sound. My understanding is that every column in 
Hive is nullalbe (there isn't any key word like "not null" or "primary key" in 
Hive). As a result, schema ["null", "int", "long"] should always be used in 
favor of ["int", "long"]. The latter is supported by Hive just for better 
compatibility. So, it should be OK to map both ["null", "int", "long"] and 
["int", "long"] to "uniontype"
Please let me know your opinions.

I will try to add qtests as you suggested.


- Yibing


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49952/#review141997
---


On July 12, 2016, 9:07 p.m., Yibing Shi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49952/
> ---
> 
> (Updated July 12, 2016, 9:07 p.m.)
> 
> 
> Review request for hive and Chaoyu Tang.
> 
> 
> Bugs: HIVE-14205
> https://issues.apache.org/jira/browse/HIVE-14205
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14205: Hive doesn't support union type with AVRO file format
> 
> 
> Diffs
> -
> 
>   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 
> 6165138 
>   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 
> 08ee62b 
>   serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java 
> 986b803 
>   serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerdeUtils.java 
> 0013b78 
> 
> Diff: https://reviews.apache.org/r/49952/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Yibing Shi
> 
>

Review Request 49984: HIVE-14215: Displaying inconsistent CPU usage data with MR execution engine

2016-07-13 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49984/
---

Review request for hive, Aihua Xu and Sergio Pena.


Bugs: HIVE-14215
https://issues.apache.org/jira/browse/HIVE-14215


Repository: hive-git


Description
---

Reordered the time caluclation, so the displayed data is consistent now


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 
cfb4a28 

Diff: https://reviews.apache.org/r/49984/diff/


Testing
---

Manually.

Should have to put sleep at the end of the while cycle, to consistently 
reproduce the problem.
Keeping the same sleep there, the displayed data is consistent with the patch.

Sleep removed from the code in the end :)

If you have any idea, how to produce a reliable test case for this, I am open 
for every suggestions.

Thanks,
Peter


Thanks,

Peter Vary

Re: Review Request 49655: HIVE-12646: beeline and HIVE CLI do not parse ; in quote properly

2016-07-13 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49655/#review142036
---



Thanks for your patch!
This was a very annoying "feature" :)

As discussed offline, please look at DDLTask.java where the SHOW CREATE TABLE 
command is implemented. There is an escapeHiveCommand method, which escapes the 
result which is expected to be copy/pasted back to command line and executed 
without change.

Another thing your patch should consider, that the comment fields could contain 
escaped ' characters, like the one below:

```
create table escape_comments_tbl1 (col1 string comment 'ab\';\');
```

Thanks,
Peter

- Peter Vary


On July 12, 2016, 4:04 a.m., Sahil Takiar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49655/
> ---
> 
> (Updated July 12, 2016, 4:04 a.m.)
> 
> 
> Review request for hive, Sergio Pena and Yongzhi Chen.
> 
> 
> Bugs: HIVE-12646
> https://issues.apache.org/jira/browse/HIVE-12646
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12646: beeline and HIVE CLI do not parse ; in quote properly
> 
> Approach:
> 
>   * Modified the `Commands.execute(...)` command to iterate throught the 
> given input line character by character
>   * It looks for single and double quotes in order to track when the iterator 
> is inside a quotation block
>   * If the iterator is inside a quotation block and it finds a semicolon, it 
> ignores it, otherwise it treats it as it normally would
>   * Moved the logic for parsing the line into a helper method called 
> `getCmList(...)` which is responsible for returning a `List` of commands that 
> need to be run
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 3a204c0 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java
>  ecfeddb 
> 
> Diff: https://reviews.apache.org/r/49655/diff/
> 
> 
> Testing
> ---
> 
> Add a unit tests which checks that Beeline can successfully run queries that 
> contain semi-colons inside quotation blocks. Confirmed existing unit tests 
> pass.
> 
> 
> Thanks,
> 
> Sahil Takiar
> 
>

[jira] [Created] (HIVE-14225) Llap slider package should support configuring YARN rolling log aggregation

2016-07-13 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-14225:
-

 Summary: Llap slider package should support configuring YARN 
rolling log aggregation
 Key: HIVE-14225
 URL: https://issues.apache.org/jira/browse/HIVE-14225
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14224) LLAP rename query specific log files once a query is complete

2016-07-13 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-14224:
-

 Summary: LLAP rename query specific log files once a query is 
complete
 Key: HIVE-14224
 URL: https://issues.apache.org/jira/browse/HIVE-14224
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Once a query is complete, rename the query specific log file so that YARN can 
aggregate the logs (once it's configured to do so).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)