Re: Review Request 43706: HIVE-13063 Create UDFs for CHAR and REPLACE

2016-03-03 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43706/
---

(Updated March 4, 2016, 7:25 a.m.)


Review request for hive, Jason Dere and Vikram Dixit Kumaraswamy.


Changes
---

Last patch after changing char(0) to nul character


Bugs: HIVE-13063
https://issues.apache.org/jira/browse/HIVE-13063


Repository: hive-git


Description
---

Create UDFS for these functions.

CHAR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If n 
is less than 0 or greater than 255, return the empty string. If n is 0, return 
null.

REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
Equals 'BLack and BLue'"


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d1e1441 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFChr.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFReplace.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFChr.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFReplace.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_chr.q PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_replace.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 789bedf 
  ql/src/test/results/clientpositive/udf_chr.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/udf_replace.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/43706/diff/


Testing
---

Verified that UDFs work.

ADD JAR hdfs://c6401.ambari.apache.org:8020/tmp/hive-exec.jar;
CREATE TEMPORARY FUNCTION char_udf AS 'org.apache.hadoop.hive.ql.udf.UDFChar';
CREATE TEMPORARY FUNCTION replace_udf AS 
'org.apache.hadoop.hive.ql.udf.UDFReplace';

SHOW FUNCTIONS;

DESCRIBE FUNCTION char_udf;
DESCRIBE FUNCTION replace_udf;

select char_udf(-1), 
char_udf(0), 
char_udf(1), 
char_udf(48), 
char_udf(65), 
char_udf(68.12), 
char_udf(32457964);

select replace_udf('', '', ''), 
replace_udf(null, '', ''), 
replace_udf('', null, ''), 
replace_udf('', '', null), 
replace_udf('Hack and Hue', 'H', 'BL'), 
replace_udf('ABABrdvABrk', 'AB', 'a');


File Attachments


Query1
  
https://reviews.apache.org/media/uploaded/files/2016/02/18/6f9c8b55-3e75-448f-bd71-93d6c3c900b8__Screen_Shot_2016-02-17_at_7.21.07_PM.png
Query2
  
https://reviews.apache.org/media/uploaded/files/2016/02/18/cedeeec0-388b-433b-a4f6-94888054d1a9__Screen_Shot_2016-02-17_at_7.20.57_PM.png


Thanks,

Alejandro Fernandez

Re: Review Request 44271: HIVE-12270 : Add DBTokenStore support to HS2 delegation token

2016-03-03 Thread Szehon Ho 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44271/#review122019
---



Yea I studied the code a little bit after our discussion, seems like if we just 
use raw MSC like I suggest then we wil never close it if DbTokenStore is 
invoked by thrift threads.. whereas Hive object does have a static close call 
later in the thread's life.

So let's go with the previous patch, it sucks to have to cache and pass the 
Hive object as its designed to be thread-local, but it's better than risking 
HMS leaks.

Sorry for the long discussion with you about it.

- Szehon Ho


On March 4, 2016, 3:24 a.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44271/
> ---
> 
> (Updated March 4, 2016, 3:24 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12270
> https://issues.apache.org/jira/browse/HIVE-12270
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add HMS APIs to support DB delegation token in HS2. 
> Only upload the patch without other thrift generated files for review here.
> 
> 
> Diffs
> -
> 
>   itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/MiniHiveKdc.java 
> dedbf35 
>   
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithDBTokenStore.java
>  PRE-CREATION 
>   
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
>  3ef2ce3 
>   metastore/if/hive_metastore.thrift e8f0a68 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> bfebfdc 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
> b5c4d1d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
> cb092d1 
>   service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 0c7455d 
>   shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java 
> de39d3d 
> 
> Diff: https://reviews.apache.org/r/44271/diff/
> 
> 
> Testing
> ---
> 
> Manuall tests
> New Unit test TestJdbcWithDBToken
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

[jira] [Created] (HIVE-13205) Job with last_value() function keep running forever.

2016-03-03 Thread Rahul (JIRA) 
Rahul created HIVE-13205:


 Summary: Job with last_value() function keep running forever.
 Key: HIVE-13205
 URL: https://issues.apache.org/jira/browse/HIVE-13205
 Project: Hive
  Issue Type: Bug
Reporter: Rahul






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 44271: HIVE-12270 : Add DBTokenStore support to HS2 delegation token

2016-03-03 Thread Chaoyu Tang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44271/
---

(Updated March 4, 2016, 3:24 a.m.)


Review request for hive.


Changes
---

Revised the patch not to pass around Hive Object but MSC instead based on 
review feedback. Thanks Szehon


Bugs: HIVE-12270
https://issues.apache.org/jira/browse/HIVE-12270


Repository: hive-git


Description
---

Add HMS APIs to support DB delegation token in HS2. 
Only upload the patch without other thrift generated files for review here.


Diffs (updated)
-

  itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/MiniHiveKdc.java 
dedbf35 
  
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithDBTokenStore.java
 PRE-CREATION 
  
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
 3ef2ce3 
  metastore/if/hive_metastore.thrift e8f0a68 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
bfebfdc 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
b5c4d1d 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
cb092d1 
  service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 0c7455d 
  shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java 
de39d3d 

Diff: https://reviews.apache.org/r/44271/diff/


Testing
---

Manuall tests
New Unit test TestJdbcWithDBToken
Precommit tests


Thanks,

Chaoyu Tang

Re: Review Request 44271: HIVE-12270 : Add DBTokenStore support to HS2 delegation token

2016-03-03 Thread Chaoyu Tang 


> On March 4, 2016, 12:37 a.m., Szehon Ho wrote:
> > service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java, line 129
> > 
> >
> > I'm a bit afraid of concurrency issues by caching Hive.  There seems to 
> > be quite some issues with Hive object lately and multi-threading and 
> > caching seems discouraged now (see HIVE-13002, HIVE-13194, HIVE-13150)
> > 
> > Can we have DbTokenStore get Hive on demand via thread-local when it 
> > needs it, say if hmsHandler is not passed in?
> > 
> > And also can you double-check if it will not leak, ie Hive object is 
> > closed somehow by the thread once its done?
> 
> Szehon Ho wrote:
> Or I guess ServerMode solves that as well.. we can just do Hive.get 
> instead of use the passed-in object if its HS2 mode?

Hi Szehon, Thanks for review. 
Yeah, initally I also thought to use Hive.get to get (actually initiate) a 
threadLocal Hive object and its contained MetaStoreClient in DBTokenStore. But 
I changed the idea because these token API calls usually happen before a 
session is opened, and the HMS connection opened for them is usually different 
from that used in session, since the HMS connection used later in the session 
may have a different user credential rather than the HS2 owner hive. So this 
HMS connection can not be reused in the session/queries. 
The Hive object now got and used for token APIs is HS2 server main thread 
local, its Hive.get is called in new HiveAuthFactory(hiveConf) <- 
ThriftBinaryCLIService.run() <-HiveServer2.start() HS2 thread, it won't be used 
by any session/queries which run in different threads. The only one other use 
of this Hive object (or MSC) is CLIService.applyAuthorizationConfigPolicy in 
CLIService.init during HS2 start, so this Hive object in the main thread is 
currently only used for token APIs during runtime, and it should not have 
concrruency issue. In addition, we only use MetaStoreClient in Hive object and 
not other instance variable values, and the MetaStoreClient is a synchronized 
client (See Hive getMSC method, 
HiveMetaStoreClient.newSynchronizedClient(metaStoreClient)), so I think passing 
in Hive (or HiveMetaStoreClient) and sharing it for token API calls should be 
quite safe and also save a lot of HMS connections. 
I wonder above consideration makes sense or not. Thanks.


- Chaoyu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44271/#review121974
---


On March 2, 2016, 4:39 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44271/
> ---
> 
> (Updated March 2, 2016, 4:39 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12270
> https://issues.apache.org/jira/browse/HIVE-12270
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add HMS APIs to support DB delegation token in HS2. 
> Only upload the patch without other thrift generated files for review here.
> 
> 
> Diffs
> -
> 
>   itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/MiniHiveKdc.java 
> dedbf35 
>   
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithDBTokenStore.java
>  PRE-CREATION 
>   
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
>  3ef2ce3 
>   metastore/if/hive_metastore.thrift e8f0a68 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> bfebfdc 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
> b5c4d1d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
> cb092d1 
>   service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 0c7455d 
>   shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java 
> de39d3d 
> 
> Diff: https://reviews.apache.org/r/44271/diff/
> 
> 
> Testing
> ---
> 
> Manuall tests
> New Unit test TestJdbcWithDBToken
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 44271: HIVE-12270 : Add DBTokenStore support to HS2 delegation token

2016-03-03 Thread Szehon Ho 


> On March 4, 2016, 12:37 a.m., Szehon Ho wrote:
> > service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java, line 129
> > 
> >
> > I'm a bit afraid of concurrency issues by caching Hive.  There seems to 
> > be quite some issues with Hive object lately and multi-threading and 
> > caching seems discouraged now (see HIVE-13002, HIVE-13194, HIVE-13150)
> > 
> > Can we have DbTokenStore get Hive on demand via thread-local when it 
> > needs it, say if hmsHandler is not passed in?
> > 
> > And also can you double-check if it will not leak, ie Hive object is 
> > closed somehow by the thread once its done?

Or I guess ServerMode solves that as well.. we can just do Hive.get instead of 
use the passed-in object if its HS2 mode?


- Szehon


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44271/#review121974
---


On March 2, 2016, 4:39 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44271/
> ---
> 
> (Updated March 2, 2016, 4:39 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12270
> https://issues.apache.org/jira/browse/HIVE-12270
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add HMS APIs to support DB delegation token in HS2. 
> Only upload the patch without other thrift generated files for review here.
> 
> 
> Diffs
> -
> 
>   itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/MiniHiveKdc.java 
> dedbf35 
>   
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithDBTokenStore.java
>  PRE-CREATION 
>   
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
>  3ef2ce3 
>   metastore/if/hive_metastore.thrift e8f0a68 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> bfebfdc 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
> b5c4d1d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
> cb092d1 
>   service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 0c7455d 
>   shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java 
> de39d3d 
> 
> Diff: https://reviews.apache.org/r/44271/diff/
> 
> 
> Testing
> ---
> 
> Manuall tests
> New Unit test TestJdbcWithDBToken
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 44271: HIVE-12270 : Add DBTokenStore support to HS2 delegation token

2016-03-03 Thread Szehon Ho 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44271/#review121974
---



Conceptually patch looks good, but had the following question below.


service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java (line 127)


I'm a bit afraid of concurrency issues by caching Hive.  There seems to be 
quite some issues with Hive object lately and multi-threading and caching seems 
discouraged now (see HIVE-13002, HIVE-13194, HIVE-13150)

Can we have DbTokenStore get Hive on demand via thread-local when it needs 
it, say if hmsHandler is not passed in?

And also can you double-check if it will not leak, ie Hive object is closed 
somehow by the thread once its done?


- Szehon Ho


On March 2, 2016, 4:39 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44271/
> ---
> 
> (Updated March 2, 2016, 4:39 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12270
> https://issues.apache.org/jira/browse/HIVE-12270
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add HMS APIs to support DB delegation token in HS2. 
> Only upload the patch without other thrift generated files for review here.
> 
> 
> Diffs
> -
> 
>   itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/MiniHiveKdc.java 
> dedbf35 
>   
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithDBTokenStore.java
>  PRE-CREATION 
>   
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
>  3ef2ce3 
>   metastore/if/hive_metastore.thrift e8f0a68 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> bfebfdc 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
> b5c4d1d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
> cb092d1 
>   service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 0c7455d 
>   shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java 
> de39d3d 
> 
> Diff: https://reviews.apache.org/r/44271/diff/
> 
> 
> Testing
> ---
> 
> Manuall tests
> New Unit test TestJdbcWithDBToken
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 44248: HIVE-12995 LLAP: Synthetic file ids need collision checks

2016-03-03 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44248/
---

(Updated March 3, 2016, 11:20 p.m.)


Review request for hive and Gopal V.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad 
  
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
 d33f724 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/Cache.java cee23a9 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
17d9fdf 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
1132171 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/NoopCache.java d0461e8 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
d2c1907 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
 b81e97d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 18191da 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 2597848 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 b36cf64 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 2e4e0c5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java
 43c8fb3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
 8479d22 
  orc/src/java/org/apache/orc/FileMetadata.java d63bdcc 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8 
  ql/src/java/org/apache/hadoop/hive/ql/io/SyntheticFileId.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4a27ee7 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
6cec80e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcBatchKey.java 
da673a5a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcCacheKey.java 9a0158e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 246ead6 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java b0ac503 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
4fafe8c 
  storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 9046589 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 3ef7abe 

Diff: https://reviews.apache.org/r/44248/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE

2016-03-03 Thread Gopal V (JIRA) 
Gopal V created HIVE-13204:
--

 Summary: Vectorization: Add ChainedCheckerFactory for LIKE 
 Key: HIVE-13204
 URL: https://issues.apache.org/jira/browse/HIVE-13204
 Project: Hive
  Issue Type: Improvement
  Components: Vectorization
Affects Versions: 2.1.0
Reporter: Gopal V
Assignee: Gopal V


Currently, Vectorization runs through a UTF-8 decode to produce a String & then 
check for mildly complex patterns like "http://%.exe; using a Regex.

Since this pattern doesn't need any backtracking patterns, using a full fledged 
Regex is too expensive.

!regex-cpu-like.png!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13203) STRUCT type not allowing more than 266 fields. Table get created with more fields but throws error while querying against the table.

2016-03-03 Thread Satyajit varma (JIRA) 
Satyajit varma created HIVE-13203:
-

 Summary: STRUCT type not allowing more than 266 fields. Table get 
created with more fields but throws error while querying against the table.
 Key: HIVE-13203
 URL: https://issues.apache.org/jira/browse/HIVE-13203
 Project: Hive
  Issue Type: Bug
  Components: Database/Schema
Reporter: Satyajit varma






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13202) LLAP: Replace use of ServerSocket with netty in LlapOutputFormatService

2016-03-03 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-13202:
-

 Summary: LLAP: Replace use of ServerSocket with netty in 
LlapOutputFormatService 
 Key: HIVE-13202
 URL: https://issues.apache.org/jira/browse/HIVE-13202
 Project: Hive
  Issue Type: Sub-task
Reporter: Jason Dere
Assignee: Jason Dere






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table

2016-03-03 Thread Wei Zheng (JIRA) 
Wei Zheng created HIVE-13201:


 Summary: Compaction shouldn't be allowed on non-ACID table
 Key: HIVE-13201
 URL: https://issues.apache.org/jira/browse/HIVE-13201
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 2.0.0
Reporter: Wei Zheng
Assignee: Wei Zheng


Looks like compaction is allowed on non-ACID table, although that's of no sense 
and does nothing. Moreover the compaction request will be enqueued into 
COMPACTION_QUEUE metastore table, which brings unnecessary overhead.
We should prevent compaction commands being allowed on non-ACID tables.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13200) Aggregation functions returning empty rows on partitioned columns

2016-03-03 Thread Yongzhi Chen (JIRA) 
Yongzhi Chen created HIVE-13200:
---

 Summary: Aggregation functions returning empty rows on partitioned 
columns
 Key: HIVE-13200
 URL: https://issues.apache.org/jira/browse/HIVE-13200
 Project: Hive
  Issue Type: Bug
  Components: Physical Optimizer
Affects Versions: 2.0.0, 1.0.0
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen


Running aggregation functions like MAX, MIN, DISTINCT against partitioned 
columns will return empty rows if table has property: 
'skip.header.line.count'='1'
Reproduce:
{noformat}
DROP TABLE IF EXISTS test;

CREATE TABLE test (a int) 
PARTITIONED BY (b int) 
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' 
TBLPROPERTIES('skip.header.line.count'='1');

INSERT OVERWRITE TABLE test PARTITION (b = 1) VALUES (1), (2), (3), (4);
INSERT OVERWRITE TABLE test PARTITION (b = 2) VALUES (1), (2), (3), (4);

SELECT * FROM test;

SELECT DISTINCT b FROM test;
SELECT MAX(b) FROM test;
SELECT DISTINCT a FROM test;
{noformat}

The output:
{noformat}
0: jdbc:hive2://localhost:1/default> SELECT * FROM test;
+-+-+--+
| test.a  | test.b  |
+-+-+--+
| 2   | 1   |
| 3   | 1   |
| 4   | 1   |
| 2   | 2   |
| 3   | 2   |
| 4   | 2   |
+-+-+--+
6 rows selected (0.631 seconds)

0: jdbc:hive2://localhost:1/default> SELECT DISTINCT b FROM test;
++--+
| b  |
++--+
++--+
No rows selected (47.229 seconds)

0: jdbc:hive2://localhost:1/default> SELECT MAX(b) FROM test;
+---+--+
|  _c0  |
+---+--+
| NULL  |
+---+--+
1 row selected (49.508 seconds)

0: jdbc:hive2://localhost:1/default> SELECT DISTINCT a FROM test;
++--+
| a  |
++--+
| 2  |
| 3  |
| 4  |
++--+
3 rows selected (46.859 seconds)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 44172: HIVE-11483: Add encoding and decoding for query string config

2016-03-03 Thread Amareshwari Sriramadasu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44172/#review121822
---


Ship it!




Make sure all occurrences of HiveConf.ConfVars.HIVEQUERYSTRING taken care, by 
doing a git grep.

- Amareshwari Sriramadasu


On March 3, 2016, 7:59 a.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44172/
> ---
> 
> (Updated March 3, 2016, 7:59 a.m.)
> 
> 
> Review request for hive and Alan Gates.
> 
> 
> Bugs: HIVE-11483
> https://issues.apache.org/jira/browse/HIVE-11483
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> We have seen some queries in production where some of the literals passed in 
> the query have control characters, which result in exception when query 
> string is set in the job xml.
> 
> Proposing a solution to encode the query string in configuration and provide 
> getters decoded string.
> 
> Here is a commit in a forked repo : 
> https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5
> 
> Suggestions are welcome on the solution.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 2723dadd76418f23b6e6358d21304e3312ef1cbd 
>   common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java 
> 365d5005d2a1f240b12bb4e9f9a920b3db4d32ba 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> 32531465cba95de6caa1bd292d489c1582efb942 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java 
> 68123d45c335e833984d3547607dc3ee7d8fd870 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 
> b184b4e554cf3f7a61932195b9b840cf13f3594b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 
> 473dbd61bf538fe08953dcf7994d5821f6bdbacc 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 70b2bc0cc15cd862cdb1adba0690d0530681ebde 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/errors/TestTaskLogProcessor.java 
> 67a86a60949a6b70f71593470a6d0ef8c265ee04 
>   ql/src/test/org/apache/hadoop/hive/ql/hooks/TestHooks.java 
> 8d27762522f0ac5f59696e019ec52a25e77318db 
> 
> Diff: https://reviews.apache.org/r/44172/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajat Khandelwal
> 
>

Re: Review Request 44201: HIVE-13188: Allow users of RetryingThriftClient to close transport

2016-03-03 Thread Amareshwari Sriramadasu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44201/#review121820
---


Ship it!




Ship It!

- Amareshwari Sriramadasu


On March 3, 2016, 6:50 a.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44201/
> ---
> 
> (Updated March 3, 2016, 6:50 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-13188
> https://issues.apache.org/jira/browse/HIVE-13188
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> RetryingThriftCLIClient opens a TTransport and leaves it open. there should 
> be a way to close that.
> 
> 
> Diffs
> -
> 
>   
> service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java
>  a81c600236e6e0f53a484806e559303279244616 
>   
> service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java
>  3798053a4a2e2b26b97b5accf96d73b58c89fb6e 
> 
> Diff: https://reviews.apache.org/r/44201/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajat Khandelwal
> 
>

Re: Review Request 44146: HIVE-13179: Allow custom HiveConf to be passed to Authentication Providers

2016-03-03 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44146/
---

(Updated March 3, 2016, 1:48 p.m.)


Review request for hive.


Bugs: HIVE-13179
https://issues.apache.org/jira/browse/HIVE-13179


Repository: hive-git


Description
---

Right now if I want to create an ldap auth provider, I have to create a 
hive-site.xml, set endpoints and other relevant properties there, then 
instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a 
new HiveConf is constructed. 

A better and more reusable design would be to ask for the conf in the 
constructor itself. That will allow an external user to create a HiveConf, set 
all relevant properties and instantiate `LdapAuthenticationProviderImpl` with 
that conf.


Diffs (updated)
-

  
service/src/java/org/apache/hive/service/auth/AuthenticationProviderFactory.java
 4b95503eb19cf862fb6efa614c9746b477337158 
  
service/src/java/org/apache/hive/service/auth/CustomAuthenticationProviderImpl.java
 3dc0aa86e2d4acc9604e27dabb8f683d4cd0705d 
  
service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java
 1d4aba2cc37930d87b620815fb317c1977f6e901 
  
service/src/java/org/apache/hive/service/auth/PamAuthenticationProviderImpl.java
 fd58081653af1a846394eddd513d94751e0129e2 
  
service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java
 e5cee37ac72001086323c94392274fb86f4307ef 
  
service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java
 5e91a0eb8a1bfc8a8d6e6774b195de9731035bdd 

Diff: https://reviews.apache.org/r/44146/diff/


Testing
---


Thanks,

Rajat Khandelwal

Re: Review Request 44172: HIVE-11483: Add encoding and decoding for query string config

2016-03-03 Thread Rajat Khandelwal 


> On March 3, 2016, noon, Amareshwari Sriramadasu wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 142
> > 
> >
> > Required any more?

Removed.


- Rajat


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44172/#review121807
---


On March 3, 2016, 1:29 p.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44172/
> ---
> 
> (Updated March 3, 2016, 1:29 p.m.)
> 
> 
> Review request for hive and Alan Gates.
> 
> 
> Bugs: HIVE-11483
> https://issues.apache.org/jira/browse/HIVE-11483
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> We have seen some queries in production where some of the literals passed in 
> the query have control characters, which result in exception when query 
> string is set in the job xml.
> 
> Proposing a solution to encode the query string in configuration and provide 
> getters decoded string.
> 
> Here is a commit in a forked repo : 
> https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5
> 
> Suggestions are welcome on the solution.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 2723dadd76418f23b6e6358d21304e3312ef1cbd 
>   common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java 
> 365d5005d2a1f240b12bb4e9f9a920b3db4d32ba 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> 32531465cba95de6caa1bd292d489c1582efb942 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java 
> 68123d45c335e833984d3547607dc3ee7d8fd870 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 
> b184b4e554cf3f7a61932195b9b840cf13f3594b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 
> 473dbd61bf538fe08953dcf7994d5821f6bdbacc 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 70b2bc0cc15cd862cdb1adba0690d0530681ebde 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/errors/TestTaskLogProcessor.java 
> 67a86a60949a6b70f71593470a6d0ef8c265ee04 
>   ql/src/test/org/apache/hadoop/hive/ql/hooks/TestHooks.java 
> 8d27762522f0ac5f59696e019ec52a25e77318db 
> 
> Diff: https://reviews.apache.org/r/44172/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajat Khandelwal
> 
>

Re: Review Request 44172: HIVE-11483: Add encoding and decoding for query string config

2016-03-03 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44172/
---

(Updated March 3, 2016, 1:29 p.m.)


Review request for hive and Alan Gates.


Bugs: HIVE-11483
https://issues.apache.org/jira/browse/HIVE-11483


Repository: hive-git


Description
---

We have seen some queries in production where some of the literals passed in 
the query have control characters, which result in exception when query string 
is set in the job xml.

Proposing a solution to encode the query string in configuration and provide 
getters decoded string.

Here is a commit in a forked repo : 
https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5

Suggestions are welcome on the solution.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
2723dadd76418f23b6e6358d21304e3312ef1cbd 
  common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java 
365d5005d2a1f240b12bb4e9f9a920b3db4d32ba 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
32531465cba95de6caa1bd292d489c1582efb942 
  ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java 
68123d45c335e833984d3547607dc3ee7d8fd870 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 
b184b4e554cf3f7a61932195b9b840cf13f3594b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 
473dbd61bf538fe08953dcf7994d5821f6bdbacc 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
70b2bc0cc15cd862cdb1adba0690d0530681ebde 
  ql/src/test/org/apache/hadoop/hive/ql/exec/errors/TestTaskLogProcessor.java 
67a86a60949a6b70f71593470a6d0ef8c265ee04 
  ql/src/test/org/apache/hadoop/hive/ql/hooks/TestHooks.java 
8d27762522f0ac5f59696e019ec52a25e77318db 

Diff: https://reviews.apache.org/r/44172/diff/


Testing
---


Thanks,

Rajat Khandelwal