Re: Review Request 65018: HIVE-18372 Create testing infra to test different HMS instances

2018-01-12 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65018/
---

(Updated Jan. 12, 2018, 11:58 a.m.)


Review request for hive, Alan Gates, Marta Kuczora, Adam Szita, and Vihang 
Karajgaonkar.


Changes
---

Addressed Vihang's and Adam's comments


Bugs: HIVE-18372
https://issues.apache.org/jira/browse/HIVE-18372


Repository: hive-git


Description
---

Created:
- AbstractMetastore class - to privide an interface for different metastore 
implementation (start/stop/warehouse path methods)
-- Implementation for Embedded/Remote/Cluster metastores
- MiniHMS with builder - to create hms instances for test
- MetaStoreFactory - to create the parameter list for parametrized test
- TestDatabases - test for database related metastore functions to showcase the 
infrastructure


Diffs (updated)
-

  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/MetaStoreFactoryForTests.java
 PRE-CREATION 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
 PRE-CREATION 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/AbstractMetaStoreService.java
 PRE-CREATION 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/ClusterMetaStoreForTests.java
 PRE-CREATION 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/EmbeddedMetaStoreForTests.java
 PRE-CREATION 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/MiniHMS.java
 PRE-CREATION 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/RemoteMetaStoreForTests.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65018/diff/5/

Changes: https://reviews.apache.org/r/65018/diff/4-5/


Testing
---

Run the new tests


Thanks,

Peter Vary

Re: Review Request 65018: HIVE-18372 Create testing infra to test different HMS instances

2018-01-12 Thread Peter Vary via Review Board 


> On Jan. 10, 2018, 6:35 p.m., Vihang Karajgaonkar wrote:
> > Overall looks good. Thanks for taking this up. I suggested some small 
> > improvements to the tests below.

Thanks Adam and Vihang for the review!


> On Jan. 10, 2018, 6:35 p.m., Vihang Karajgaonkar wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
> > Lines 350 (patched)
> > 
> >
> > I think we should catch the exception and assert the exception type is 
> > InvalidOperationException so that we catch errors like if someone changes 
> > the thrown exception in the future.
> 
> Peter Vary wrote:
> Shall I separate the test case into two? With/Without cascade? There is a 
> little extra stuff there creating the table/function/index in two tests which 
> I wanted to avoid with one test case. That is why I decied to keep the 
> cascade tests in one test case and used try-catch here to check the exception 
> without the cascade option, and then proceed with the test and drop the 
> database with the cascade option.
> 
> What do you think?
> 
> Adam Szita wrote:
> I'd vote for separation. One test case for cascade and one for without 
> cascade. The latter should have an @Test(expected = ..) annotation IMHO

Since this seems like a consensus, spearated the tests :)


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65018/#review195158
---


On Jan. 12, 2018, 11:58 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65018/
> ---
> 
> (Updated Jan. 12, 2018, 11:58 a.m.)
> 
> 
> Review request for hive, Alan Gates, Marta Kuczora, Adam Szita, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-18372
> https://issues.apache.org/jira/browse/HIVE-18372
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Created:
> - AbstractMetastore class - to privide an interface for different metastore 
> implementation (start/stop/warehouse path methods)
> -- Implementation for Embedded/Remote/Cluster metastores
> - MiniHMS with builder - to create hms instances for test
> - MetaStoreFactory - to create the parameter list for parametrized test
> - TestDatabases - test for database related metastore functions to showcase 
> the infrastructure
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/MetaStoreFactoryForTests.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/AbstractMetaStoreService.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/ClusterMetaStoreForTests.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/EmbeddedMetaStoreForTests.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/MiniHMS.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/RemoteMetaStoreForTests.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65018/diff/5/
> 
> 
> Testing
> ---
> 
> Run the new tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Review Request 65130: HIVE-18350 : load data should rename files consistent with insert statements

2018-01-12 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65130/
---

Review request for hive, Eugene Koifman and Jason Dere.


Bugs: HIVE-18350
https://issues.apache.org/jira/browse/HIVE-18350


Repository: hive-git


Description
---

Made changes for both bucketed and non-bucketed tables.
Added a positive test for non-bucketed table which renames the loaded file.
Added couple of negative tests for bucketed table which reject a load with 
inconsistent file name.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1a2b3c1f6c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
4535c3edc2 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java 
cc1d8574b0 
  ql/src/test/queries/clientnegative/load_data_bucketed_1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/load_data_bucketed_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/load_data_rename.q PRE-CREATION 
  ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496 
  ql/src/test/results/clientnegative/load_data_bucketed_1.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/load_data_bucketed_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5 
  ql/src/test/results/clientpositive/llap/load_data_rename.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913 


Diff: https://reviews.apache.org/r/65130/diff/1/


Testing
---


Thanks,

Deepak Jaiswal

Re: Review Request 65075: HIVE-18426: Memory leak in RoutingAppender for every hive operation

2018-01-12 Thread kalyan kumar kalvagadda via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65075/
---

(Updated Jan. 12, 2018, 8:09 p.m.)


Review request for hive, Aihua Xu, Andrew Sherman, and Sergio Pena.


Changes
---

Addressed comments.


Bugs: HIVE-18426
https://issues.apache.org/jira/browse/HIVE-18426


Repository: hive-git


Description
---

Each new operation creates new entry in the ConcurrentMap in RoutingAppender 
but when the operation ends, AppenderControl stored in the map is retrieved and 
stopped but the entry in ConcurrentMap is never cleaned up.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/LogUtils.java 
0a3e0c72011951b6b1543352308bd51233c847fb 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java
 8febe3e79ff892c54b696b6c6ef92f7026c46033 


Diff: https://reviews.apache.org/r/65075/diff/2/

Changes: https://reviews.apache.org/r/65075/diff/1-2/


Testing
---

Made sure that the new tests updated to verify this change are passing.


Thanks,

kalyan kumar kalvagadda

[jira] [Created] (HIVE-18446) Flaky test - bucket_map_join_tez1

2018-01-12 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-18446:
-

 Summary: Flaky test - bucket_map_join_tez1
 Key: HIVE-18446
 URL: https://issues.apache.org/jira/browse/HIVE-18446
 Project: Hive
  Issue Type: Bug
  Components: Tests
Reporter: Jason Dere


Tried running this several times and see a mix of passes/failures.
Not sure what is going on here - looks like some operators/op trees change 
order, some slight stats differences, plus the shared work optimizer seems to 
work differently in that case. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 65084: HIVE-18430 Add new determinism category for runtime constants (current_date, current_timestamp)

2018-01-12 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65084/
---

(Updated Jan. 12, 2018, 7:33 p.m.)


Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.


Changes
---

Missed another use of the deterministic UDFType annotation in the constant 
propagation.


Bugs: HIVE-18430
https://issues.apache.org/jira/browse/HIVE-18430


Repository: hive-git


Description
---

Add a runtimeConstant field to UDFType. Also add a new method 
isConsistentWithinQuery() to FunctionRegistry/ExprNodeEvaluator, which 
indicates whether the UDF/expression returns a consistent value during the life 
of the query. Most existing calls to isDeterministic() should be switched to 
use isConsistentWithinQuery().

current_timestamp/current_date/current_user/current_database/logged_in_user are 
now tagged as non-determinstic, and runtime constants.
Jesus/Ashutosh: How does setting these functions to non-deterministic affect 
Calcite? It appears that for Calcite operators, there is a isDynamicFunction() 
which should return true in the case of runtime constants. Within Calcite 
should isDeterministic() return true or false for these functions?


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java 375d65f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java 
cc40cae 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 
8b9baa6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 7ca950d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java
 b0a2da8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/PrunerExpressionOperatorFactory.java
 306e714 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
 5b72dbd 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectRewriteRule.java
 a5d950a 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/PartitionPrune.java
 0e5e2b9 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
 13ee4e5 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
461dbe5 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 
ac37cc4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 5a88a96 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 67ea32c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 067fbe0 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerProcFactory.java d4df1e8 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java e265863 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFType.java ac3ec58 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java ef8dcf0 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentDate.java 
1f027a2 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentTimestamp.java
 2f13a22 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentUser.java 
d97583d 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLoggedInUser.java 
2915b86 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMacro.java 
3f505f2 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/UDFCurrentDB.java a5bab4f 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFunctionRegistry.java 3589938 


Diff: https://reviews.apache.org/r/65084/diff/2/

Changes: https://reviews.apache.org/r/65084/diff/1-2/


Testing
---


Thanks,

Jason Dere

[jira] [Created] (HIVE-18445) qtests: auto_join25.q fails permanently

2018-01-12 Thread Laszlo Bodor (JIRA) 
Laszlo Bodor created HIVE-18445:
---

 Summary: qtests: auto_join25.q fails permanently
 Key: HIVE-18445
 URL: https://issues.apache.org/jira/browse/HIVE-18445
 Project: Hive
  Issue Type: Bug
  Components: Tests
Reporter: Laszlo Bodor
Assignee: Laszlo Bodor


org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 65075: HIVE-18426: Memory leak in RoutingAppender for every hive operation

2018-01-12 Thread Andrew Sherman via Review Board 


> On Jan. 11, 2018, 9:51 p.m., Andrew Sherman wrote:
> > common/src/java/org/apache/hadoop/hive/common/LogUtils.java
> > Lines 259 (patched)
> > 
> >
> > Does deleteAppender() call stop() internally? If so then can we delete 
> > the previous call to stop the subordinateAppender?
> 
> kalyan kumar kalvagadda wrote:
> You are right. deleteAppender internallu calls stop() on the appender. we 
> need to explicitly look-up and stop it. I will update this in my next patch.

Thanks


> On Jan. 11, 2018, 9:51 p.m., Andrew Sherman wrote:
> > itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java
> > Line 162 (original), 163 (patched)
> > 
> >
> > This code was to check the case described in 
> > https://issues.apache.org/jira/browse/HIVE-17826?focusedCommentId=16208636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16208636
> > I think this is no longer tested. Is that OK?
> 
> kalyan kumar kalvagadda wrote:
> I have couple of observations with the limited knowledge that I have. 
> Correct me if i'm wrong here.
> 
> 1. HIVE-17128: Code changes done as part of this jira makes sure that 
> log4j Appender is closed when operation is closed to avoid file descriptors.
> 2. HIVE-17826: Added HushableRandomAccessFileAppender which is a copy of 
> RandomAccessFileAppender but has a explicit check in append() method to see 
> if the appender is closed.
> 
>  I think issue reported in HIVE-17826 is seen only because the 
> RoutingAppender is not properly cleaned-up when the operation is stopped. If 
> we have the patch i submitted we may not see the issue reported in HIVE-17826 
> even with out the fix of adding HushableRandomAccessFileAppender.

OK


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65075/#review195255
---


On Jan. 11, 2018, 2:11 p.m., kalyan kumar kalvagadda wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65075/
> ---
> 
> (Updated Jan. 11, 2018, 2:11 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, and Sergio Pena.
> 
> 
> Bugs: HIVE-18426
> https://issues.apache.org/jira/browse/HIVE-18426
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Each new operation creates new entry in the ConcurrentMap in RoutingAppender 
> but when the operation ends, AppenderControl stored in the map is retrieved 
> and stopped but the entry in ConcurrentMap is never cleaned up.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/LogUtils.java 
> 0a3e0c72011951b6b1543352308bd51233c847fb 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java
>  8febe3e79ff892c54b696b6c6ef92f7026c46033 
> 
> 
> Diff: https://reviews.apache.org/r/65075/diff/1/
> 
> 
> Testing
> ---
> 
> Made sure that the new tests updated to verify this change are passing.
> 
> 
> Thanks,
> 
> kalyan kumar kalvagadda
> 
>

Review Request 65132: HIVE-18422 : Vectorized input format should not be used when input format is excluded and row.serde is enabled

2018-01-12 Thread Vihang Karajgaonkar via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65132/
---

Review request for hive and Matt McCline.


Repository: hive-git


Description
---

HIVE-18422 : Vectorized input format should not be used when input format is 
excluded and row.serde is enabled


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
6f289703a70b2fc2c9cf6fc38707ae0fc4a68809 
  ql/src/test/queries/clientpositive/vectorization_input_format_excludes.q 
f8e82455e059a4ed25b1e1d2dce7f2fe0a6bffa3 
  
ql/src/test/results/clientpositive/llap/vectorization_input_format_excludes.q.out
 ab8752a8d22fb6288396896368124ff7c6561762 
  
ql/src/test/results/clientpositive/spark/vectorization_input_format_excludes.q.out
 887f44bb51665547530430a3ff2f3469ad68bebc 


Diff: https://reviews.apache.org/r/65132/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

[jira] [Created] (HIVE-18452) work around HADOOP-15171

2018-01-12 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-18452:
---

 Summary: work around HADOOP-15171
 Key: HIVE-18452
 URL: https://issues.apache.org/jira/browse/HIVE-18452
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 65148: HIVE-18452 work around HADOOP-15171

2018-01-12 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65148/
---

(Updated Jan. 13, 2018, 3:17 a.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0564b50276 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 1e0eccfa34 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
627e617d91 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java df536ead1c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
203ef69cf6 


Diff: https://reviews.apache.org/r/65148/diff/2/

Changes: https://reviews.apache.org/r/65148/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 65148: HIVE-18452 work around HADOOP-15171

2018-01-12 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65148/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0564b50276 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 1e0eccfa34 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
627e617d91 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java df536ead1c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
203ef69cf6 


Diff: https://reviews.apache.org/r/65148/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 65148: HIVE-18452 work around HADOOP-15171

2018-01-12 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65148/
---

(Updated Jan. 13, 2018, 3:14 a.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0564b50276 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 1e0eccfa34 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
627e617d91 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java df536ead1c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
203ef69cf6 


Diff: https://reviews.apache.org/r/65148/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 65148: HIVE-18452 work around HADOOP-15171

2018-01-12 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65148/
---

(Updated Jan. 13, 2018, 3:19 a.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0564b50276 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 1e0eccfa34 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
627e617d91 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java df536ead1c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
203ef69cf6 


Diff: https://reviews.apache.org/r/65148/diff/3/

Changes: https://reviews.apache.org/r/65148/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 65130: HIVE-18350 : load data should rename files consistent with insert statements

2018-01-12 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65130/
---

(Updated Jan. 13, 2018, 6:28 a.m.)


Review request for hive, Eugene Koifman and Jason Dere.


Changes
---

rebased the patch.


Bugs: HIVE-18350
https://issues.apache.org/jira/browse/HIVE-18350


Repository: hive-git


Description
---

Made changes for both bucketed and non-bucketed tables.
Added a positive test for non-bucketed table which renames the loaded file.
Added couple of negative tests for bucketed table which reject a load with 
inconsistent file name.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1a2b3c1f6c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
4535c3edc2 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java 
cc1d8574b0 
  ql/src/test/queries/clientnegative/load_data_bucketed_1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/load_data_bucketed_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/load_data_rename.q PRE-CREATION 
  ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496 
  ql/src/test/results/clientnegative/load_data_bucketed_1.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/load_data_bucketed_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5 
  ql/src/test/results/clientpositive/llap/load_data_rename.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913 


Diff: https://reviews.apache.org/r/65130/diff/2/

Changes: https://reviews.apache.org/r/65130/diff/1-2/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-18451) Support UDTF Operator in SimpleFetchOptimizer

2018-01-12 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-18451:
--

 Summary: Support UDTF Operator in SimpleFetchOptimizer
 Key: HIVE-18451
 URL: https://issues.apache.org/jira/browse/HIVE-18451
 Project: Hive
  Issue Type: Improvement
  Components: Physical Optimizer
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


After HIVE-18416, simple insert into table values{} is still resulting in 
launching of job as earlier which can be avoided by optimizing that job to 
FetchTask.

This issue is to extend SimpleFetchOptimizer, which does not fire on insert 
queries as it does not handle UDTF.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18450) Support TABLE function in CBO

2018-01-12 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-18450:
--

 Summary: Support TABLE function in CBO
 Key: HIVE-18450
 URL: https://issues.apache.org/jira/browse/HIVE-18450
 Project: Hive
  Issue Type: Bug
  Components: Logical Optimizer
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Follow-up of HIVE-18416 to support TABLE function in CBO.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18447) JDBC: Provide a way for JDBC users to pass cookie info via connection string

2018-01-12 Thread Vaibhav Gumashta (JIRA) 
Vaibhav Gumashta created HIVE-18447:
---

 Summary: JDBC: Provide a way for JDBC users to pass cookie info 
via connection string
 Key: HIVE-18447
 URL: https://issues.apache.org/jira/browse/HIVE-18447
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Reporter: Vaibhav Gumashta


Some authentication mechanisms like Single Sign On, need the ability to pass a 
cookie to some intermediate authentication service like Knox via the JDBC 
driver. We need to add the mechanism in Hive's JDBC driver (when used in HTTP 
transport mode).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18448) Drop Support For Indexes From Apache Hive

2018-01-12 Thread BELUGA BEHR (JIRA) 
BELUGA BEHR created HIVE-18448:
--

 Summary: Drop Support For Indexes From Apache Hive
 Key: HIVE-18448
 URL: https://issues.apache.org/jira/browse/HIVE-18448
 Project: Hive
  Issue Type: Improvement
  Components: Indexing
Reporter: BELUGA BEHR
Priority: Minor


If a user needs to look up a small subset of records quickly, they can use 
Apache HBase, if they need fast retrieval of larger sets of data, or fast 
joins, aggregations, they can use Apache Impala.  It seems to me that Hive 
indexes do not serve much of a role in the future of Hive.

Even without moving workloads to other products, columnar file formats with 
their statistics achieve similar goals as Hive indexes.

Please consider dropping Indexes from the Apache Hive project.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18449) Add configurable policy for choosing the HMS URI from hive.metastore.uris

2018-01-12 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-18449:
---

 Summary: Add configurable policy for choosing the HMS URI from 
hive.metastore.uris
 Key: HIVE-18449
 URL: https://issues.apache.org/jira/browse/HIVE-18449
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Sahil Takiar


HIVE-10815 added logic to randomly choose a HMS URI from 
{{hive.metastore.uris}}. It would be nice if there was a configurable policy 
that determined how a URI is chosen from this list - e.g. one option can be to 
randomly pick a URI, another option can be to choose the first URI in the list 
(which was the behavior prior to HIVE-10815).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)