Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

2018-04-12 Thread Thejas Nair 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/#review201050
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2914 (patched)


It would be good to add these to "hive.conf.restricted.list" values



metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql
Lines 1306 (patched)


this is not easy to read where the parenthesis start and end, can you 
reformat with some indendation to make it easier to read ?



ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLs.java
Lines 61 (patched)


EnumMap could be used for efficiency here



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
Lines 51 (patched)


the if/else can be replaced with -

enabled = new BooleanWritable(HiveConf.getBoolVar(hiveConf, 
ConfVars.HIVE_SERVER2_RESTRICT_INFORMATION_SCHEMA)



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 896 (patched)


does it have to also have dynamic sevice discovery enabled ?
We could assume that if HIVE_SERVER2_PRIVILEGE_SYNCHRONIZER=true and 
namespace is set, zookeeper is setup appropriately.



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 905 (patched)


call variable authorizer (not authorizor) ?


- Thejas Nair


On April 11, 2018, 10:33 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66571/
> ---
> 
> (Updated April 11, 2018, 10:33 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-19161
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  801de7a 
>   itests/hive-unit/pom.xml f473d25 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
>  PRE-CREATION 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLs.java
>  53e221f 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  450da4f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  ebbf465 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  b2c40c2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  2056930 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  f6c46ee 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  c47856d 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  304f567 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  85c6727 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
>  7d37262 
> 
> 
> Diff: https://reviews.apache.org/r/66571/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

[jira] [Created] (HIVE-19198) Few flaky hcatalog tests

2018-04-12 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-19198:
---

 Summary: Few flaky hcatalog tests
 Key: HIVE-19198
 URL: https://issues.apache.org/jira/browse/HIVE-19198
 Project: Hive
  Issue Type: Sub-task
Reporter: Ashutosh Chauhan


TestPermsGrp : Consider removing this since hcat cli is not widely used.
TestHCatPartitionPublish.testPartitionPublish
TestHCatMultiOutputFormat.testOutputFormat



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 64511: HIVE-18252 Limit the size of the object inspector caches

2018-04-12 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64511/
---

(Updated April 12, 2018, 11:45 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

New approach:
- Remove caching for the complex object inspectors (list/map/union/struct).
- This requires implementing equals()/hashcode() for the complex object 
inspectors, as well as for the constant object inspectors.


Bugs: HIVE-18252
https://issues.apache.org/jira/browse/HIVE-18252


Repository: hive-git


Description
---

Patch to limit size of the object inspector cache for complex types, and to not 
cache complex object inspectors that contain constants.


Diffs (updated)
-

  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
 c0f9d08e48 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java
 c0cdae4d32 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java
 55b9fc8c2f 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantStructObjectInspector.java
 cf0be75d48 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardListObjectInspector.java
 da35c5c901 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardMapObjectInspector.java
 de41b97bbd 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardStructObjectInspector.java
 3de9a2ee31 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardUnionObjectInspector.java
 002514f304 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantBinaryObjectInspector.java
 2a4c6d9577 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantBooleanObjectInspector.java
 d52b278bcc 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantByteObjectInspector.java
 e24ed7c480 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantDateObjectInspector.java
 bd86c2225d 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantDoubleObjectInspector.java
 dcfdc142f6 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantFloatObjectInspector.java
 c1bd97689c 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantHiveCharObjectInspector.java
 d04140fd9f 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantHiveDecimalObjectInspector.java
 92f80982dd 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantHiveVarcharObjectInspector.java
 e470f06148 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantIntObjectInspector.java
 c0ff664f9a 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantLongObjectInspector.java
 3fccd3c3aa 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantShortObjectInspector.java
 6460318118 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantStringObjectInspector.java
 7be737ba75 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaConstantTimestampObjectInspector.java
 2453bc67cb 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantBinaryObjectInspector.java
 b5582a4fab 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantBooleanObjectInspector.java
 6e8152bec3 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java
 7c88d4461d 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantDateObjectInspector.java
 290fcd3ca3 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantDoubleObjectInspector.java
 09c2a514be 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantFloatObjectInspector.java
 f02bf3e995 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantHiveCharObjectInspector.java
 0746e4afb2 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantHiveDecimalObjectInspector.java
 4e1d970eca 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantHiveIntervalDayTimeObjectInspector.java
 6ef81189fc 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantHiveIntervalYearMonthObjectInspector.java
 a2f97dad95 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantHiveVarcharObjectInspector.java
 76cd40369d

[jira] [Created] (HIVE-19197) TestReplicationScenarios is flaky

2018-04-12 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-19197:
---

 Summary: TestReplicationScenarios is flaky
 Key: HIVE-19197
 URL: https://issues.apache.org/jira/browse/HIVE-19197
 Project: Hive
  Issue Type: Sub-task
  Components: repl, Test
Reporter: Ashutosh Chauhan


Fails once in a while.
{code}
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyResults(TestReplicationScenarios.java:3629)
at 
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyRun(TestReplicationScenarios.java:3711)
at 
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyRun(TestReplicationScenarios.java:3706)
at 
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyAndReturnTblReplStatus(TestReplicationScenarios.java:3600)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19196) TestTriggersMoveWorkloadManager is flaky

2018-04-12 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-19196:
---

 Summary: TestTriggersMoveWorkloadManager is flaky
 Key: HIVE-19196
 URL: https://issues.apache.org/jira/browse/HIVE-19196
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Ashutosh Chauhan


This is a flaky test which randomly fails. Consider improving its stability.
{code}

org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill

Failing for the past 1 build (Since Failed#10161 )
Took 2.4 sec.
Error Message
'"eventType" : "GET"' expected in STDERR capture, but not found.
Stacktrace
java.lang.AssertionError: '"eventType" : "GET"' expected in STDERR capture, but 
not found.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hive.jdbc.AbstractJdbcTriggersTest.runQueryWithTrigger(AbstractJdbcTriggersTest.java:169)
at 
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill(TestTriggersMoveWorkloadManager.java:261)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19195) .testCliDriver[auto_sortmerge_join_2] is flaky

2018-04-12 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-19195:
---

 Summary: .testCliDriver[auto_sortmerge_join_2] is flaky
 Key: HIVE-19195
 URL: https://issues.apache.org/jira/browse/HIVE-19195
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Ashutosh Chauhan


This test is certainly flaky. Seems like it makes some assumption about 
available memory. Consider dropping this altogether.

 {code}
Client Execution succeeded but contained differences (error code = 1) after 
executing auto_sortmerge_join_2.q 
1101,1103d1100
< Hive Runtime Error: Map local work exhausted memory
< FAILED: Execution Error, return code 3 from 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
< ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTask
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19194) TestDruidStorageHandler fails

2018-04-12 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-19194:
---

 Summary: TestDruidStorageHandler fails
 Key: HIVE-19194
 URL: https://issues.apache.org/jira/browse/HIVE-19194
 Project: Hive
  Issue Type: Sub-task
  Components: Druid integration
Reporter: Ashutosh Chauhan


This tests fails randomly. If its not reproducible locally consider improving 
its stability since it does fail once in a while on Hive QA. 

{code}

java.lang.AssertionError: expected:<0> but was:<1> at 
org.junit.Assert.fail(Assert.java:88) at 
org.junit.Assert.failNotEquals(Assert.java:743) at 
org.junit.Assert.assertEquals(Assert.java:118) at 
org.junit.Assert.assertEquals(Assert.java:555) at 
org.junit.Assert.assertEquals(Assert.java:542) at 
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitMultiInsertOverwriteTable(TestDruidStorageHandler.java:414)

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19193) TestActivePassiveHA fails

2018-04-12 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-19193:
---

 Summary: TestActivePassiveHA fails
 Key: HIVE-19193
 URL: https://issues.apache.org/jira/browse/HIVE-19193
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2
Reporter: Ashutosh Chauhan


This looks like a flaky test. If its not reproducible locally try improving on 
stability of this test, since it does fail randomly with Hive QA.
{code:java}
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover(TestActivePassiveHA.java:356)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

2018-04-12 Thread Thejas Nair 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/#review201038
---




metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql
Line 1079 (original), 1079 (patched)


some use cases of sys is quoted while others are not



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java
Lines 38 (patched)


I think something on lines of following better describes it - 
"Returns all groups the current user belongs to"



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 6066 (patched)


This revokes all privileges for the db/table, not just the ones in 
revokePrivileges call.
In that case, I am wondering if it should be passed list 
instead of PrivilegeBag ?
That would make it more clear about the purpose.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 6108 (patched)


can you add a comment for this block (ie optimization done in that) ?


- Thejas Nair


On April 11, 2018, 10:33 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66571/
> ---
> 
> (Updated April 11, 2018, 10:33 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-19161
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  801de7a 
>   itests/hive-unit/pom.xml f473d25 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
>  PRE-CREATION 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLs.java
>  53e221f 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  450da4f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  ebbf465 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  b2c40c2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  2056930 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  f6c46ee 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  c47856d 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  304f567 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  85c6727 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
>  7d37262 
> 
> 
> Diff: https://reviews.apache.org/r/66571/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

[jira] [Created] (HIVE-19192) HiveServer2 query compilation : query compilation time increases sql has multiple unions

2018-04-12 Thread Rajkumar Singh (JIRA) 
Rajkumar Singh created HIVE-19192:
-

 Summary: HiveServer2 query compilation : query compilation time 
increases sql has multiple unions 
 Key: HIVE-19192
 URL: https://issues.apache.org/jira/browse/HIVE-19192
 Project: Hive
  Issue Type: Improvement
  Components: Hive, HiveServer2
Affects Versions: 2.1.0, 1.2.1
 Environment: Hive-1.2.1

Hive-2.1.0

 
Reporter: Rajkumar Singh
 Attachments: query-with-100-union.q, query-with-200-union.q, 
query-with-50-union.q

query compilation time suffer a lot if SQL has many unions, here is the simple 
reproduce of the problem. PFA attached query with 50,100 and 200 unions(forgive 
me for this bad SQL). when run explain against hiveserver2 I can see the 
compilation time increase many folds.

{code}

query-with-50-union.q
1,671 rows selected (10.662 seconds)

query-with-100-union.q
3,321 rows selected (101.709 seconds)

query-with-200-union.q
6,588 rows selected (1074.487 seconds)

{code}

Running such SQL against hiveserver2 can starve other SQL to run into single 
threaded compilation stage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66503: HIVE-19126: CachedStore: Use memory estimation to limit cache size during prewarm

2018-04-12 Thread Thejas Nair 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/#review201041
---


Ship it!




Ship It!

- Thejas Nair


On April 11, 2018, 10:53 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66503/
> ---
> 
> (Updated April 11, 2018, 10:53 p.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Bugs: HIVE-19126
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
>  6f4ec6f1ea 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
>  2f7fa24558 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  0bbaf7e459 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  c47856de87 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  89b400697b 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  f007261daf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66503/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 66368: HIVE-18609: Results cache invalidation based on table updates

2018-04-12 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66368/
---

(Updated April 12, 2018, 8:09 p.m.)


Review request for hive, Gopal V and Jesús Camacho Rodríguez.


Changes
---

- When removing invalid entries during lookup, make sure we have exited read 
lock section.
- Add results_cache_transactional.q to testconfiguration.properties


Bugs: HIVE-18609
https://issues.apache.org/jira/browse/HIVE-18609


Repository: hive-git


Description
---

- Save ValidTxnWriteIdList when saving query to the results cache.
- Compare the write ID list for each transactional table during results cache 
lookup.
- Add configuration to determine if queries with non-transactional tables 
should be cached.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e540d023bd 
  itests/src/test/resources/testconfiguration.properties 48d62a8bf9 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java a88453c978 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
b1a3646624 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 44a7496136 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 10982ddbd1 
  ql/src/test/queries/clientpositive/results_cache_1.q 4aea60e1e5 
  ql/src/test/queries/clientpositive/results_cache_2.q 96a90925f6 
  ql/src/test/queries/clientpositive/results_cache_capacity.q 9f54577009 
  ql/src/test/queries/clientpositive/results_cache_empty_result.q 621367141e 
  ql/src/test/queries/clientpositive/results_cache_invalidation.q PRE-CREATION 
  ql/src/test/queries/clientpositive/results_cache_lifetime.q 60ffe96a04 
  ql/src/test/queries/clientpositive/results_cache_quoted_identifiers.q 
4802f43ba9 
  ql/src/test/queries/clientpositive/results_cache_temptable.q 9e0de765cb 
  ql/src/test/queries/clientpositive/results_cache_transactional.q PRE-CREATION 
  ql/src/test/queries/clientpositive/results_cache_with_masking.q b4fcdd57eb 
  ql/src/test/results/clientpositive/llap/results_cache_invalidation.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/results_cache_transactional.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/results_cache_invalidation.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/results_cache_transactional.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/66368/diff/4/

Changes: https://reviews.apache.org/r/66368/diff/3-4/


Testing
---

qtests added.


Thanks,

Jason Dere

[jira] [Created] (HIVE-19191) Assertion error while running materialized view rewriting

2018-04-12 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-19191:
--

 Summary: Assertion error while running materialized view rewriting
 Key: HIVE-19191
 URL: https://issues.apache.org/jira/browse/HIVE-19191
 Project: Hive
  Issue Type: Bug
  Components: Materialized views
Affects Versions: 3.0.0
Reporter: Nita Dembla
Assignee: Jesus Camacho Rodriguez


{code:sql}
jdbc:hive2://localhost:10007/tpcds_bin_par> explain select 
w_warehouse_name,avg(inv_quantity_on_hand)  from inventory ,warehouse where 
inv_warehouse_sk = w_warehouse_sk group by w_warehouse_name;

Error: Error running query: java.lang.AssertionError: rel 
[rel#3663:HiveProject.HIVE.[](input=rel#3662:Subset#8.HIVE.[],$f0=$0,$f1=/($1, 
$2))] has lower cost {1.248 rows, 1.248 cpu, 0.0 io} than best cost 
{8.963453683875E8 rows, 7.967514387E8 cpu, 0.0 io} of subset 
[rel#3664:Subset#9.HIVE.[]] (state=,code=0)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66585: HIVE-19104 When test MetaStore is started with retry the instances should be independent

2018-04-12 Thread Sahil Takiar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66585/#review201027
---


Ship it!




Ship It!

- Sahil Takiar


On April 12, 2018, 4:29 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66585/
> ---
> 
> (Updated April 12, 2018, 4:29 p.m.)
> 
> 
> Review request for hive, Aihua Xu and Sahil Takiar.
> 
> 
> Bugs: HIVE-19104
> https://issues.apache.org/jira/browse/HIVE-19104
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The main goal here is to have independent warehouse dirs, and jdbc urls when 
> metastores are started with MetaStoreTestUtils.startMetaStoreWithRetry
> 
> This revealed several issues:
> - HCatalog tests (TestPermsGrp.java, TestHCatMultiOutputFormat.java, 
> TestHCatPartitionPublish.java) still used the default configurations, so the 
> new configs should be set
> - MiniHS2 warehouse directory has to be updated after the metastore is started
> - Warehouse directory permissions had to be set to 777
> - One of the tests in TestHiveMetaStore.java modifies the configuration 
> object, so it has to be copied
> 
> 
> Diffs
> -
> 
>   hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/TestPermsGrp.java 
> 8a2151c 
>   
> hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java
>  d9de10e 
>   
> hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitionPublish.java
>  fb6a7f4 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 23fcbe8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  f007261 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/MetaStoreTestUtils.java
>  1d12cf9 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
>  6f52a52 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java
>  e1f81bb 
> 
> 
> Diff: https://reviews.apache.org/r/66585/diff/2/
> 
> 
> Testing
> ---
> 
> Run the previously failed tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 66585: HIVE-19104 When test MetaStore is started with retry the instances should be independent

2018-04-12 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66585/
---

(Updated April 12, 2018, 4:29 p.m.)


Review request for hive, Aihua Xu and Sahil Takiar.


Changes
---

MetaStoreTestUtil generates its' own JDBC url, and the URL generation is 
removed from the MiniHS2 code


Bugs: HIVE-19104
https://issues.apache.org/jira/browse/HIVE-19104


Repository: hive-git


Description
---

The main goal here is to have independent warehouse dirs, and jdbc urls when 
metastores are started with MetaStoreTestUtils.startMetaStoreWithRetry

This revealed several issues:
- HCatalog tests (TestPermsGrp.java, TestHCatMultiOutputFormat.java, 
TestHCatPartitionPublish.java) still used the default configurations, so the 
new configs should be set
- MiniHS2 warehouse directory has to be updated after the metastore is started
- Warehouse directory permissions had to be set to 777
- One of the tests in TestHiveMetaStore.java modifies the configuration object, 
so it has to be copied


Diffs (updated)
-

  hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/TestPermsGrp.java 
8a2151c 
  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java
 d9de10e 
  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitionPublish.java
 fb6a7f4 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 23fcbe8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 f007261 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/MetaStoreTestUtils.java
 1d12cf9 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
 6f52a52 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java
 e1f81bb 


Diff: https://reviews.apache.org/r/66585/diff/2/

Changes: https://reviews.apache.org/r/66585/diff/1-2/


Testing
---

Run the previously failed tests


Thanks,

Peter Vary

[jira] [Created] (HIVE-19190) Improve Logging for SemanticException Handling

2018-04-12 Thread BELUGA BEHR (JIRA) 
BELUGA BEHR created HIVE-19190:
--

 Summary: Improve Logging for SemanticException Handling
 Key: HIVE-19190
 URL: https://issues.apache.org/jira/browse/HIVE-19190
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 2.3.2, 3.0.0
Reporter: BELUGA BEHR
 Attachments: table_not_found_example.txt

Please improve the logging for queries that fail with SemanticException.  For 
example, when performing an action on a table that does not exist.  The most 
common reason why this happens is that someone fat fingers the table/column 
name.  It is not a system error pe se, but a user validation error.  This is 
not something the cluster administrator should have to worry about.  Yet, Hive 
performs some pretty extreme logging on the matter.  I have attached to this 
JIRA the logging produced by a single submission of the following query:

{code:sql} 
select * from madeup';
{code}
 
For SemanticException exceptions, please print the Exception {{getMessage()}} 
to the server INFO logging so that the query's life-cycle can be traced, but do 
not blast ERRORs and stack traces to the log file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66586: HIVE-12192

2018-04-12 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66586/
---

(Updated April 12, 2018, 12:11 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-12192
https://issues.apache.org/jira/browse/HIVE-12192


Repository: hive-git


Description
---

HIVE-12192


Diffs
-

  
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/mr/TestHiveAccumuloTypes.java
 926f5720ac6ff0cfaf3858f2eba4e9e4ef37a889 
  common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java 
90ffddba0dd6c608857ee645023532de86c731b8 
  common/src/java/org/apache/hive/common/util/DateParser.java 
949fdbafcfd599e47889297955ed1e39b4f747b5 
  common/src/java/org/apache/hive/common/util/TimestampParser.java 
f674b5d30bd8addde1dad70dba3553651c0bea60 
  common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java 
5a3f0481bc23e807f1f5e7b12cb9e5422b7f0e4a 
  common/src/test/org/apache/hive/common/util/TestDateParser.java 
0553b3d38718b4b4fabc808a607a98e3aa00efd4 
  common/src/test/org/apache/hive/common/util/TestTimestampParser.java 
c982af65c6228bf515aa06c2b586a7651e100ccc 
  druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidSerDe.java 
d991adb088b9f5149e1759d1d495e3bda5f5907d 
  druid-handler/src/test/org/apache/hadoop/hive/druid/serde/TestDruidSerDe.java 
e4fa1a2e59bd1e1b4f7ac0cec573886240450738 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
bc4e1466f513bc1d009f6ce64ce24e1cae592e44 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
 05cc30a62137733e7e94cb8d61c79503ec4f16e2 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java 
d94dbe8d8aadc78cfc9b60799165f378dcc8486b 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/DataType.java 
6dcee4024ba16da30fb55a8c07a2439292324115 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/JsonSerDe.java 
114c205c83cefa23827af4072d0dc1dc4056d83c 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/ReaderWriter.java 
d2954e030a8d7fe195c913590e6d3ad69e8e0ff4 
  
hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/HCatBaseStorer.java
 9fca565ca4e79efbc61e13f91b75579aa5545835 
  
itests/hive-jmh/src/main/java/org/apache/hive/benchmark/vectorization/ColumnVectorGenUtil.java
 d80b6d43fe44b19401a79d5190935bbabb9d522e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsUpdateTask.java 
a7465a7ab6a5cefb064b54366f70048d88191e08 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
d59bf1fb6e62a821d79c7d4040bdfb7b43cd3ba5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/TimestampUtils.java 
58252e1b7f40a699966909b5306e4632999e1b55 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAssignRow.java 
e96619cf86c9a802994513ee5bccecef6f58b0c6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorBatchDebug.java 
47eaf367ea48174a7d8ef7f95184ec6a10677d4d 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorColumnAssignFactory.java
 c5554642800950956a78fc6bda78e3398fe68777 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorDeserializeRow.java 
8ea625e46605b73b57281c83c839de40ef8c0031 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExtractRow.java 
152d75b9490a1676331a1ac59b14e231b854918c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSerializeRow.java 
cb2efb7dbe270539bd7e398d4a42a80042d48b3c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
abbbe9a42bf91969a1f80eaec533c872d4448248 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 
d92ec320b58c5fb81b07fa0be2ca3519ccfc280f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java 
6588385b9fdb882ac25f99402e7098d3d95d7081 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastDateToString.java
 e559886659a467b7dadf966dee94c9f06a80929b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastDecimalToTimestamp.java
 8107c446227d442d00d4c3e52a69f7ca2dd5f413 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastDoubleToTimestamp.java
 549457921e63185f8c71c5ab738db55ba869877d 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastStringToDate.java
 b55712a89e39fd28c93e33f28a555b2bca7f3507 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastTimestampToChar.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastTimestampToString.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastTimestampToVarChar.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java
 252a8163d958d8d61452ad64f51d229342a50f08

Review Request 66586: HIVE-12192

2018-04-12 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66586/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-12192
https://issues.apache.org/jira/browse/HIVE-12192


Repository: hive-git


Description
---

HIVE-12192


Diffs
-

  
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/mr/TestHiveAccumuloTypes.java
 926f5720ac6ff0cfaf3858f2eba4e9e4ef37a889 
  common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java 
90ffddba0dd6c608857ee645023532de86c731b8 
  common/src/java/org/apache/hive/common/util/DateParser.java 
949fdbafcfd599e47889297955ed1e39b4f747b5 
  common/src/java/org/apache/hive/common/util/TimestampParser.java 
f674b5d30bd8addde1dad70dba3553651c0bea60 
  common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java 
5a3f0481bc23e807f1f5e7b12cb9e5422b7f0e4a 
  common/src/test/org/apache/hive/common/util/TestDateParser.java 
0553b3d38718b4b4fabc808a607a98e3aa00efd4 
  common/src/test/org/apache/hive/common/util/TestTimestampParser.java 
c982af65c6228bf515aa06c2b586a7651e100ccc 
  druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidSerDe.java 
d991adb088b9f5149e1759d1d495e3bda5f5907d 
  druid-handler/src/test/org/apache/hadoop/hive/druid/serde/TestDruidSerDe.java 
e4fa1a2e59bd1e1b4f7ac0cec573886240450738 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
bc4e1466f513bc1d009f6ce64ce24e1cae592e44 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
 05cc30a62137733e7e94cb8d61c79503ec4f16e2 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java 
d94dbe8d8aadc78cfc9b60799165f378dcc8486b 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/DataType.java 
6dcee4024ba16da30fb55a8c07a2439292324115 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/JsonSerDe.java 
114c205c83cefa23827af4072d0dc1dc4056d83c 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/ReaderWriter.java 
d2954e030a8d7fe195c913590e6d3ad69e8e0ff4 
  
hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/HCatBaseStorer.java
 9fca565ca4e79efbc61e13f91b75579aa5545835 
  
itests/hive-jmh/src/main/java/org/apache/hive/benchmark/vectorization/ColumnVectorGenUtil.java
 d80b6d43fe44b19401a79d5190935bbabb9d522e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsUpdateTask.java 
a7465a7ab6a5cefb064b54366f70048d88191e08 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
d59bf1fb6e62a821d79c7d4040bdfb7b43cd3ba5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/TimestampUtils.java 
58252e1b7f40a699966909b5306e4632999e1b55 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAssignRow.java 
e96619cf86c9a802994513ee5bccecef6f58b0c6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorBatchDebug.java 
47eaf367ea48174a7d8ef7f95184ec6a10677d4d 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorColumnAssignFactory.java
 c5554642800950956a78fc6bda78e3398fe68777 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorDeserializeRow.java 
8ea625e46605b73b57281c83c839de40ef8c0031 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExtractRow.java 
152d75b9490a1676331a1ac59b14e231b854918c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSerializeRow.java 
cb2efb7dbe270539bd7e398d4a42a80042d48b3c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
abbbe9a42bf91969a1f80eaec533c872d4448248 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 
d92ec320b58c5fb81b07fa0be2ca3519ccfc280f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java 
6588385b9fdb882ac25f99402e7098d3d95d7081 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastDateToString.java
 e559886659a467b7dadf966dee94c9f06a80929b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastDecimalToTimestamp.java
 8107c446227d442d00d4c3e52a69f7ca2dd5f413 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastDoubleToTimestamp.java
 549457921e63185f8c71c5ab738db55ba869877d 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastStringToDate.java
 b55712a89e39fd28c93e33f28a555b2bca7f3507 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastTimestampToChar.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastTimestampToString.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastTimestampToVarChar.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DynamicValueVectorExpression.java
 252a8163d958d8d61452ad64f51d229342a50f08 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/TimestampToStringUnaryUDF.java

Review Request 66585: HIVE-19104 When test MetaStore is started with retry the instances should be independent

2018-04-12 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66585/
---

Review request for hive, Aihua Xu and Sahil Takiar.


Bugs: HIVE-19104
https://issues.apache.org/jira/browse/HIVE-19104


Repository: hive-git


Description
---

The main goal here is to have independent warehouse dirs, and jdbc urls when 
metastores are started with MetaStoreTestUtils.startMetaStoreWithRetry

This revealed several issues:
- HCatalog tests (TestPermsGrp.java, TestHCatMultiOutputFormat.java, 
TestHCatPartitionPublish.java) still used the default configurations, so the 
new configs should be set
- MiniHS2 warehouse directory has to be updated after the metastore is started
- Warehouse directory permissions had to be set to 777
- One of the tests in TestHiveMetaStore.java modifies the configuration object, 
so it has to be copied


Diffs
-

  hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/TestPermsGrp.java 
8a2151c 
  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java
 d9de10e 
  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitionPublish.java
 fb6a7f4 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 23fcbe8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 f007261 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/MetaStoreTestUtils.java
 1d12cf9 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
 6f52a52 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java
 e1f81bb 


Diff: https://reviews.apache.org/r/66585/diff/1/


Testing
---

Run the previously failed tests


Thanks,

Peter Vary

[jira] [Created] (HIVE-19189) Replication Resiliency: when hiveserver2 copies data to target

2018-04-12 Thread anishek (JIRA) 
anishek created HIVE-19189:
--

 Summary: Replication Resiliency: when hiveserver2 copies data to 
target
 Key: HIVE-19189
 URL: https://issues.apache.org/jira/browse/HIVE-19189
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: anishek


 For a copy if there is an exception (only get IOExcpetion), so network errors 
also will be reported as the same, we should create a new FileSystem object 
before we retry. We should also have a wait time before we retry as well. 
Additional condition to take care of is, the exception might have occurred, 
because the file was moved from original location to CM root.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19188) Replication Resiliency: for listing files for a partition / table

2018-04-12 Thread anishek (JIRA) 
anishek created HIVE-19188:
--

 Summary: Replication Resiliency: for listing files for a partition 
/  table
 Key: HIVE-19188
 URL: https://issues.apache.org/jira/browse/HIVE-19188
 Project: Hive
  Issue Type: Bug
Reporter: anishek


When for listing files for a partition /  table fails during dump we should 
have ability to *retry with wait,* so the whole dump process does not fail. 
additionally  the filesystem object used to list the files might have to be 
recreated since only IOException is thrown by the object for various operations 
and we dont know if the internal DFSClient it holds has a broken connection due 
to network error.  This has to be done using Filesystem.newInstance() since 
caching of fs object might be enabled.
currently caching of the filesystem object is depended upon
* Scheme
* Authority
* UGI
* And a long called unique defaults to 0 unless explicitly provided, which is 
never the case from Hive since we call default constructor which uses 0L by 
default internally.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)