[jira] [Created] (HIVE-22019) alter_table_update_status/alter_table_update_status_disable_bitvector/alter_partition_update_status fail when DbNotificationListener is installed

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-22019:
-

 Summary: 
alter_table_update_status/alter_table_update_status_disable_bitvector/alter_partition_update_status
 fail when DbNotificationListener is installed
 Key: HIVE-22019
 URL: https://issues.apache.org/jira/browse/HIVE-22019
 Project: Hive
  Issue Type: Sub-task
Reporter: Daniel Dai


Statement like:
ALTER TABLE src_stat_n0 UPDATE STATISTICS for column key SET 
('numDVs'='','avgColLen'='1.111')
fail when DbNotificationListener is installed with the message:
{code}
See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or 
check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/ 
for specific test cases logs.
 org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.IllegalArgumentException: Could not serialize 
JSONUpdateTableColumnStatMessage : 
 at 
org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:5350)
 at 
org.apache.hadoop.hive.ql.exec.ColumnStatsUpdateTask.persistColumnStats(ColumnStatsUpdateTask.java:339)
 at 
org.apache.hadoop.hive.ql.exec.ColumnStatsUpdateTask.execute(ColumnStatsUpdateTask.java:347)
 at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
 at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
 at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2343)
 at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1995)
 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1662)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1422)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1416)
 at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
 at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
 at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
 at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:340)
 at 
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:680)
 at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:651)
 at 
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:182)
 at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
 at 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:59)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at 
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
 at org.junit.runners.Suite.runChild(Suite.java:127)
 at org.junit.runners.Suite.runChild(Suite.java:26)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
 at 
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:73)
 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
 at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
 at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159

[jira] [Created] (HIVE-22018) Add table id to HMS get methods

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-22018:
-

 Summary: Add table id to HMS get methods
 Key: HIVE-22018
 URL: https://issues.apache.org/jira/browse/HIVE-22018
 Project: Hive
  Issue Type: Sub-task
Reporter: Daniel Dai


It is possible we remove a table and immediately move another table to occupy 
the same name. CachedStore may retrieve the wrong table in this case. We shall 
add tableid in every get_(table/partition) api, so we can compare the one 
stored in TBLS (tableid is part of Table object) and check if the same id, if 
not, HMS shall fail the read request. The initial table id can be retrieved 
along with writeid (in DbTxnManager.getValidWriteIds call, to join the TBLS 
table)



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (HIVE-22017) HMS interface backward compatible after HIVE-21637

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-22017:
-

 Summary: HMS interface backward compatible after HIVE-21637
 Key: HIVE-22017
 URL: https://issues.apache.org/jira/browse/HIVE-22017
 Project: Hive
  Issue Type: Sub-task
Reporter: Daniel Dai


HIVE-21637 changes a bunch HMS interface to add writeid into all get_xxx calls. 
Ideally we shall provide original version and forward it to the new api to make 
the change backward compatible. The downside is double the size of HMS methods. 
We shall mark those deprecated and remove in future version.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (HIVE-22016) Do not open transaction for readonly query

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-22016:
-

 Summary: Do not open transaction for readonly query
 Key: HIVE-22016
 URL: https://issues.apache.org/jira/browse/HIVE-22016
 Project: Hive
  Issue Type: Sub-task
Reporter: Daniel Dai


Open/abort/commit transaction would increment transaction id which is a burden 
unnecessarily. In addition, it spams the notification log and make CachedStore 
(and of cause other components rely on notification log) harder to catch up.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (HIVE-22015) Cache table constraints in CachedStore

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-22015:
-

 Summary: Cache table constraints in CachedStore
 Key: HIVE-22015
 URL: https://issues.apache.org/jira/browse/HIVE-22015
 Project: Hive
  Issue Type: Sub-task
Reporter: Daniel Dai


Currently table constraints are not cached. Hive will pull all constraints from 
tables involved in query, which results multiple db reads (including 
get_primary_keys, get_foreign_keys, get_unique_constraints, etc). The effort to 
cache this is small as it's just another table component.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (HIVE-22014) Tear down locks in CachedStore

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-22014:
-

 Summary: Tear down locks in CachedStore
 Key: HIVE-22014
 URL: https://issues.apache.org/jira/browse/HIVE-22014
 Project: Hive
  Issue Type: Sub-task
Reporter: Daniel Dai


There's a lot of locks in CachedStore. After HIVE-21637, only notification log 
puller thread will update the cache. And when it process event, the first thing 
is to mark the entry invalid. The only exception may be 
TableWrapperSizeUpdater, but we can also make it synchronous (maybe run it once 
after every iteration of notification log puller). There should be no 
synchronization issue and we can tear down existing locks to simplify the code.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (HIVE-21697) Remove periodical full refresh in HMS cache

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21697:
-

 Summary: Remove periodical full refresh in HMS cache
 Key: HIVE-21697
 URL: https://issues.apache.org/jira/browse/HIVE-21697
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


In HIVE-18661, we added periodical notification based refresh in HMS cache. We 
shall remove periodical full refresh to simplify the code as it will no longer 
be used. In the mean time, we introduced mechanism to provide monotonic reads 
through the CachedStore.commitTransaction. This will no longer be needed after 
HIVE-21637. So I will remove related code as well. This will provide some 
performance benefits include:
1. We don't have to slow down write to catch up notification logs. Write can be 
done immediately and tag the cache with writeids
2. We can read from cache even if updateUsingNotificationEvents is running. 
Read will compare the writeids of the cache so monotonic reads will be 
guaranteed

I'd like to put a patch separately with HIVE-21637 so it can be tested 
independently. HMW will use periodical notification based refresh to update 
cache. And it will temporarily lift the monotonic reads guarantee until 
HIVE-21637 checkin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21637) Synchronized metastore cache

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21637:
-

 Summary: Synchronized metastore cache
 Key: HIVE-21637
 URL: https://issues.apache.org/jira/browse/HIVE-21637
 Project: Hive
  Issue Type: New Feature
Reporter: Daniel Dai
Assignee: Daniel Dai


Currently, HMS has a cache implemented by CachedStore. The cache is 
asynchronized and in HMS HA setting, we can only get eventual consistency. In 
this Jira, we try to make it synchronized.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21625) Fix TxnIdUtils.checkEquivalentWriteIds, also provides a comparison method

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21625:
-

 Summary: Fix TxnIdUtils.checkEquivalentWriteIds, also provides a 
comparison method
 Key: HIVE-21625
 URL: https://issues.apache.org/jira/browse/HIVE-21625
 Project: Hive
  Issue Type: Bug
 Environment: TxnIdUtils.checkEquivalentWriteIds has a bug which thinks 
({1,2,3,4}, 6) and ({1,2,3,4,5,6}, 8) compatible (the notation is (invalidlist, 
hwm)). Here is a patch to fix it, also provide a comparison method to check 
which is newer.
Reporter: Daniel Dai
Assignee: Daniel Dai
 Attachments: HIVE-21625.1.patch





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21583) KillTriggerActionHandler should use "hive" credential

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21583:
-

 Summary: KillTriggerActionHandler should use "hive" credential
 Key: HIVE-21583
 URL: https://issues.apache.org/jira/browse/HIVE-21583
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


Currently SessionState.username is set to null, which is invalid as 
KillQueryImplementation will valid the user privilege.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21479) NPE during metastore cache update

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21479:
-

 Summary: NPE during metastore cache update
 Key: HIVE-21479
 URL: https://issues.apache.org/jira/browse/HIVE-21479
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


Saw the following stack during a long periodical update:
{code}
2019-03-12T10:01:43,015 ERROR [CachedStore-CacheUpdateService: Thread-36] 
cache.CachedStore: Update failure:java.lang.NullPointerException
at 
org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.updateTableColStats(CachedStore.java:508)
at 
org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.update(CachedStore.java:461)
at 
org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.run(CachedStore.java:396)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}

The reason is we get the table list at very early stage and then refresh table 
one by one. It is likely table is removed during the interim. We need to deal 
with this case during cache update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21478) Metastore cache update shall capture exception

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21478:
-

 Summary: Metastore cache update shall capture exception
 Key: HIVE-21478
 URL: https://issues.apache.org/jira/browse/HIVE-21478
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai
 Attachments: HIVE-21478.1.patch

We definitely need to capture any exception during 
CacheUpdateMasterWork.update(), otherwise, Java would refuse to schedule future 
update().



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21389) Hive distribution miss javax.ws.rs-api.jar after HIVE-21247

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21389:
-

 Summary: Hive distribution miss javax.ws.rs-api.jar after 
HIVE-21247
 Key: HIVE-21389
 URL: https://issues.apache.org/jira/browse/HIVE-21389
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21379) Mask password in DDL commands for table properties

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21379:
-

 Summary: Mask password in DDL commands for table properties
 Key: HIVE-21379
 URL: https://issues.apache.org/jira/browse/HIVE-21379
 Project: Hive
  Issue Type: Improvement
Reporter: Daniel Dai
Assignee: Daniel Dai
 Attachments: HIVE-21379.1.patch

We need to mask password related table properties (such as 
hive.sql.dbcp.password) in DDL output, such as describe extended/describe 
formatted/show create table/show tblproperties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21296) Dropping varchar partition throw exception

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21296:
-

 Summary: Dropping varchar partition throw exception
 Key: HIVE-21296
 URL: https://issues.apache.org/jira/browse/HIVE-21296
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


Drop partition fail if the partition column is varchar. For example:
{code:java}
create external table BS_TAB_0_211494(c_date_SAD_29630 date) PARTITIONED BY 
(part_varchar_37229 varchar(56)) STORED AS orc;

INSERT INTO BS_TAB_0_211494 values('4740-04-04','BrNTRsv3c');

ALTER TABLE BS_TAB_0_211494 DROP PARTITION 
(part_varchar_37229='BrNTRsv3c');{code}

Exception:
{code}
2019-02-19T22:12:55,843  WARN [HiveServer2-Handler-Pool: Thread-42] 
thrift.ThriftCLIService: Error executing statement: 
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: 
FAILED: SemanticException [Error 10006]: Partition not found 
(part_varchar_37229 = 'BrNTRsv3c')
at 
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:356)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:269)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.Operation.run(Operation.java:268) 
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:576)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:561)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_202]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_202]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_202]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) 
~[?:1.8.0_202]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_202]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
 ~[hadoop-common-3.1.0.jar:?]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy43.executeStatementAsync(Unknown Source) ~[?:?]
at 
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:568)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
~[?:1.8.0_202]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
~[?:1.8.0_202]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Partition not 
found (part_varchar_37229 = 'BrNTRsv3c')
at 
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.addTableDropPartsOutputs(DDLSemanticAnalyzer.java:4110)
 ~[hive

[jira] [Created] (HIVE-21295) StorageHandler shall convert date to string using Hive convention

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21295:
-

 Summary: StorageHandler shall convert date to string using Hive 
convention
 Key: HIVE-21295
 URL: https://issues.apache.org/jira/browse/HIVE-21295
 Project: Hive
  Issue Type: Improvement
Reporter: Daniel Dai
Assignee: Daniel Dai
 Attachments: HIVE-21295.1.patch

If we have date datatype in mysql, string datatype defined in hive, 
JdbcStorageHandler will translate the date to string with the format -MM-dd 
HH:mm:ss. However, Hive convention is -MM-dd, we shall follow Hive 
convention. Eg:

mysql: CREATE TABLE test ("datekey" DATE);
hive: CREATE TABLE test (datekey string) STORED BY 
'org.apache.hive.storage.jdbc.JdbcStorageHandler' TBLPROPERTIES 
(.."hive.sql.table" = "test"..);

Then in hive, do: select datekey from test;

We get: 1999-03-24 00:00:00

But should be: 1999-03-24



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21255) Remove QueryConditionBuilder in JdbcStorageHandler

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21255:
-

 Summary: Remove QueryConditionBuilder in JdbcStorageHandler
 Key: HIVE-21255
 URL: https://issues.apache.org/jira/browse/HIVE-21255
 Project: Hive
  Issue Type: Improvement
Reporter: Daniel Dai
Assignee: Daniel Dai


QueryConditionBuilder is not correctly implemented. Always see the following 
exception even the query finish succeeded:
{code}
2019-02-13 01:09:53,406 [ERROR] [TezChild] |jdbc.QueryConditionBuilder|: Error 
during condition build
java.lang.ArrayIndexOutOfBoundsException: 0
at java.beans.XMLDecoder.readObject(XMLDecoder.java:250)
at 
org.apache.hive.storage.jdbc.QueryConditionBuilder.createConditionString(QueryConditionBuilder.java:125)
at 
org.apache.hive.storage.jdbc.QueryConditionBuilder.buildCondition(QueryConditionBuilder.java:74)
at 
org.apache.hive.storage.jdbc.conf.JdbcStorageConfigManager.getQueryToExecute(JdbcStorageConfigManager.java:155)
at 
org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:158)
at 
org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58)
at 
org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
at 
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
We don't actually need QueryConditionBuilder when cbo is enabled since 
predicate push down is handled by calcite (HIVE-20822). One can argue when cbo 
is disabled we might still need that since calcite will not do the push down, 
but that's a minor code path and removing QueryConditionBuilder won't cause any 
correctness issue. So I'd like to remove it for simplicity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21253) Support DB2 in JDBC StorageHandler

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21253:
-

 Summary: Support DB2 in JDBC StorageHandler
 Key: HIVE-21253
 URL: https://issues.apache.org/jira/browse/HIVE-21253
 Project: Hive
  Issue Type: Improvement
  Components: StorageHandler
Reporter: Daniel Dai
Assignee: Daniel Dai


Make DB2 a first class member of JdbcStorageHandler. It can even work before 
the patch by using POSTGRES as DB type and add db2 jdbc jar manually. This 
patch make it a standard feature. Note this is only for DB2 tables as external 
JdbcStorageHandler table. We haven't tested DB2 as a metastore backend and it's 
not a goal for this ticket.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21249) Reduce memory footprint in ObjectStore.refreshPrivileges

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21249:
-

 Summary: Reduce memory footprint in ObjectStore.refreshPrivileges  
 Key: HIVE-21249
 URL: https://issues.apache.org/jira/browse/HIVE-21249
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


We found there're could be many records in TBL_COL_PRIVS for a single table (a 
table granted to many users), thus result a OOM in 
ObjectStore.listTableAllColumnGrants. We shall reduce the memory footprint for 
ObjectStore.refreshPrivileges. Here is the stack of OOM:
{code}
org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:690)
org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:710)
org.apache.hadoop.hive.metastore.ObjectStore.listTableAllColumnGrants(ObjectStore.java:6629)
org.apache.hadoop.hive.metastore.ObjectStore.refreshPrivileges(ObjectStore.java:6200)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
com.sun.proxy.$Proxy32.refreshPrivileges(, line not available)
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.refresh_privileges(HiveMetaStore.java:6507)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
com.sun.proxy.$Proxy34.refresh_privileges(, line not available)
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17608)
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17592)
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
java.security.AccessController.doPrivileged(Native method)
javax.security.auth.Subject.doAs(Subject.java:422)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21248) WebHCat returns HTTP error code 500 rather than 429 when submitting large number of jobs in stress tests

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21248:
-

 Summary: WebHCat returns HTTP error code 500 rather than 429 when 
submitting large number of jobs in stress tests
 Key: HIVE-21248
 URL: https://issues.apache.org/jira/browse/HIVE-21248
 Project: Hive
  Issue Type: Bug
  Components: WebHCat
Reporter: Daniel Dai
Assignee: Daniel Dai


Saw the exception in webhcat.log:
{code}
java.lang.NoSuchMethodError: 
javax.ws.rs.core.Response$Status$Family.familyOf(I)Ljavax/ws/rs/core/Response$Status$Family;
at 
org.glassfish.jersey.message.internal.Statuses$StatusImpl.(Statuses.java:63)
 ~[jersey-common-2.25.1.jar:?]
at 
org.glassfish.jersey.message.internal.Statuses$StatusImpl.(Statuses.java:54)
 ~[jersey-common-2.25.1.jar:?]
at 
org.glassfish.jersey.message.internal.Statuses.from(Statuses.java:132) 
~[jersey-common-2.25.1.jar:?]
at 
org.glassfish.jersey.message.internal.OutboundJaxrsResponse$Builder.status(OutboundJaxrsResponse.java:414)
 ~[jersey-common-2.25.1.jar:?]
at javax.ws.rs.core.Response.status(Response.java:128) 
~[jsr311-api-1.1.1.jar:?]
at 
org.apache.hive.hcatalog.templeton.SimpleWebException.buildMessage(SimpleWebException.java:67)
 ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
at 
org.apache.hive.hcatalog.templeton.SimpleWebException.getResponse(SimpleWebException.java:51)
 ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
at 
org.apache.hive.hcatalog.templeton.SimpleExceptionMapper.toResponse(SimpleExceptionMapper.java:33)
 ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
at 
org.apache.hive.hcatalog.templeton.SimpleExceptionMapper.toResponse(SimpleExceptionMapper.java:29)
 ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
at 
com.sun.jersey.spi.container.ContainerResponse.mapException(ContainerResponse.java:480)
 ~[jersey-server-1.19.jar:1.19]
at 
com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException(ContainerResponse.java:417)
 ~[jersey-server-1.19.jar:1.19]
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1477)
 ~[jersey-server-1.19.jar:1.19]
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
 ~[jersey-server-1.19.jar:1.19]
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
 ~[jersey-server-1.19.jar:1.19]
at 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
 ~[jersey-servlet-1.19.jar:1.19]
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
 ~[jersey-servlet-1.19.jar:1.19]
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
 ~[jersey-servlet-1.19.jar:1.19]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) 
~[javax.servlet-api-3.1.0.jar:3.1.0]
at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) 
~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
 ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.apache.hive.hcatalog.templeton.Main$XFrameOptionsFilter.doFilter(Main.java:299)
 ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
 ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
 ~[hadoop-auth-3.1.1.3.0.2.0-50.jar:?]
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
 ~[hadoop-auth-3.1.1.3.0.2.0-50.jar:?]
at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:90) 
~[hadoop-hdfs-3.1.1.3.0.2.0-50.jar:?]
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
 ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) 
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
 [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) 
[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
 [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
[jetty-runner-9.3.20.v20170531.jar:9.3.20

[jira] [Created] (HIVE-21247) Webhcat beeline in secure mode

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21247:
-

 Summary: Webhcat beeline in secure mode
 Key: HIVE-21247
 URL: https://issues.apache.org/jira/browse/HIVE-21247
 Project: Hive
  Issue Type: Improvement
  Components: WebHCat
Reporter: Daniel Dai
Assignee: Daniel Dai


Follow up HIVE-20550, we need to make beeline work in secure mode. That means, 
we need to get a delegation token from hiveserver2, and pass that to beeline. 
This is similar to HIVE-5133, I make two changes:
1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, 
and pass along
2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, and 
extract delegation token if exists



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21013) JdbcStorageHandler fail to find partition column in Oracle

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-21013:
-

 Summary: JdbcStorageHandler fail to find partition column in Oracle
 Key: HIVE-21013
 URL: https://issues.apache.org/jira/browse/HIVE-21013
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


Stack:
{code}
ERROR : Vertex failed, vertexName=Map 1, 
vertexId=vertex_1543830849610_0048_1_00, diagnostics=[Task failed, 
taskId=task_1543830849610_0048_1_00_05, diagnostics=[TaskAttempt 0 failed, 
info=[Error: Error while running task ( failure ) : 
attempt_1543830849610_0048_1_00_05_0:java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: 
java.io.IOException: 
org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Caught 
exception while trying to execute query:Cannot find salaries in sql query 
salaries 
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.io.IOException: java.io.IOException: 
org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Caught 
exception while trying to execute query:Cannot find salaries in sql query 
salaries 
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:80)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
... 16 more
Caused by: java.io.IOException: java.io.IOException: 
org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Caught 
exception while trying to execute query:Cannot find salaries in sql query 
salaries 
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
at 
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
... 18 more
Caused by: java.io.IOException: 
org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Caught 
exception while trying to execute query:Cannot find salaries in sql query 
salaries 
at 
org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:85)
at 
org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
... 24 more
Caused

[jira] [Created] (HIVE-20978) "hive.jdbc.*" should add to sqlStdAuthSafeVarNameRegexes

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20978:
-

 Summary: "hive.jdbc.*" should add to sqlStdAuthSafeVarNameRegexes  
 Key: HIVE-20978
 URL: https://issues.apache.org/jira/browse/HIVE-20978
 Project: Hive
  Issue Type: Bug
  Components: Configuration
Reporter: Daniel Dai
Assignee: Daniel Dai
 Attachments: HIVE-20978.1.patch

User should be able to change hive.jdbc settings, include 
"hive.jdbc.pushdown.enable".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20944) Not validate stats during query compilation

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20944:
-

 Summary: Not validate stats during query compilation 
 Key: HIVE-20944
 URL: https://issues.apache.org/jira/browse/HIVE-20944
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


In a discussion with [~ashutoshc], we find currently query planning only use 
valid stats. If the stats are outdated, Hive will not get any stats. Hive shall 
use whatever we can find in metastore. It does not need to be up to date during 
query planning.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20937) Postgres jdbc query fail with "LIMIT must not be negative"

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20937:
-

 Summary: Postgres jdbc query fail with "LIMIT must not be negative"
 Key: HIVE-20937
 URL: https://issues.apache.org/jira/browse/HIVE-20937
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai
 Attachments: HIVE-20937.1.patch

PostgresDatabaseAccessor does not handle limit=-1. Likely to affect 
Oracle/Mssql as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20921) Oracle backed DbLockManager fail when drop/truncate acid table with large partitions

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20921:
-

 Summary: Oracle backed DbLockManager fail when drop/truncate acid 
table with large partitions
 Key: HIVE-20921
 URL: https://issues.apache.org/jira/browse/HIVE-20921
 Project: Hive
  Issue Type: Bug
  Components: Locking
Reporter: Daniel Dai
Assignee: Daniel Dai


Stack:
{code}
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: 
FAILED: Error in acquiring locks: Error communicating with the metastore 
at 
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:324)
 
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:199)
 
at 
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
 
at 
org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:422) 
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at 
org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745) 
Caused by: org.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating 
with the metastore 
at org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:177) 
at 
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:357)
 
at 
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocksWithHeartbeatDelay(DbTxnManager.java:373)
 
at 
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:182)
 
at org.apache.hadoop.hive.ql.Driver.acquireLocksAndOpenTxn(Driver.java:1082) 
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1284) 
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) 
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156) 
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
 
... 11 more 
Caused by: MetaException(message:How did we get here, we heartbeated our lock 
before we started! ( lockid:466073 intLockId:701 txnid:0 db:v5x2442 
table:tbstcnf_load_stg_step 
partition:src_system_cd=MAXIMO/src_hostname_cd=PRD1310/src_table_name=LABTRANS 
state:WAITING type:EXCLUSIVE)) 
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2642) 
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:1187) 
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.check_lock(HiveMetaStore.java:6161)
 
at sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 
at java.lang.reflect.Method.invoke(Method.java:497) 
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
 
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
 
at com.sun.proxy.$Proxy14.check_lock(Unknown Source) 
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.checkLock(HiveMetaStoreClient.java:1984)
 
at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 
at java.lang.reflect.Method.invoke(Method.java:497) 
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:178)
 
at com.sun.proxy.$Proxy15.checkLock(Unknown Source) 
at org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:114) 
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20896) CachedStore fail to cache stats in multiple code paths

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20896:
-

 Summary: CachedStore fail to cache stats in multiple code paths
 Key: HIVE-20896
 URL: https://issues.apache.org/jira/browse/HIVE-20896
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


Bunch of issues discovered in CachedStore to keep up column statistics:
1. The criteria for partition/non-partition is wrong 
(table.isSetPartitionKeys() is always true)
2. In update(), partition column stats are removed when populate table basic 
stats
3. Dirty flags are true right after prewarm(), so the first update() does not 
do anything
4. Could invoke cacheLock without holding the lock, which results a freeze in 
update()



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[SECURITY] CVE-2018-11777: Blocking local resource access in HiveServer2

[Daniel Dai]

CVE-2018-11777: Blocking local resource access in HiveServer2

Severity: Important

Vendor: The Apache Software Foundation

Versions Affected: This vulnerability affects all versions of Hive,
including 2.3.3, 3.1.0 and earlier

Description: Local resources on HiveServer2 machines are not properly
protected against malicious user if ranger, sentry or sql standard
authorizer is not in use.

Mitigation: It is recommended to upgrade to 2.3.4 or 3.1.1 or later if
HiveServer2 is used, and ranger, sentry or sql standard authorizer
is not in use. Admin needs to specify the following entries in
hiveserver2-site.xml:


  hive.security.authorization.enabled
  true


  hive.security.authorization.manager
  
org.apache.hadoop.hive.ql.security.authorization.plugin.fallback.FallbackHiveAuthorizerFactory


FallbackHiveAuthorizerFactory will do the following to mitigate above
mentioned threat:
1. Disallow local file location in sql statements except for admin
2. Allow "set" only selected whitelist parameters
3. Disallow dfs commands except for admin
4. Disallow "ADD JAR" statement
5. Disallow "COMPILE" statement
6. Disallow "TRANSFORM" statement

Credit: This issue was discovered by Mithun Radhakrishnan of Oath Inc

[SECURITY] CVE-2018-1314: Hive explain query not being authorized

[Daniel Dai]

CVE-2018-1314: Hive explain query not being authorized

Severity: Important

Vendor: The Apache Software Foundation

Versions Affected: This vulnerability affects all versions of Hive,
including 2.3.3, 3.1.0 and earlier

Description: Hive "EXPLAIN" operation does not check for necessary
authorization of involved entities in a query. An unauthorized user
can do "EXPLAIN" on arbitrary table or view and expose table metadata
and statistics.

Mitigation: all Hive users shall upgrade to 2.3.4 or 3.1.1 or later

[ANNOUNCE] Apache Hive 2.3.4 Released

[Daniel Dai]

The Apache Hive team is proud to announce the release of Apache Hive
version 2.3.4.

The Apache Hive (TM) data warehouse software facilitates querying and
managing large datasets residing in distributed storage. Built on top
of Apache Hadoop (TM), it provides, among others:

* Tools to enable easy data extract/transform/load (ETL)

* A mechanism to impose structure on a variety of data formats

* Access to files stored either directly in Apache HDFS (TM) or in other
  data storage systems such as Apache HBase (TM)

* Query execution via Apache Hadoop MapReduce, Apache Tez and Apache Spark
frameworks.

For Hive release details and downloads, please visit:
https://hive.apache.org/downloads.html

Hive 2.3.4 Release Notes are available here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344319=Text=12310843

We would like to thank the many contributors who made this release
possible.

Regards,

The Apache Hive Team

[RESULT][VOTE] Apache Hive 2.3.4 Release Candidate 0

[Daniel Dai]

With three +1’s (Alan, Thejas, Daniel) and no -1’s the vote passes.

I will wrap up the release and send announcement once done.
 

[ANNOUNCE] Apache Hive 3.1.1 Released

[Daniel Dai]

The Apache Hive team is proud to announce the release of Apache Hive
version 3.1.1.

The Apache Hive (TM) data warehouse software facilitates querying and
managing large datasets residing in distributed storage. Built on top
of Apache Hadoop (TM), it provides, among others:

* Tools to enable easy data extract/transform/load (ETL)

* A mechanism to impose structure on a variety of data formats

* Access to files stored either directly in Apache HDFS (TM) or in other
  data storage systems such as Apache HBase (TM)

* Query execution via Apache Hadoop MapReduce, Apache Tez and Apache Spark
frameworks.

For Hive release details and downloads, please visit:
https://hive.apache.org/downloads.html

Hive 3.1.1 Release Notes are available here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344240=Text=12310843

We would like to thank the many contributors who made this release
possible.

Regards,

The Apache Hive Team

Re: Review Request 69143: CachedStore: Add more UT coverage (outside of .q files)

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69143/#review210272
---


Ship it!




Ship It!

- Daniel Dai


On Nov. 2, 2018, 12:46 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69143/
> ---
> 
> (Updated Nov. 2, 2018, 12:46 a.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-20613
> https://issues.apache.org/jira/browse/HIVE-20613
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-20613
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  944c81313a 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  70490f09e7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  c24e7160ac 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  bb20d9f42a 
>   standalone-metastore/metastore-server/src/test/resources/log4j2.properties 
> 365687e1c9 
> 
> 
> Diff: https://reviews.apache.org/r/69143/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[VOTE] Apache Hive 2.3.4 Release Candidate 0

[Daniel Dai]

Apache Hive 2.3.4 Release Candidate 0 is available here:

http://people.apache.org/~daijy/apache-hive-2.3.4-rc-0/

Maven artifacts are available here:

https://repository.apache.org/content/repositories/orgapachehive-1093

Source tag for RCN is at:

https://github.com/apache/hive/tree/release-2.3.4-rc0

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

[RESULT][VOTE] Apache Hive 3.1.1 Release Candidate 0

[Daniel Dai]

[RESULT][VOTE] Apache Hive 3.1.1 Release Candidate 0

With three +1’s (Thejas, Zoltan, Daniel) and no -1’s the vote passes.

I will wrap up the release and send announcement once done.

[jira] [Created] (HIVE-20830) JdbcStorageHandler range query assertion failure in some cases

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20830:
-

 Summary: JdbcStorageHandler range query assertion failure in some 
cases
 Key: HIVE-20830
 URL: https://issues.apache.org/jira/browse/HIVE-20830
 Project: Hive
  Issue Type: Bug
  Components: StorageHandler
Reporter: Daniel Dai
Assignee: Daniel Dai


{code}
2018-10-29T10:10:16,325 ERROR [b4bf5eb2-a986-4aae-908e-93b9908acd32 
HiveServer2-HttpHandler-Pool: Thread-124]: dao.GenericJdbcDatabaseAccessor 
(:()) - Caught exception while trying to execute query
java.lang.IllegalArgumentException: null
at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:108) 
~[guava-19.0.jar:?]
at 
org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.addBoundaryToQuery(GenericJdbcDatabaseAccessor.java:238)
 ~[hive-jdbc-handler-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-99]
at 
org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:161)
 ~[hive-jdbc-handler-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-99]
at 
org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58) 
~[hive-jdbc-handler-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-99]
at 
org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35) 
~[hive-jdbc-handler-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-99]
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:569) 
~[hive-exec-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) 
~[hive-exec-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) 
~[hive-exec-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2734) 
~[hive-exec-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) 
~[hive-exec-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:469)
 ~[hive-service-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328)
 ~[hive-service-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:910)
 ~[hive-service-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:564) 
~[hive-service-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:790)
 ~[hive-service-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
 ~[hive-exec-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
 ~[hive-exec-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
~[hive-exec-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[hive-exec-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) 
~[hive-exec-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:208)
 ~[hive-service-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) 
~[javax.servlet-api-3.1.0.jar:3.1.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) 
~[javax.servlet-api-3.1.0.jar:3.1.0]
at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) 
~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) 
~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
 ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
 ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) 
~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
 ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java

[jira] [Created] (HIVE-20829) JdbcStorageHandler range split throws NPE

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20829:
-

 Summary: JdbcStorageHandler range split throws NPE
 Key: HIVE-20829
 URL: https://issues.apache.org/jira/browse/HIVE-20829
 Project: Hive
  Issue Type: Bug
  Components: StorageHandler
Reporter: Daniel Dai
Assignee: Daniel Dai


{code}
2018-10-29T06:37:14,982 ERROR [HiveServer2-Background-Pool: Thread-44466]: 
operation.Operation (:()) - Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: 
FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, 
vertexId=vertex_1540588928441_0121_2_00, diagnostics=[Vertex 
vertex_1540588928441_0121_2_00 [Map 1] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: employees initializer failed, 
vertex=vertex_1540588928441_0121_2_00 [Map 1], java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:272)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1540588928441_0121_2_01, 
diagnostics=[Vertex received Kill in INITED state., Vertex 
vertex_1540588928441_0121_2_01 [Reducer 2] killed/failed due 
to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. 
failedVertices:1 killedVertices:1
at 
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
 ~[hive-service-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:228)
 ~[hive-service-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
 ~[hive-service-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:318)
 ~[hive-service-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at java.security.AccessController.doPrivileged(Native Method) 
~[?:1.8.0_161]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_161]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
 ~[hadoop-common-3.1.1.3.0.3.0-150.jar:?]
at 
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:338)
 ~[hive-service-3.1.0.3.0.3.0-150.jar:3.1.0.3.0.3.0-150]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[?:1.8.0_161]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[?:1.8.0_161]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[?:1.8.0_161]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[?:1.8.0_161]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
~[?:1.8.0_161]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
~[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Vertex failed, 
vertexName=Map 1, vertexId=vertex_1540588928441_0121_2_00, diagnostics=[Vertex 
vertex_1540588928441_0121_2_00 [Map 1] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: employees initializer failed, 
vertex=vertex_1540588928441_0121_2_00 [Map 1], java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:272

[jira] [Created] (HIVE-20815) JdbcRecordReader.next shall not eat exception

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20815:
-

 Summary: JdbcRecordReader.next shall not eat exception
 Key: HIVE-20815
 URL: https://issues.apache.org/jira/browse/HIVE-20815
 Project: Hive
  Issue Type: Bug
  Components: StorageHandler
Reporter: Daniel Dai
Assignee: Daniel Dai






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69143: CachedStore: Add more UT coverage (outside of .q files)

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69143/#review210011
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 94 (patched)
<https://reviews.apache.org/r/69143/#comment294650>

Can you add a comment what's the difference between isCachePrewarmed?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 227 (patched)
<https://reviews.apache.org/r/69143/#comment294647>

Log full stack



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 491 (patched)
<https://reviews.apache.org/r/69143/#comment294646>

Can you log the full stack trace (maybe using 
org.apache.hadoop.util.StringUtils.stringifyException)? There's another 
instance below.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 522 (patched)
<https://reviews.apache.org/r/69143/#comment294648>

Make maxTries a constant (and reuse it below).



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 544 (patched)
<https://reviews.apache.org/r/69143/#comment294649>

maxTries should be constant.


- Daniel Dai


On Oct. 24, 2018, 5:04 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69143/
> ---
> 
> (Updated Oct. 24, 2018, 5:04 p.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-20613
> https://issues.apache.org/jira/browse/HIVE-20613
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-20613
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  944c81313a 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  70490f09e7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  c24e7160ac 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  bb20d9f42a 
>   standalone-metastore/metastore-server/src/test/resources/log4j2.properties 
> 365687e1c9 
> 
> 
> Diff: https://reviews.apache.org/r/69143/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[VOTE] Apache Hive 3.1.1 Release Candidate 0

[Daniel Dai]

Apache Hive 3.1.1 Release Candidate 0 is available here:

http://people.apache.org/~daijy/apache-hive-3.1.1-rc-0/

Maven artifacts are available here:

https://repository.apache.org/content/repositories/orgapachehive-1092

Source tag for RCN is at:

https://github.com/apache/hive/tree/release-3.1.1-rc0

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

Re: Incorrect Release Notes for Hive-2.3.3

[Daniel Dai]

Yes, I cleared fixed version from the Jiras. It should be fixed.

Thanks,
Daniel

From: Oleksiy S 
Reply-To: "u...@hive.apache.org" 
Date: Friday, October 5, 2018 at 5:02 AM
To: "dev@hive.apache.org" , "u...@hive.apache.org" 

Subject: Re: Incorrect Release Notes for Hive-2.3.3

Guys any updates?

On Thu, Oct 4, 2018 at 11:15 AM Oleksiy S 
mailto:osayankin.superu...@gmail.com>> wrote:
Current release notes for Hive-2.3.3 are: 
RNs

Release Notes - Hive - Version 2.3.3

** Bug
* [HIVE-16939] - metastore error: 'export: -Dproc_metastore : not a valid 
identifier'
* [HIVE-18767] - Some alterPartitions invocations throw 
'NumberFormatException: null'
* [HIVE-18788] - Clean up inputs in JDBC PreparedStatement
* [HIVE-18815] - Remove unused feature in HPL/SQL
* [HIVE-18879] - Disallow embedded element in UDFXPathUtil needs to work if 
xercesImpl.jar in classpath
* [HIVE-18885] - DbNotificationListener has a deadlock between Java and DB 
locks (2.x line)
* [HIVE-20304] - When hive.optimize.skewjoin and hive.auto.convert.join are 
both set to true, and the execution engine is mr, same stage may launch twice 
due to the wrong generated plan
* [HIVE-20441] - NPE in ExprNodeGenericFuncDesc  when 
hive.allow.udf.load.on.demand is set to true

** Improvement
* [HIVE-19900] - HiveCLI HoS Performs Invalid Impersonation If User Name 
Truncated
* [HIVE-20284] - In strict mode, if constant propagation is enable, the 
partition filter may be folded before partition pruner lead to error "No 
partition predicate for Alias"


See table for short summary.
Issue

Comment

HIVE-18767

Issue is not in Hive-2.3.3

HIVE-20304

Issue is unresolved in Apache Hive JIRA

HIVE-20441

Issue is unresolved in Apache Hive JIRA

HIVE-19900

Resolved: Workaround. No commit to see

HIVE-20284

Issue is unresolved in Apache Hive JIRA


Please remove incorrect JIRAs from RNs.

--
Oleksiy


--
Oleksiy

Re: Review Request 69050: HIVE-20720: Add partition column option to JDBC handler

[Daniel Dai]

> On Oct. 17, 2018, 8:29 p.m., Jesús Camacho Rodríguez wrote:
> > jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
> > Lines 233 (patched)
> > <https://reviews.apache.org/r/69050/diff/2/?file=2098532#file2098532line298>
> >
> > Could we use a case insensitive regex that would match something like 
> > 'FROM\s+tableString'? I believe it would be safer. If you do not tackle it 
> > in this issue, let's create a follow-up.

Yes, it is better as it won't match anything before the first FROM. However, it 
could be quite complex to get it absolutely right. We also need to make sure it 
will search after "JOIN" and won't search beyond "WHERE/GROUP" etc. And can 
there be multiple "FROM"? I adopt the "from" part in the patch, but still leave 
the TODO tag and will think about a better way in the future.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69050/#review209714
-------


On Oct. 16, 2018, 5:08 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69050/
> ---
> 
> (Updated Oct. 16, 2018, 5:08 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-20720.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/Constants.java 1190679 
>   itests/src/test/resources/testconfiguration.properties 9a87464 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputFormat.java 
> 74999db 
>   jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputSplit.java 
> 3a6ada8 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcRecordReader.java 
> 1da6213 
>   jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java 
> 5947628 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
>  18e2397 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessor.java
>  fdaa794 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
>  abdc5f0 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
>  a95aca2 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JethroDatabaseAccessor.java
>  db0454e 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MySqlDatabaseAccessor.java
>  86fde7c 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DateIntervalSplitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DecimalIntervalSplitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DoubleIntervalSplitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitterFactory.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/LongIntervalSpitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/TimestampIntervalSplitter.java
>  PRE-CREATION 
>   
> jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/TestJdbcInputFormat.java
>  b146633 
>   
> jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/dao/TestGenericJdbcDatabaseAccessor.java
>  34f061e 
>   ql/src/test/queries/clientpositive/external_jdbc_table_partition.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/external_jdbc_table_typeconversion.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/external_jdbc_table_partition.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/external_jdbc_table_typeconversion.q.out
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69050/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: Review Request 69050: HIVE-20720: Add partition column option to JDBC handler

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69050/
---

(Updated Oct. 16, 2018, 5:08 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-20720.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/Constants.java 1190679 
  itests/src/test/resources/testconfiguration.properties 9a87464 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputFormat.java 
74999db 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputSplit.java 
3a6ada8 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcRecordReader.java 
1da6213 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java 
5947628 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
 18e2397 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessor.java
 fdaa794 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 abdc5f0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 a95aca2 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JethroDatabaseAccessor.java
 db0454e 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MySqlDatabaseAccessor.java
 86fde7c 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DateIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DecimalIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DoubleIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitterFactory.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/LongIntervalSpitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/TimestampIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/TestJdbcInputFormat.java
 b146633 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/dao/TestGenericJdbcDatabaseAccessor.java
 34f061e 
  ql/src/test/queries/clientpositive/external_jdbc_table_partition.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/external_jdbc_table_typeconversion.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_partition.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/external_jdbc_table_typeconversion.q.out
 PRE-CREATION 


Diff: https://reviews.apache.org/r/69050/diff/2/

Changes: https://reviews.apache.org/r/69050/diff/1-2/


Testing
---


Thanks,

Daniel Dai

Review Request 69050: HIVE-20720: Add partition column option to JDBC handler

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69050/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-20720.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/Constants.java 1190679 
  itests/src/test/resources/testconfiguration.properties 9a87464 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputFormat.java 
74999db 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputSplit.java 
3a6ada8 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcRecordReader.java 
1da6213 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcSerDe.java 
5947628 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
 18e2397 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessor.java
 fdaa794 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 abdc5f0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 a95aca2 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JethroDatabaseAccessor.java
 db0454e 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MySqlDatabaseAccessor.java
 86fde7c 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DateIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DecimalIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/DoubleIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/IntervalSplitterFactory.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/LongIntervalSpitter.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/spitter/TimestampIntervalSplitter.java
 PRE-CREATION 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/TestJdbcInputFormat.java
 b146633 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/dao/TestGenericJdbcDatabaseAccessor.java
 34f061e 
  ql/src/test/queries/clientpositive/external_jdbc_table_partition.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/external_jdbc_table_typeconversion.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_partition.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/external_jdbc_table_typeconversion.q.out
 PRE-CREATION 


Diff: https://reviews.apache.org/r/69050/diff/1/


Testing
---


Thanks,

Daniel Dai

[jira] [Created] (HIVE-20732) conf.HiveConf: HiveConf of name hive.metastore.cached.rawstore.cached.object.whitelist does not exist

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20732:
-

 Summary: conf.HiveConf: HiveConf of name 
hive.metastore.cached.rawstore.cached.object.whitelist does not exist
 Key: HIVE-20732
 URL: https://issues.apache.org/jira/browse/HIVE-20732
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Vaibhav Gumashta


[~ndembla] saw this message in hs2 log. MetastoreConf properties should also 
add to HiveConf.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20731) keystore file in JdbcStorageHandler should be authorized

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20731:
-

 Summary: keystore file in JdbcStorageHandler should be authorized
 Key: HIVE-20731
 URL: https://issues.apache.org/jira/browse/HIVE-20731
 Project: Hive
  Issue Type: Improvement
  Components: StorageHandler
Reporter: Daniel Dai
Assignee: Daniel Dai


The keystore file introduced in HIVE-20651 shall be authorized with configured 
authorizer. Otherwise any user knows the keystore file location can access the 
password.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20720) Add partition column option to JDBC handler

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20720:
-

 Summary: Add partition column option to JDBC handler
 Key: HIVE-20720
 URL: https://issues.apache.org/jira/browse/HIVE-20720
 Project: Hive
  Issue Type: New Feature
  Components: StorageHandler
Reporter: Daniel Dai
Assignee: Daniel Dai


Currently JdbcStorageHandler does not split input in Tez. The reason is 
numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" in 
Tez. And "mapreduce.job.maps" is not a valid param if ranger is in use. User 
ends up always use 1 split.

We need to rely on this new feature if we want to support multi-splits. Here is 
my proposal:
1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound in 
tblproperties if user want to split jdbc data source. In case 
lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min 
query to get this in planner. We can currently limit partitionColumn to only 
numeric/date/timestamp column for simplicity
2. If partitionColumn/numPartitions are not specified, don't split input
3. Splits are equal intervals without respect to data distribution
4. There is also a "hive.sql.query.split" flag vetos the split (can be set 
manually or automatically by calcite)
5. If partitionColumn is not defined, but numPartitions is defined, use 
original limit/offset logic (however, don't rely on numSplit).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68917: HIVE-20651: JdbcStorageHandler password should be encrypted

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68917/
---

(Updated Oct. 5, 2018, 5:24 a.m.)


Review request for hive and Sankar Hariappan.


Repository: hive-git


Description
---

See HIVE-20651


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties fdd8ecc 
  jdbc-handler/pom.xml f90892f 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputFormat.java 
caa823f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
 55fc0ea 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 b2ad9a6 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/TestJdbcInputFormat.java
 e904774 
  pom.xml 5008923 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 74fb1ba 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 99b33a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
7cd853f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
ac43917 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java a86656c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 82c3ca9 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 250a085 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java b73faa5 


Diff: https://reviews.apache.org/r/68917/diff/2/

Changes: https://reviews.apache.org/r/68917/diff/1-2/


Testing
---


Thanks,

Daniel Dai

Review Request 68917: HIVE-20651: JdbcStorageHandler password should be encrypted

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68917/
---

Review request for hive and Sankar Hariappan.


Repository: hive-git


Description
---

See HIVE-20651


Diffs
-

  itests/src/test/resources/testconfiguration.properties fdd8ecc 
  jdbc-handler/pom.xml f90892f 
  jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcInputFormat.java 
caa823f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
 55fc0ea 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 b2ad9a6 
  
jdbc-handler/src/test/java/org/apache/hive/storage/jdbc/TestJdbcInputFormat.java
 e904774 
  pom.xml 5008923 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 74fb1ba 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 99b33a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
7cd853f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
ac43917 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java a86656c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 82c3ca9 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 250a085 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java b73faa5 
  ql/src/test/queries/clientpositive/external_jdbc_auth.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/external_jdbc_auth.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/68917/diff/1/


Testing
---


Thanks,

Daniel Dai

[jira] [Created] (HIVE-20675) Log pollution from PrivilegeSynchronizer if zk is not configured

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20675:
-

 Summary: Log pollution from PrivilegeSynchronizer if zk is not 
configured
 Key: HIVE-20675
 URL: https://issues.apache.org/jira/browse/HIVE-20675
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Daniel Dai
Assignee: Daniel Dai


Shall stop PrivilegeSynchronizer if "hive.zookeeper.quorum" is not configured. 
Note "hive.privilege.synchronizer" is on by default.

{code}
2018-10-02T16:04:12,488  WARN [main-SendThread(localhost:2181)] 
zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing 
socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 
~[?:1.8.0_91]
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 
~[?:1.8.0_91]
at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
 ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 
~[zookeeper-3.4.6.jar:3.4.6-1569965]
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20674) TestJdbcWithMiniLlapArrow.testKillQuery fail frequently

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20674:
-

 Summary: TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
 Key: HIVE-20674
 URL: https://issues.apache.org/jira/browse/HIVE-20674
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20658) "show tables" should show view as well

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20658:
-

 Summary: "show tables" should show view as well
 Key: HIVE-20658
 URL: https://issues.apache.org/jira/browse/HIVE-20658
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


"show tables" changed behavior to show real table (no view) in HIVE-19408. This 
breaks backward compatibility and we shall restore default behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20653) Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20653:
-

 Summary: Schema change in HIVE-19166 should also go to 
hive-schema-4.0.0.hive.sql
 Key: HIVE-20653
 URL: https://issues.apache.org/jira/browse/HIVE-20653
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20652:
-

 Summary: JdbcStorageHandler push join of two different datasource 
to jdbc driver
 Key: HIVE-20652
 URL: https://issues.apache.org/jira/browse/HIVE-20652
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Reporter: Daniel Dai
 Attachments: external_jdbc_table2.q

Test case attached. The following query fail:
{code}
SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
{code}
Error message:
{code}
2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main] 
dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
FROM (SELECT *
FROM "SIMPLE_DERBY_TABLE1"
WHERE "ikey" IS NOT NULL) AS "t"
INNER JOIN (SELECT *
FROM "SIMPLE_DERBY_TABLE2"
WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main] 
dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not 
exist.
at 
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) 
~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
Source) ~[derby-10.14.1.0.jar:?]
at 
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
Source) ~[derby-10.14.1.0.jar:?]
at 
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
Source) ~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown 
Source) ~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown 
Source) ~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown 
Source) ~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown 
Source) ~[derby-10.14.1.0.jar:?]
at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown 
Source) ~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown 
Source) ~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown 
Source) ~[derby-10.14.1.0.jar:?]
at 
org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
 ~[commons-dbcp-1.4.jar:1.4]
at 
org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
 ~[commons-dbcp-1.4.jar:1.4]
at 
org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
 [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78) 
[hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54) 
[hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540) 
[hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
 [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
 [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
 [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277) 
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
 [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
 [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
 [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
 [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12319)
 [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:356)
 [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
 [hive-exec-4.0.0-SNAPSHOT.jar:4.

[jira] [Created] (HIVE-20651) JdbcStorageHandler password should be encrypted

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20651:
-

 Summary: JdbcStorageHandler password should be encrypted
 Key: HIVE-20651
 URL: https://issues.apache.org/jira/browse/HIVE-20651
 Project: Hive
  Issue Type: Improvement
  Components: StorageHandler
Reporter: Daniel Dai
Assignee: Daniel Dai


Currently, external jdbc table with JdbcStorageHandler store password as 
"hive.sql.dbcp.password" table property in clear text. We should put it in a 
keystore file. Here is the proposed change:
{code:java}
….
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.dbcp.password.keystore" = 
"hdfs:///user/hive/credential/postgres.jceks",
"hive.sql.dbcp.password.key" = "mydb.password"
);
{code}
 
The jceks file is created with:
{code}
hadoop credential create mydb.password -provider 
hdfs:///user/hive/credential/postgres.jceks -v secretpassword
{code}

User can choose to put all db password in one jceks, or a separate jceks for 
each db.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68733: HIVE-20549: Allow user set query tag, and kill query with tag

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68733/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-20549


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa58d74 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniLlapArrow.java
 4942ed9 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java dad2035 
  ql/src/java/org/apache/hadoop/hive/ql/QueryState.java 028dd60 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
e4186c4 
  ql/src/test/queries/clientnegative/authorization_kill_query.q 5379f87 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
8db6a29 
  service/src/java/org/apache/hive/service/server/KillQueryImpl.java 490a04d 


Diff: https://reviews.apache.org/r/68733/diff/1/


Testing
---


Thanks,

Daniel Dai

[jira] [Created] (HIVE-20550) Switch WebHCat to use beeline to submit Hive queries

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20550:
-

 Summary: Switch WebHCat to use beeline to submit Hive queries
 Key: HIVE-20550
 URL: https://issues.apache.org/jira/browse/HIVE-20550
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


Since hive cli is deprecated, we shall switch WebHCat to use beeline instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20549) Allow user set query tag, and kill query with tag

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20549:
-

 Summary: Allow user set query tag, and kill query with tag
 Key: HIVE-20549
 URL: https://issues.apache.org/jira/browse/HIVE-20549
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


HIVE-19924 add capacity for replication job set a query tag and kill the 
replication distcp job with the tag. Here I make it more general, user can set 
arbitrary "hive.query.tag" in sql script, and kill query with the tag. Hive 
will cancel the corresponding operation in hs2, along with Tez/MR application 
launched for the query. For example:
{code}
set hive.query.tag=mytag;
select . -- long running query
{code}

In another session:
{code}
kill query 'mytag';
{code}

There're limitations in the implementation:
1. No tag duplication check. There's nothing to prevent conflicting tag for 
same user, and kill query will kill queries share the same tag. However, kill 
query will not kill queries from different user unless admin. So different user 
might share the same tag
2. In multiple hs2 environment, kill statement should be issued to all hs2 to 
make sure the corresponding operation is canceled. When beeline/jdbc connects 
to hs2 using regular way (zookeeper url), the session will connect to random 
hs2, which might be different than the hs2 where query run on. User can use 
HiveConnection.getAllUrls or beeline --getUrlsFromBeelineSite (HIVE-20507) to 
get a list of all hs2 instances.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68630: HIVE-20420: Provide a fallback authorizer when no other authorizer is in use

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68630/
---

(Updated Sept. 11, 2018, 11:19 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-20420


Diffs (updated)
-

  ql/pom.xml a55cbe3 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/SettableConfigUpdater.java
 12be41c 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizerFactory.java
 PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_addjar.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_compile.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_create_func1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_create_func2.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_dfs.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_disallow_transform.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_load.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_set_invalidconf.q 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_addjar.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_compile.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_create_func1.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_create_func2.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_dfs.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_disallow_transform.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_load.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_set_invalidconf.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/68630/diff/2/

Changes: https://reviews.apache.org/r/68630/diff/1-2/


Testing
---


Thanks,

Daniel Dai

Review Request 68630: HIVE-20420: Provide a fallback authorizer when no other authorizer is in use

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68630/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-20420


Diffs
-

  ql/pom.xml a55cbe3 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/SettableConfigUpdater.java
 12be41c 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizerFactory.java
 PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_addjar.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_compile.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_create_func1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_create_func2.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_dfs.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_disallow_transform.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_load.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_set_invalidconf.q 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_addjar.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_compile.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_create_func1.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_create_func2.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_dfs.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_disallow_transform.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_load.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_set_invalidconf.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/68630/diff/1/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 68590: HIVE-20493: Unit test getGroupNames for SessionStateUserAuthenticator

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68590/#review208201
---


Ship it!




- Daniel Dai


On Aug. 31, 2018, 8:24 p.m., Karthik Manamcheri wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68590/
> ---
> 
> (Updated Aug. 31, 2018, 8:24 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Daniel Dai.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20493: Unit test getGroupNames for SessionStateUserAuthenticator
> 
> 
> Diffs
> -
> 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestSessionUserName.java
>  4b645f390f 
> 
> 
> Diff: https://reviews.apache.org/r/68590/diff/1/
> 
> 
> Testing
> ---
> 
> Ran 'mvn -Dtest=TestSessionUserName test' successfully.
> 
> 
> Thanks,
> 
> Karthik Manamcheri
> 
>

Re: Review Request 68590: HIVE-20493: Unit test getGroupNames for SessionStateUserAuthenticator

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68590/#review208200
---


Ship it!




Ship It!

- Daniel Dai


On Aug. 31, 2018, 8:24 p.m., Karthik Manamcheri wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68590/
> ---
> 
> (Updated Aug. 31, 2018, 8:24 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Daniel Dai.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20493: Unit test getGroupNames for SessionStateUserAuthenticator
> 
> 
> Diffs
> -
> 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestSessionUserName.java
>  4b645f390f 
> 
> 
> Diff: https://reviews.apache.org/r/68590/diff/1/
> 
> 
> Testing
> ---
> 
> Ran 'mvn -Dtest=TestSessionUserName test' successfully.
> 
> 
> Thanks,
> 
> Karthik Manamcheri
> 
>

[jira] [Created] (HIVE-20494) GenericUDFRestrictInformationSchema is broken after HIVE-19440

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20494:
-

 Summary: GenericUDFRestrictInformationSchema is broken after 
HIVE-19440
 Key: HIVE-20494
 URL: https://issues.apache.org/jira/browse/HIVE-20494
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20444) Parameter is not properly quoted in DbNotificationListener.addWriteNotificationLog

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20444:
-

 Summary: Parameter is not properly quoted in 
DbNotificationListener.addWriteNotificationLog
 Key: HIVE-20444
 URL: https://issues.apache.org/jira/browse/HIVE-20444
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


See exception:
{code}
2018-08-22T04:44:22,758 INFO  [pool-8-thread-190]: 
listener.DbNotificationListener 
(DbNotificationListener.java:addWriteNotificationLog(765)) - Going to execute 
insert 
2018-08-22T04:44:22,773 ERROR [pool-8-thread-190]: metastore.RetryingHMSHandler 
(RetryingHMSHandler.java:invokeInternal(201)) - MetaException(message:Unable to 
add write notification log org.postgresql.util.PSQLException: ERROR: syntax 
error at or near "UTC"
  Position: 1032
at 
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2284)
at 
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2003)
at 
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
at 
org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:321)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:313)
at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:92)
at 
com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
at 
org.apache.hive.hcatalog.listener.DbNotificationListener.addWriteNotificationLog(DbNotificationListener.java:766)
at 
org.apache.hive.hcatalog.listener.DbNotificationListener.onAcidWrite(DbNotificationListener.java:657)
at 
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.lambda$static$12(MetaStoreListenerNotifier.java:249)
at 
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEventWithDirectSql(MetaStoreListenerNotifier.java:305)
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.addWriteNotificationLog(TxnHandler.java:1617)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addTxnWriteNotificationLog(HiveMetaStore.java:7563)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.add_write_notification_log(HiveMetaStore.java:7589)
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy34.add_write_notification_log(Unknown Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_write_notification_log.getResult(ThriftHiveMetastore.java:19071)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_write_notification_log.getResult(ThriftHiveMetastore.java:19056)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at 
org.apache.hive.hcatalog.listener.DbNotificationListener.onAcidWrite(DbNotificationListener.java:659)
at 
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.lambda$static$12(MetaStoreListenerNotifier.java:249)
at 
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEventWithDirectSql(MetaStoreListenerNotifier.java:305)
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.addWriteNotificationLog(TxnHandler.java:1617)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addTxnWriteNotificationLog(HiveMetaStore

[jira] [Created] (HIVE-20424) schematool shall not pollute beeline history

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20424:
-

 Summary: schematool shall not pollute beeline history
 Key: HIVE-20424
 URL: https://issues.apache.org/jira/browse/HIVE-20424
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20420) Provide a fallback authorizer when no other authorizer is in use

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20420:
-

 Summary: Provide a fallback authorizer when no other authorizer is 
in use
 Key: HIVE-20420
 URL: https://issues.apache.org/jira/browse/HIVE-20420
 Project: Hive
  Issue Type: New Feature
Reporter: Daniel Dai
Assignee: Daniel Dai






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20413) "cannot insert NULL" for TXN_WRITE_NOTIFICATION_LOG in Oracle

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20413:
-

 Summary: "cannot insert NULL" for TXN_WRITE_NOTIFICATION_LOG in 
Oracle
 Key: HIVE-20413
 URL: https://issues.apache.org/jira/browse/HIVE-20413
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20389) NPE in SessionStateUserAuthenticator when authenticator=SessionStateUserAuthenticator

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20389:
-

 Summary: NPE in SessionStateUserAuthenticator when 
authenticator=SessionStateUserAuthenticator
 Key: HIVE-20389
 URL: https://issues.apache.org/jira/browse/HIVE-20389
 Project: Hive
  Issue Type: Improvement
Reporter: Daniel Dai
Assignee: Daniel Dai


Introduced in HIVE-20118, get the following stack in schematool:
{code}
Caused by: java.lang.IllegalArgumentException: Null user
at 
org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1221)
 ~[hadoop-common-3.1.0.jar:?]
at 
org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1208)
 ~[hadoop-common-3.1.0.jar:?]
at 
org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator.getGroupNames(SessionStateUserAuthenticator.java:44)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.session.SessionState.getGroupsFromAuthenticator(SessionState.java:1288)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFCurrentGroups.initialize(GenericUDFCurrentGroups.java:53)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:148)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:260)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1215)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1516)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:241)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:187)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:12752)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:12707)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:12675)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:3469)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:3449)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10549)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11526)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11396)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12160)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:628)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12250)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:356)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
 ~[hive-exec-4.0.0-SNAPSHOT.jar

[jira] [Created] (HIVE-20357) Introduce initOrUpgradeSchema option to schema tool

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20357:
-

 Summary: Introduce initOrUpgradeSchema option to schema tool
 Key: HIVE-20357
 URL: https://issues.apache.org/jira/browse/HIVE-20357
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


Currently, schematool has two option: initSchema/upgradeSchema. User needs to 
use different command line for different action. However, from the schema 
version stored in db, we shall able to figure out if there's a need to 
init/upgrade, and choose the right action automatically.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20355) Clean up parameter of HiveConnection.setSchema

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20355:
-

 Summary: Clean up parameter of HiveConnection.setSchema
 Key: HIVE-20355
 URL: https://issues.apache.org/jira/browse/HIVE-20355
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Reporter: Daniel Dai
Assignee: Daniel Dai


Not immediately exploitable, as HS2 only allow one statement a time. But in 
future, we may support multiple statement in HiveStatement, so better to clean 
up the database parameter to avoid potential sql injection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20344) PrivilegeSynchronizer for SBA might hit AccessControlException

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20344:
-

 Summary: PrivilegeSynchronizer for SBA might hit 
AccessControlException
 Key: HIVE-20344
 URL: https://issues.apache.org/jira/browse/HIVE-20344
 Project: Hive
  Issue Type: Improvement
Reporter: Daniel Dai
Assignee: Daniel Dai


If "hive" user does not have privilege of corresponding hdfs folders, 
PrivilegeSynchronizer won't be able to get metadata of the table because SBA is 
preventing it. Here is a sample stack:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.security.AccessControlException: Permission denied: user=hive, 
access=EXECUTE, inode="/tmp/sba_is/sba_db":hrt_7:hrt_qa:dr
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:315)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:242)
at 
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:512)
at 
org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:305)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1850)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1834)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1784)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAccess(FSNamesystem.java:7767)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.checkAccess(NameNodeRpcServer.java:2217)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.checkAccess(ClientNamenodeProtocolServerSideTranslatorPB.java:1659)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)

at 
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(StorageBasedAuthorizationProvider.java:424)
at 
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(StorageBasedAuthorizationProvider.java:382)
at 
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(StorageBasedAuthorizationProvider.java:355)
at 
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(StorageBasedAuthorizationProvider.java:203)
at 
org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener.authorizeReadTable(AuthorizationPreEventListener.java:192)
... 23 more
{code}
I simply skip the table if that happens. In practice, managed tables are owned 
by "hive" user, so only external tables will be impacted. User need to grant 
execute permission of db folder and read permission of the table folders to 
"hive" user if they want to query the information schema for the tables, whose 
permission is only granted via SBA. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20130) Better logging for information schema synchronizer

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20130:
-

 Summary: Better logging for information schema synchronizer
 Key: HIVE-20130
 URL: https://issues.apache.org/jira/browse/HIVE-20130
 Project: Hive
  Issue Type: Improvement
Reporter: Daniel Dai
Assignee: Daniel Dai
 Attachments: HIVE-20130.1.patch

The logging of information schema synchronizer should be more useful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20118) SessionStateUserAuthenticator.getGroupNames() is always empty

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20118:
-

 Summary: SessionStateUserAuthenticator.getGroupNames() is always 
empty
 Key: HIVE-20118
 URL: https://issues.apache.org/jira/browse/HIVE-20118
 Project: Hive
  Issue Type: Improvement
  Components: Authorization
Reporter: Daniel Dai
Assignee: Daniel Dai






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20002) Shipping jdbd-storage-handler dependency jars in LLAP

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20002:
-

 Summary: Shipping jdbd-storage-handler dependency jars in LLAP
 Key: HIVE-20002
 URL: https://issues.apache.org/jira/browse/HIVE-20002
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Daniel Dai
Assignee: Daniel Dai
 Attachments: HIVE-20002.1.patch

Shipping the following jars to LLAP to make jdbc storage-handler work: 
commons-dbcp, commons-pool, db specific jdbc jar whichever exists in classpath.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19938) Upgrade scripts for information schema

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19938:
-

 Summary: Upgrade scripts for information schema
 Key: HIVE-19938
 URL: https://issues.apache.org/jira/browse/HIVE-19938
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


To make schematool -upgradeSchema work for information schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19920) Schematool fails in embedded mode when auth is on

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19920:
-

 Summary: Schematool fails in embedded mode when auth is on
 Key: HIVE-19920
 URL: https://issues.apache.org/jira/browse/HIVE-19920
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


This is a follow up of HIVE-19775. We need to override more properties in 
embedded hs2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19913) OWNER_TYPE is missing in some metastore upgrade script

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19913:
-

 Summary: OWNER_TYPE is missing in some metastore upgrade script
 Key: HIVE-19913
 URL: https://issues.apache.org/jira/browse/HIVE-19913
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


OWNER_TYPE introduced in HIVE-19372 is missing in upgrade-2.3.0-to-3.0.0.*.sql 
except derby.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19872) hive-schema-3.1.0.hive.sql is missing on master and branch-3

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19872:
-

 Summary: hive-schema-3.1.0.hive.sql is missing on master and 
branch-3
 Key: HIVE-19872
 URL: https://issues.apache.org/jira/browse/HIVE-19872
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


Information schema initialization will fail with "Unknown version specified for 
initialization: 3.1.0".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19862) Postgres init script has a glitch around UNIQUE_DATABASE

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19862:
-

 Summary: Postgres init script has a glitch around UNIQUE_DATABASE
 Key: HIVE-19862
 URL: https://issues.apache.org/jira/browse/HIVE-19862
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


{code}
ALTER TABLE ONLY "DBS" ADD CONSTRAINT "UNIQUE_DATABASE" UNIQUE ("NAME");
{code}
Should also include "CTLG_NAME".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19825) HiveServer2 leader selection shall use different zookeeper znode

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19825:
-

 Summary: HiveServer2 leader selection shall use different 
zookeeper znode
 Key: HIVE-19825
 URL: https://issues.apache.org/jira/browse/HIVE-19825
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Daniel Dai
Assignee: Daniel Dai


Currently, HiveServer2 leader selection (used only by privilegesynchronizer 
now) is reuse /hiveserver2 parent znode which is already used for HiveServer2 
service discovery. This interfere the service discovery. I'd like to switch to 
a different znode /hiveserver2-leader.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19813) SessionState.start don't have to be synchronized

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19813:
-

 Summary: SessionState.start don't have to be synchronized
 Key: HIVE-19813
 URL: https://issues.apache.org/jira/browse/HIVE-19813
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


This is introduced in HIVE-14690. However, only check-set block needs to be 
synchronized, not the whole block. The method will start Tez AM, which is a 
long operation. Make the method synchronized will serialize session start thus 
slow down hs2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19810:
-

 Summary: StorageHandler fail to ship jars in Tez intermittently
 Key: HIVE-19810
 URL: https://issues.apache.org/jira/browse/HIVE-19810
 Project: Hive
  Issue Type: Bug
  Components: Tez
Reporter: Daniel Dai
Assignee: Daniel Dai


Hive relies on StorageHandler to ship jars to backend automatically in several 
cases: JdbcStorageHandler, HBaseStorageHandler, AccumuloStorageHandler. This 
does not work reliably, in particular, the first dag in the session will have 
those jars, the second will not unless container is reused. In the later case, 
the containers allocated to first dag will be reused in the second dag so the 
container will have additional resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19737) Missing update schema version in 3.1 db scripts

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19737:
-

 Summary: Missing update schema version in 3.1 db scripts
 Key: HIVE-19737
 URL: https://issues.apache.org/jira/browse/HIVE-19737
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai
 Attachments: HIVE-19737.1.patch

I miss several places to update schema version string in 
standalone-metastore/src/main/sql/xxx/hive-schema-3.1.0.xxx.sql when creating 
those scripts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67102: HIVE-19440: Make StorageBasedAuthorizer work with information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67102/
---

(Updated May 15, 2018, 9:04 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19440


Diffs (updated)
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/storagehandler/DummyHCatAuthProvider.java
 a53028f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestHDFSPermissionPolicyProvider.java
 PRE-CREATION 
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/security/DummyHiveMetastoreAuthorizationProvider.java
 31e795c 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql d9606d8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a1f549a 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HDFSPermissionPolicyProvider.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProviderBase.java
 8a7c06d 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveMetastoreAuthorizationProvider.java
 0dab334 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PolicyProviderContainer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 9b2e6cd 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java
 b66d188 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveV1Authorizer.java
 48798d8 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLAuthorizationUtils.java
 02ed7aa 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentAuthorizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 3eb0914 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 661beb5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 92d2e3f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 6af2aa5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 09f9bb1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 264fdb9 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 ce7d286 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 b223920 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/client/builder/HiveObjectPrivilegeBuilder.java
 d802e1a 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MDBPrivilege.java
 3d8fa21 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MGlobalPrivilege.java
 5b496e0 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MPartitionColumnPrivilege.java
 ab50a92 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MPartitionPrivilege.java
 3193bc1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTableColumnPrivilege.java
 ad7322f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTablePrivilege.java
 6460400 
  standalone-metastore/src/main/resources/package.jdo 2d2cb19 
  standalone-metastore/src/main/sql/derby/hive-schema-3.1.0.derby.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/derby/upgrade-3.0.0-to-3.1.0.derby.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mssql/hive-schema-3.1.0.mssql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mssql/upgrade-3.0.0-to-3.1.0.mssql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mysql/hive-schema-3.1.0.mysql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mysql/upgrade-3.0.0-to-3.1.0.mysql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/oracle/hive-schema-3.1.0.oracle.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/oracle/upgrade-3.0.0-to-3.1.0.oracle.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/postgres/hive-schema-3.1.0.postgres.sql 
PRE-CREATION 
  
standalone-metastore/src/main/sql/postgres/upgrade-3.0.0-to-3.1.0.postgres.sql 
PRE-CREATION 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 19d4433 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f6899be 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 98a85cc 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7186add 


Diff: https://reviews.apache.org/r/67102/diff/2/

Changes: https://reviews.apache.org/r/67102/diff/1-2/


Testing
---


Thanks,

Daniel Dai

Review Request 67102: HIVE-19440: Make StorageBasedAuthorizer work with information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67102/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-19440


Diffs
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/storagehandler/DummyHCatAuthProvider.java
 a53028f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestHDFSPermissionPolicyProvider.java
 PRE-CREATION 
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/security/DummyHiveMetastoreAuthorizationProvider.java
 31e795c 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql d9606d8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 4611ce9 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HDFSPermissionPolicyProvider.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProviderBase.java
 8a7c06d 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveMetastoreAuthorizationProvider.java
 0dab334 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PolicyProviderContainer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 9b2e6cd 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java
 b66d188 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveV1Authorizer.java
 48798d8 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLAuthorizationUtils.java
 02ed7aa 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentAuthorizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 3eb0914 
  service/src/java/org/apache/hive/service/server/HiveServer2.java cf3edbf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 3978b88 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 6af2aa5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 09f9bb1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 b43334b 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 ce7d286 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 b223920 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/client/builder/HiveObjectPrivilegeBuilder.java
 d802e1a 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MDBPrivilege.java
 3d8fa21 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MGlobalPrivilege.java
 5b496e0 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MPartitionColumnPrivilege.java
 ab50a92 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MPartitionPrivilege.java
 3193bc1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTableColumnPrivilege.java
 ad7322f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTablePrivilege.java
 6460400 
  standalone-metastore/src/main/resources/package.jdo 2d2cb19 
  standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql 8e09755 
  standalone-metastore/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql 
73bef36 
  standalone-metastore/src/main/sql/mssql/hive-schema-3.0.0.mssql.sql 51df92c 
  standalone-metastore/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 
a7232dd 
  standalone-metastore/src/main/sql/mysql/hive-schema-3.0.0.mysql.sql 8e55e94 
  standalone-metastore/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql 
9a48346 
  standalone-metastore/src/main/sql/oracle/hive-schema-3.0.0.oracle.sql 3a12e08 
  standalone-metastore/src/main/sql/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 
3be7e65 
  standalone-metastore/src/main/sql/postgres/hive-schema-3.0.0.postgres.sql 
0152f48 
  
standalone-metastore/src/main/sql/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
fed8a93 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 19d4433 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f6899be 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 98a85cc 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7186add 


Diff: https://reviews.apache.org/r/67102/diff/1/


Testing
---


Thanks,

Daniel Dai

[jira] [Created] (HIVE-19440) Make StorageBasedAuthorizer work with information schema

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19440:
-

 Summary: Make StorageBasedAuthorizer work with information schema
 Key: HIVE-19440
 URL: https://issues.apache.org/jira/browse/HIVE-19440
 Project: Hive
  Issue Type: Improvement
Reporter: Daniel Dai
Assignee: Daniel Dai


With HIVE-19161, Hive information schema works with external authorizer (such 
as ranger). However, we also need to make StorageBasedAuthorizer 
synchronization work as it is also widely use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19381) Function replication in cloud fail when download resource from AWS

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19381:
-

 Summary: Function replication in cloud fail when download resource 
from AWS
 Key: HIVE-19381
 URL: https://issues.apache.org/jira/browse/HIVE-19381
 Project: Hive
  Issue Type: Bug
  Components: repl
Reporter: Daniel Dai
Assignee: Daniel Dai
 Fix For: 3.0.0, 3.1.0


Another case replication shall use the config in with clause.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 28, 2018, 1:09 a.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/HiveAccumuloHelper.java
 9fccb49 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/Utils.java 3a2facf 
  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/CompositeAccumuloRowIdFactory.java
 d8b9aa3 
  
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/DefaultAccumuloRowIdFactory.java
 bae2930 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java f40c606 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 8ecbaad 
  itests/hive-unit/pom.xml 3ae7f2f 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestRestrictedList.java 
79fdb68 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java 
df55272 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 6d3c8d9 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 772bc5d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/JarUtils.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 60b63d4 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 9850276 
  ql/src/test/results/clientpositive/show_functions.q.out 4df555b 
  service/src/java/org/apache/hive/service/server/HiveServer2.java e373628 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 397a081 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 1c8d223 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 aee416d 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 184ecb6 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 2c9f2e5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 92d000b 
  standalone-metastore/src/main/thrift/hive_metastore.thrift c56a4f9 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 defc68f 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 20c5d8a 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 bf87cfc 


Diff: https://reviews.apache.org/r/66571/diff/6/

Changes: https://reviews.apache.org/r/66571/diff/5-6/


Testing
---


Thanks,

Daniel Dai

[jira] [Created] (HIVE-19331) Repl load config in "with" clause not pass to Context.getStagingDir

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19331:
-

 Summary: Repl load config in "with" clause not pass to 
Context.getStagingDir
 Key: HIVE-19331
 URL: https://issues.apache.org/jira/browse/HIVE-19331
 Project: Hive
  Issue Type: Bug
  Components: repl
Reporter: Daniel Dai
Assignee: Daniel Dai


Another failure similar to HIVE-18626, causing exception when s3 credentials 
are in "REPL LOAD" with clause.

{code}
Caused by: java.lang.IllegalStateException: Error getting FileSystem for 
s3a://nat-yc-r7-nmys-beacon-cloud-s3-2/hive_incremental_testing.db/hive_incremental_testing_new_tabl...:
 org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on 
nat-yc-r7-nmys-beacon-cloud-s3-2: com.amazonaws.AmazonClientException: No AWS 
Credentials provided by BasicAWSCredentialsProvider 
EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider 
: com.amazonaws.AmazonClientException: Unable to load credentials from Amazon 
EC2 metadata service: No AWS Credentials provided by 
BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
SharedInstanceProfileCredentialsProvider : com.amazonaws.AmazonClientException: 
Unable to load credentials from Amazon EC2 metadata service
at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:359)
at 
org.apache.hadoop.hive.ql.Context.getExternalScratchDir(Context.java:487)
at 
org.apache.hadoop.hive.ql.Context.getExternalTmpPath(Context.java:565)
at 
org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.loadTable(ImportSemanticAnalyzer.java:370)
at 
org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.createReplImportTasks(ImportSemanticAnalyzer.java:926)
at 
org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.prepareImport(ImportSemanticAnalyzer.java:329)
at 
org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.handle(TableHandler.java:43)
... 24 more
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66503: HIVE-19126: CachedStore: Use memory estimation to limit cache size during prewarm

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/#review201706
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
Lines 52 (patched)
<https://reviews.apache.org/r/66503/#comment283258>

standalone-metastore shall not depend on ql.


- Daniel Dai


On April 16, 2018, 9:42 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66503/
> ---
> 
> (Updated April 16, 2018, 9:42 p.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Bugs: HIVE-19126
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
>  6f4ec6f1ea 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
>  2f7fa24558 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  0bbaf7e459 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  1ce86bbdba 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  89b400697b 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  f007261daf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  d451f966b0 
> 
> 
> Diff: https://reviews.apache.org/r/66503/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[jira] [Created] (HIVE-19251) ObjectStore.getNextNotification with LIMIT should use less memory

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19251:
-

 Summary: ObjectStore.getNextNotification with LIMIT should use 
less memory
 Key: HIVE-19251
 URL: https://issues.apache.org/jira/browse/HIVE-19251
 Project: Hive
  Issue Type: Bug
  Components: repl, Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


Experience OOM when Hive metastore try to retrieve huge amount of notification 
logs even there's limit clause. Hive shall only retrieve necessary rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 14, 2018, 7:06 a.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e533ee6 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml 05c362e 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 7dc690f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 178c97d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 47f84b5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 565549a 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 9a43b2c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 98674cf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 c5da7b5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 1ce86bb 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 612afe1 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 ecddc7b 


Diff: https://reviews.apache.org/r/66571/diff/5/

Changes: https://reviews.apache.org/r/66571/diff/4-5/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 13, 2018, 10:30 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 7dc690f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 178c97d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/4/

Changes: https://reviews.apache.org/r/66571/diff/3-4/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 13, 2018, 8:49 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 7dc690f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 178c97d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/3/

Changes: https://reviews.apache.org/r/66571/diff/2-3/


Testing
---


Thanks,

Daniel Dai

Re: Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

(Updated April 13, 2018, 8:48 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/DatabaseAccessorFactory.java
 7dc690f 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
 178c97d 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/JdbcRecordIterator.java
 638e2b0 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/MsSqlDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/OracleDatabaseAccessor.java
 PRE-CREATION 
  
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/PostgresDatabaseAccessor.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProvider.java
 60d9dc1 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLsImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/2/

Changes: https://reviews.apache.org/r/66571/diff/1-2/


Testing
---


Thanks,

Daniel Dai

Review Request 66571: HIVE-19161: Add authorizations to information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLs.java
 53e221f 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/1/


Testing
---


Thanks,

Daniel Dai

[jira] [Created] (HIVE-19161) Add authorizations to information schema

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-19161:
-

 Summary: Add authorizations to information schema
 Key: HIVE-19161
 URL: https://issues.apache.org/jira/browse/HIVE-19161
 Project: Hive
  Issue Type: Sub-task
Reporter: Daniel Dai
Assignee: Daniel Dai


We need to control the access of information schema so user can only query the 
information authorized to.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[SECURITY] CVE-2018-1284: Hive UDF series UDFXPathXXXX allow users to pass carefully crafted XML to access arbitrary files

[Daniel Dai]

CVE-2018-1284: Hive UDF series UDFXPath allow users to pass
carefully crafted XML to access arbitrary files

Severity: Important

Vendor: The Apache Software Foundation

Versions Affected: This vulnerability affects all versions from 0.6.0

Description: Malicious user might use any xpath UDFs
(xpath/xpath_string/xpath_boolean/xpath_number/xpath_double/xpath_float/xpath_long/xpath_int/xpath_short)
to expose the content of a file on the machine running HiveServer2
owned by HiveServer2 user (usually hive) if
hive.server2.enable.doAs=false.

Mitigation: Users who use xpath UDFs in HiveServer2 and
hive.server2.enable.doAs=false are recommended to upgrade to 2.3.3, or
update UDFXPathUtil.java to the head of branch-2.3 and rebuild
hive-exec.jar: 
https://git1-us-west.apache.org/repos/asf?p=hive.git;a=blob;f=ql/src/java/org/apache/hadoop/hive/ql/udf/xml/UDFXPathUtil.java;hb=refs/heads/branch-2.3.
If these functions are not being used at present, you can also
disable its use by adding them to the value of the config
hive.server2.builtin.udf.blacklist.

[SECURITY] CVE-2018-1282 JDBC driver is susceptible to SQL injection attack if the input parameters are not properly cleaned

[Daniel Dai]

CVE-2018-1282: JDBC driver is susceptible to SQL injection attack if
the input parameters are not properly cleaned

Severity: Important

Vendor: The Apache Software Foundation

Versions Affected: This vulnerability affects all versions of Hive
JDBC driver from 0.7.1

Description: This vulnerability in Hive allows carefully crafted arguments to be
used to bypass the argument escaping/cleanup that JDBC driver does in
PreparedStatement implementation.

Mitigation: It is recommended to upgrade prior version of Hive JDBC
driver to 2.3.3.
Note Hive JDBC driver is not backward compatible with HiveServer2,
which means newer version of Hive JDBC driver may not talk to older version
of HiveServer2. In particular, Hive JDBC driver 2.3.3 won't talk
to HiveServer2 2.1.1 or prior. If user is using Hive code 2.1.1 or below
they might need to upgrade all the Hive instances to 2.3.3.


Alternative to the upgrade, is to take the follow two actions in your
Hive JDBC client code/application when dealing with user provided
input in PreparedStatement:
1. Avoid passing user input PreparedStatement.setBinaryStream
2. Sanitize the user input for PreparedStatement.setString, by
replacing all occurrences of \' to '

Credit: This issue was discovered by Bear Giles of SnapLogic

[SECURITY] CVE-2018-1315 'COPY FROM FTP' statement in HPL/SQL can write to arbitrary location if the FTP server is compromised

[Daniel Dai]

CVE-2018-1315: 'COPY FROM FTP' statement in HPL/SQL can write to
arbitrary location if the FTP server is compromised

Severity: Moderate

Vendor: The Apache Software Foundation

Versions Affected: Hive 2.1.0 to 2.3.2

Description: When 'COPY FROM FTP' statement is run using HPL/SQL extension to
Hive, a compromised/malicious FTP server can cause the file to be
written to an arbitrary location on the cluster where the command is
run from. This is because FTP client code in HPL/SQL does not verify
the destination
location of the downloaded file. This does not affect hive
cli user and hiveserver2 user as hplsql is a separate command line
script and needs to be invoked differently.

Mitigation: User who use HPL/SQL with Hive 2.1.0 through 2.3.2 should upgrade to
2.3.3 which removes support for "COPY FROM FTP". Alternatively, the
usage of HPL/SQL can be disabled through
other means.

Credit: This issue was discovered by Danny Grander of Snyk