[jira] [Work logged] (HIVE-25605) JdbcStorageHandler Create table fails when hive.sql.schema is specified and is not the default one
[
https://issues.apache.org/jira/browse/HIVE-25605?focusedWorklogId=663110=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-663110
]
ASF GitHub Bot logged work on HIVE-25605:
-
Author: ASF GitHub Bot
Created on: 09/Oct/21 03:49
Start Date: 09/Oct/21 03:49
Worklog Time Spent: 10m
Work Description: asolimando commented on a change in pull request #2710:
URL: https://github.com/apache/hive/pull/2710#discussion_r725426532
##
File path:
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java
##
@@ -125,9 +125,15 @@ public static String getConfigValue(JdbcStorageConfig key,
Configuration config)
public static String getOrigQueryToExecute(Configuration config) {
String query;
String tableName = config.get(Constants.JDBC_TABLE);
+String schemaName = config.get(Constants.JDBC_SCHEMA);
if (tableName != null) {
// We generate query as select *
- query = "select * from " + tableName;
+ if (schemaName != null) {
+query = "select * from "+schemaName+"."+tableName;
Review comment:
Can you put whitespaces before/after "+"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 663110)
Time Spent: 20m (was: 10m)
> JdbcStorageHandler Create table fails when hive.sql.schema is specified and
> is not the default one
> --
>
> Key: HIVE-25605
> URL: https://issues.apache.org/jira/browse/HIVE-25605
> Project: Hive
> Issue Type: Bug
> Components: JDBC storage handler
>Affects Versions: 4.0.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We have observed create table statement failure for JdbcStorageHandler with
> Oracle when Schema name is specified in Table properties and that schema is
> not the default one for user.
> eg:-
> Consider Username: DI_METADATA with default schema DI_METADATA in Oracle,
> however this user has access to other schemas as well like schema name
> CHIRAN, when using below create statement in Hive it fails with error
> {code}
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
> MetaException(message:org.apache.hadoop.hive.serde2.SerDeException
> org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Error
> while trying to get column names: ORA-00942: table or view does not exist
> {code}
> *Create Statement*:
> {code}
> CREATE EXTERNAL TABLE if not exists
> query_fed_oracle.ABCD_TEST_pw_case_jceks_diff(
> YEAR INT,
> QUANTITY INT,
> NAME STRING
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "bucketing_version"="2",
> "hive.sql.database.type" = "ORACLE",
> "hive.sql.jdbc.driver" = "oracle.jdbc.OracleDriver",
> "hive.sql.jdbc.url" =
> "jdbc:oracle:thin:@//obfuscated.compute-1.amazonaws.com",
> "hive.sql.dbcp.username" = "DI_METADATA",
> "hive.sql.dbcp.password.keystore" =
> "jceks://s3a@obfuscated-bucket/test.jceks",
> "hive.sql.dbcp.password.key" = "oracle.secret",
> "hive.sql.schema" = "CHIRAN",
> "hive.sql.table" = "ABCD_TEST_1",
> "hive.sql.dbcp.maxActive" = "1"
> );
> {code}
> This can be fixed by using "hive.sql.table" = "CHIRAN.ABCD_TEST_1", but this
> will break CBO as pushdown wont happen. Possible fix would be to include
> schemaName check too after below call.
> https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java#L166
> Attaching patch 1. Let me know if this looks good.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426417#comment-17426417
]
Szehon Ho edited comment on HIVE-25522 at 10/8/21, 11:46 PM:
-
Update, as I posted on the PR, realized its not affecting current master branch
with default settings but only 3.x, because of new added AcidMetricsService
which is initialized eagerly and will fail HMS rather than leave the static
TxnHandler fields in the corrupt state.
So removing 4.x from affected issues (though it might be an issue if
AcidMetricsService is explicitly disabled).
was (Author: szehon):
Update, as I posted on the PR, realized its not affecting current master branch
with default settings but only 3.x, because of new added AcidMetricsService
which is initialized eagerly and will fail HMS rather than leave the static
TxnHandler fields in the corrupt state. So taking it off of affected issues
(though it might be an issue if AcidMetricsService is explicitly disabled).
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
>
[jira] [Commented] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426417#comment-17426417
]
Szehon Ho commented on HIVE-25522:
--
Update, as I posted on the PR, realized its not affecting current master branch
with default settings but only 3.x, because of new added AcidMetricsService
which is initialized eagerly and will fail HMS rather than leave the static
TxnHandler fields in the corrupt state. So taking it off of affected issues
(though it might be an issue if AcidMetricsService is explicitly disabled).
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
>
[jira] [Updated] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-25522:
-
Affects Version/s: (was: 4.0.0)
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
>
[jira] [Work logged] (HIVE-25514) Alter table with partitions should honor {OWNER} policies from Apache Ranger in the HMS
[
https://issues.apache.org/jira/browse/HIVE-25514?focusedWorklogId=663042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-663042
]
ASF GitHub Bot logged work on HIVE-25514:
-
Author: ASF GitHub Bot
Created on: 08/Oct/21 23:34
Start Date: 08/Oct/21 23:34
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on a change in pull
request #2634:
URL: https://github.com/apache/hive/pull/2634#discussion_r725392290
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -5461,6 +5462,37 @@ public GetPartitionResponse
get_partition_req(GetPartitionRequest req)
*/
private void fireReadTablePreEvent(String catName, String dbName, String
tblName)
throws MetaException, NoSuchObjectException {
+if (catName == null) {
+ throw new NullPointerException("catName is null");
+}
+
+if (isBlank(catName)) {
+ throw new NoSuchObjectException("catName is not valid");
+}
+
+if (dbName == null) {
+ throw new NullPointerException("dbName is null");
+}
+
+if (isBlank(dbName)) {
+ throw new NoSuchObjectException("dbName is not valid");
+}
+
+List filteredDb =
FilterUtils.filterDbNamesIfEnabled(isServerFilterEnabled, filterHook,
+Collections.singletonList(dbName));
+
+if (filteredDb.isEmpty()) {
+ throw new NoSuchObjectException("Database " + dbName + " does not
exist");
+}
+
+if (tblName == null) {
Review comment:
Ack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 663042)
Time Spent: 1h 20m (was: 1h 10m)
> Alter table with partitions should honor {OWNER} policies from Apache Ranger
> in the HMS
> ---
>
> Key: HIVE-25514
> URL: https://issues.apache.org/jira/browse/HIVE-25514
> Project: Hive
> Issue Type: Bug
> Components: Hive, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The following commands should honor \{OWNER} policies from Apache Ranger in
> the HMS.
> {code:java}
> Show partitions table_name;
> alter table foo.table_name partition (country='us') rename to partition
> (country='canada);
> alter table foo.table_name drop partition (id='canada');{code}
> The examples above are tables with partitions. So the partition APIs in HMS
> should be modifed to honor \{owner} policies from Apache ranger.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25514) Alter table with partitions should honor {OWNER} policies from Apache Ranger in the HMS
[
https://issues.apache.org/jira/browse/HIVE-25514?focusedWorklogId=663041=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-663041
]
ASF GitHub Bot logged work on HIVE-25514:
-
Author: ASF GitHub Bot
Created on: 08/Oct/21 23:32
Start Date: 08/Oct/21 23:32
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on a change in pull
request #2634:
URL: https://github.com/apache/hive/pull/2634#discussion_r725391998
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -5461,6 +5462,37 @@ public GetPartitionResponse
get_partition_req(GetPartitionRequest req)
*/
private void fireReadTablePreEvent(String catName, String dbName, String
tblName)
throws MetaException, NoSuchObjectException {
+if (catName == null) {
+ throw new NullPointerException("catName is null");
+}
+
+if (isBlank(catName)) {
Review comment:
Yeah. I tried doing it but few tests in TestGetPartitions is failing
because it is expecting to fail with NullPointerException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 663041)
Time Spent: 1h 10m (was: 1h)
> Alter table with partitions should honor {OWNER} policies from Apache Ranger
> in the HMS
> ---
>
> Key: HIVE-25514
> URL: https://issues.apache.org/jira/browse/HIVE-25514
> Project: Hive
> Issue Type: Bug
> Components: Hive, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The following commands should honor \{OWNER} policies from Apache Ranger in
> the HMS.
> {code:java}
> Show partitions table_name;
> alter table foo.table_name partition (country='us') rename to partition
> (country='canada);
> alter table foo.table_name drop partition (id='canada');{code}
> The examples above are tables with partitions. So the partition APIs in HMS
> should be modifed to honor \{owner} policies from Apache ranger.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?focusedWorklogId=662985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662985
]
ASF GitHub Bot logged work on HIVE-25522:
-
Author: ASF GitHub Bot
Created on: 08/Oct/21 21:30
Start Date: 08/Oct/21 21:30
Worklog Time Spent: 10m
Work Description: szehon-ho commented on pull request #2647:
URL: https://github.com/apache/hive/pull/2647#issuecomment-939126615
Looks like random failure, as previous run succeeded (intermediate change is
only to add a license header), retriggering
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 662985)
Time Spent: 6.5h (was: 6h 20m)
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
>
[jira] [Comment Edited] (HIVE-25505) Incorrect results with header. skip.header.line.count if first line is blank
[
https://issues.apache.org/jira/browse/HIVE-25505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426212#comment-17426212
]
Panagiotis Garefalakis edited comment on HIVE-25505 at 10/8/21, 8:39 PM:
-
I have a repro for this – updating description and assigning to myself
was (Author: pgaref):
I have a repro for this -- updating description and assigned to myself
> Incorrect results with header. skip.header.line.count if first line is blank
>
>
> Key: HIVE-25505
> URL: https://issues.apache.org/jira/browse/HIVE-25505
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Steve Carlin
>Assignee: Panagiotis Garefalakis
>Priority: Major
>
> aAtable with header. skip.header.line.count=1 does not skip the first line if
> it is blank, except in a fetch task.
> To reproduce, create a csv table, ans set header. skip.header.line.count=1 in
> table properties.
> In the table location, create a single file, with a blank (empty) first line,
> and say 2 further lines.
> If you do a select * on it, you see 2 rows (correct)
> If you do select count(*) on it, you get 3 (incorrect)
> {code:java}
> CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
> LOCATION '${system:test.tmp.dir}/testcase1'
> TBLPROPERTIES ("skip.header.line.count"="1");
> SET hive.fetch.task.conversion = more;
> select * from testcase1;
> select count(*) from testcase1;
> set hive.fetch.task.conversion=none;
> select * from testcase1;
> select count(*) from testcase1;
> Test file:
> 1,2019-12-31
> 2,2019-12-31
> 3,2019-12-31
> Should both yield (with the above test file):
> A masked pattern was here
> 1 2019-12-31
> 2 2019-12-31
> 3 2019-12-31
> 3
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25521) Data corruption when concatenating files with different compressions in same table/partition
[ https://issues.apache.org/jira/browse/HIVE-25521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis resolved HIVE-25521. --- Resolution: Fixed > Data corruption when concatenating files with different compressions in same > table/partition > > > Key: HIVE-25521 > URL: https://issues.apache.org/jira/browse/HIVE-25521 > Project: Hive > Issue Type: Bug >Reporter: Harish JP >Assignee: Harish JP >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently if files of different compressions are in same directory then > concatenate can fail and cause data corruption. This happens because file can > be moved by one task as incompatible file and the other tasks will fail after > this. > > This issue is addressed in this Jira by only processing a file in one task > where offset 0 is process and ignoring the the file in all other tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25521) Data corruption when concatenating files with different compressions in same table/partition
[ https://issues.apache.org/jira/browse/HIVE-25521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426356#comment-17426356 ] Panagiotis Garefalakis commented on HIVE-25521: --- Resolved via https://github.com/apache/hive/pull/2639 > Data corruption when concatenating files with different compressions in same > table/partition > > > Key: HIVE-25521 > URL: https://issues.apache.org/jira/browse/HIVE-25521 > Project: Hive > Issue Type: Bug >Reporter: Harish JP >Assignee: Harish JP >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently if files of different compressions are in same directory then > concatenate can fail and cause data corruption. This happens because file can > be moved by one task as incompatible file and the other tasks will fail after > this. > > This issue is addressed in this Jira by only processing a file in one task > where offset 0 is process and ignoring the the file in all other tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25521) Data corruption when concatenating files with different compressions in same table/partition
[ https://issues.apache.org/jira/browse/HIVE-25521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-25521: -- Fix Version/s: 4.0.0 > Data corruption when concatenating files with different compressions in same > table/partition > > > Key: HIVE-25521 > URL: https://issues.apache.org/jira/browse/HIVE-25521 > Project: Hive > Issue Type: Bug >Reporter: Harish JP >Assignee: Harish JP >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently if files of different compressions are in same directory then > concatenate can fail and cause data corruption. This happens because file can > be moved by one task as incompatible file and the other tasks will fail after > this. > > This issue is addressed in this Jira by only processing a file in one task > where offset 0 is process and ignoring the the file in all other tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25605) JdbcStorageHandler Create table fails when hive.sql.schema is specified and is not the default one
[
https://issues.apache.org/jira/browse/HIVE-25605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani reassigned HIVE-25605:
Assignee: Chiran Ravani
> JdbcStorageHandler Create table fails when hive.sql.schema is specified and
> is not the default one
> --
>
> Key: HIVE-25605
> URL: https://issues.apache.org/jira/browse/HIVE-25605
> Project: Hive
> Issue Type: Bug
> Components: JDBC storage handler
>Affects Versions: 4.0.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We have observed create table statement failure for JdbcStorageHandler with
> Oracle when Schema name is specified in Table properties and that schema is
> not the default one for user.
> eg:-
> Consider Username: DI_METADATA with default schema DI_METADATA in Oracle,
> however this user has access to other schemas as well like schema name
> CHIRAN, when using below create statement in Hive it fails with error
> {code}
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
> MetaException(message:org.apache.hadoop.hive.serde2.SerDeException
> org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Error
> while trying to get column names: ORA-00942: table or view does not exist
> {code}
> *Create Statement*:
> {code}
> CREATE EXTERNAL TABLE if not exists
> query_fed_oracle.ABCD_TEST_pw_case_jceks_diff(
> YEAR INT,
> QUANTITY INT,
> NAME STRING
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "bucketing_version"="2",
> "hive.sql.database.type" = "ORACLE",
> "hive.sql.jdbc.driver" = "oracle.jdbc.OracleDriver",
> "hive.sql.jdbc.url" =
> "jdbc:oracle:thin:@//obfuscated.compute-1.amazonaws.com",
> "hive.sql.dbcp.username" = "DI_METADATA",
> "hive.sql.dbcp.password.keystore" =
> "jceks://s3a@obfuscated-bucket/test.jceks",
> "hive.sql.dbcp.password.key" = "oracle.secret",
> "hive.sql.schema" = "CHIRAN",
> "hive.sql.table" = "ABCD_TEST_1",
> "hive.sql.dbcp.maxActive" = "1"
> );
> {code}
> This can be fixed by using "hive.sql.table" = "CHIRAN.ABCD_TEST_1", but this
> will break CBO as pushdown wont happen. Possible fix would be to include
> schemaName check too after below call.
> https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java#L166
> Attaching patch 1. Let me know if this looks good.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25605) JdbcStorageHandler Create table fails when hive.sql.schema is specified and is not the default one
[
https://issues.apache.org/jira/browse/HIVE-25605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25605:
--
Labels: pull-request-available (was: )
> JdbcStorageHandler Create table fails when hive.sql.schema is specified and
> is not the default one
> --
>
> Key: HIVE-25605
> URL: https://issues.apache.org/jira/browse/HIVE-25605
> Project: Hive
> Issue Type: Bug
> Components: JDBC storage handler
>Affects Versions: 4.0.0
>Reporter: Chiran Ravani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We have observed create table statement failure for JdbcStorageHandler with
> Oracle when Schema name is specified in Table properties and that schema is
> not the default one for user.
> eg:-
> Consider Username: DI_METADATA with default schema DI_METADATA in Oracle,
> however this user has access to other schemas as well like schema name
> CHIRAN, when using below create statement in Hive it fails with error
> {code}
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
> MetaException(message:org.apache.hadoop.hive.serde2.SerDeException
> org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Error
> while trying to get column names: ORA-00942: table or view does not exist
> {code}
> *Create Statement*:
> {code}
> CREATE EXTERNAL TABLE if not exists
> query_fed_oracle.ABCD_TEST_pw_case_jceks_diff(
> YEAR INT,
> QUANTITY INT,
> NAME STRING
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "bucketing_version"="2",
> "hive.sql.database.type" = "ORACLE",
> "hive.sql.jdbc.driver" = "oracle.jdbc.OracleDriver",
> "hive.sql.jdbc.url" =
> "jdbc:oracle:thin:@//obfuscated.compute-1.amazonaws.com",
> "hive.sql.dbcp.username" = "DI_METADATA",
> "hive.sql.dbcp.password.keystore" =
> "jceks://s3a@obfuscated-bucket/test.jceks",
> "hive.sql.dbcp.password.key" = "oracle.secret",
> "hive.sql.schema" = "CHIRAN",
> "hive.sql.table" = "ABCD_TEST_1",
> "hive.sql.dbcp.maxActive" = "1"
> );
> {code}
> This can be fixed by using "hive.sql.table" = "CHIRAN.ABCD_TEST_1", but this
> will break CBO as pushdown wont happen. Possible fix would be to include
> schemaName check too after below call.
> https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java#L166
> Attaching patch 1. Let me know if this looks good.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25605) JdbcStorageHandler Create table fails when hive.sql.schema is specified and is not the default one
[
https://issues.apache.org/jira/browse/HIVE-25605?focusedWorklogId=662951=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662951
]
ASF GitHub Bot logged work on HIVE-25605:
-
Author: ASF GitHub Bot
Created on: 08/Oct/21 20:16
Start Date: 08/Oct/21 20:16
Worklog Time Spent: 10m
Work Description: cravani opened a new pull request #2710:
URL: https://github.com/apache/hive/pull/2710
…a is specified and is not the default one
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 662951)
Remaining Estimate: 0h
Time Spent: 10m
> JdbcStorageHandler Create table fails when hive.sql.schema is specified and
> is not the default one
> --
>
> Key: HIVE-25605
> URL: https://issues.apache.org/jira/browse/HIVE-25605
> Project: Hive
> Issue Type: Bug
> Components: JDBC storage handler
>Affects Versions: 4.0.0
>Reporter: Chiran Ravani
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We have observed create table statement failure for JdbcStorageHandler with
> Oracle when Schema name is specified in Table properties and that schema is
> not the default one for user.
> eg:-
> Consider Username: DI_METADATA with default schema DI_METADATA in Oracle,
> however this user has access to other schemas as well like schema name
> CHIRAN, when using below create statement in Hive it fails with error
> {code}
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
> MetaException(message:org.apache.hadoop.hive.serde2.SerDeException
> org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Error
> while trying to get column names: ORA-00942: table or view does not exist
> {code}
> *Create Statement*:
> {code}
> CREATE EXTERNAL TABLE if not exists
> query_fed_oracle.ABCD_TEST_pw_case_jceks_diff(
> YEAR INT,
> QUANTITY INT,
> NAME STRING
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "bucketing_version"="2",
> "hive.sql.database.type" = "ORACLE",
> "hive.sql.jdbc.driver" = "oracle.jdbc.OracleDriver",
> "hive.sql.jdbc.url" =
> "jdbc:oracle:thin:@//obfuscated.compute-1.amazonaws.com",
> "hive.sql.dbcp.username" = "DI_METADATA",
> "hive.sql.dbcp.password.keystore" =
> "jceks://s3a@obfuscated-bucket/test.jceks",
> "hive.sql.dbcp.password.key" = "oracle.secret",
> "hive.sql.schema" = "CHIRAN",
> "hive.sql.table" = "ABCD_TEST_1",
> "hive.sql.dbcp.maxActive" = "1"
> );
> {code}
> This can be fixed by using "hive.sql.table" = "CHIRAN.ABCD_TEST_1", but this
> will break CBO as pushdown wont happen. Possible fix would be to include
> schemaName check too after below call.
> https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java#L166
> Attaching patch 1. Let me know if this looks good.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25605) JdbcStorageHandler Create table fails when hive.sql.schema is specified and is not the default one
[
https://issues.apache.org/jira/browse/HIVE-25605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-25605:
-
Description:
We have observed create table statement failure for JdbcStorageHandler with
Oracle when Schema name is specified in Table properties and that schema is not
the default one for user.
eg:-
Consider Username: DI_METADATA with default schema DI_METADATA in Oracle,
however this user has access to other schemas as well like schema name CHIRAN,
when using below create statement in Hive it fails with error
{code}
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
MetaException(message:org.apache.hadoop.hive.serde2.SerDeException
org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Error
while trying to get column names: ORA-00942: table or view does not exist
{code}
*Create Statement*:
{code}
CREATE EXTERNAL TABLE if not exists
query_fed_oracle.ABCD_TEST_pw_case_jceks_diff(
YEAR INT,
QUANTITY INT,
NAME STRING
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"bucketing_version"="2",
"hive.sql.database.type" = "ORACLE",
"hive.sql.jdbc.driver" = "oracle.jdbc.OracleDriver",
"hive.sql.jdbc.url" =
"jdbc:oracle:thin:@//obfuscated.compute-1.amazonaws.com",
"hive.sql.dbcp.username" = "DI_METADATA",
"hive.sql.dbcp.password.keystore" =
"jceks://s3a@obfuscated-bucket/test.jceks",
"hive.sql.dbcp.password.key" = "oracle.secret",
"hive.sql.schema" = "CHIRAN",
"hive.sql.table" = "ABCD_TEST_1",
"hive.sql.dbcp.maxActive" = "1"
);
{code}
This can be fixed by using "hive.sql.table" = "CHIRAN.ABCD_TEST_1", but this
will break CBO as pushdown wont happen. Possible fix would be to include
schemaName check too after below call.
https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java#L166
Attaching patch 1. Let me know if this looks good.
was:
We have observed create table statement failure for JdbcStorageHandler with
Oracle when Schema name is specified in Table properties and that schema is not
the default one for user.
eg:-
Consider Username: DI_METADATA with default schema DI_METADATA in Oracle,
however this user has access to other schemas as well like schema name CHIRAN,
when using below create statement in Hive it fails with error
{code}
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
MetaException(message:org.apache.hadoop.hive.serde2.SerDeException
org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Error
while trying to get column names: ORA-00942: table or view does not exist
{code}
CREATE EXTERNAL TABLE if not exists
query_fed_oracle.ABCD_TEST_pw_case_jceks_diff(
YEAR INT,
QUANTITY INT,
NAME STRING
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"bucketing_version"="2",
"hive.sql.database.type" = "ORACLE",
"hive.sql.jdbc.driver" = "oracle.jdbc.OracleDriver",
"hive.sql.jdbc.url" =
"jdbc:oracle:thin:@//obfuscated.compute-1.amazonaws.com",
"hive.sql.dbcp.username" = "DI_METADATA",
"hive.sql.dbcp.password.keystore" =
"jceks://s3a@obfuscated-bucket/test.jceks",
"hive.sql.dbcp.password.key" = "oracle.secret",
"hive.sql.schema" = "CHIRAN",
"hive.sql.table" = "ABCD_TEST_1",
"hive.sql.dbcp.maxActive" = "1"
);
This can be fixed by using "hive.sql.table" = "CHIRAN.ABCD_TEST_1", but this
will break CBO as pushdown wont happen. Possible fix would be to include
schemaName check too after below call.
https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/conf/JdbcStorageConfigManager.java#L166
Attaching patch 1. Let me know if this looks good.
> JdbcStorageHandler Create table fails when hive.sql.schema is specified and
> is not the default one
> --
>
> Key: HIVE-25605
> URL: https://issues.apache.org/jira/browse/HIVE-25605
> Project: Hive
> Issue Type: Bug
> Components: JDBC storage handler
>Affects Versions: 4.0.0
>Reporter: Chiran Ravani
>Priority: Major
>
> We have observed create table statement failure for JdbcStorageHandler with
> Oracle when Schema name is specified in Table properties and that schema is
> not the default one for user.
> eg:-
> Consider Username: DI_METADATA with default schema DI_METADATA in Oracle,
> however this user has access to other schemas as well like schema name
> CHIRAN, when using below create statement in Hive it fails with error
> {code}
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
> MetaException(message:org.apache.hadoop.hive.serde2.SerDeException
>
[jira] [Work logged] (HIVE-25521) Data corruption when concatenating files with different compressions in same table/partition
[ https://issues.apache.org/jira/browse/HIVE-25521?focusedWorklogId=662899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662899 ] ASF GitHub Bot logged work on HIVE-25521: - Author: ASF GitHub Bot Created on: 08/Oct/21 18:34 Start Date: 08/Oct/21 18:34 Worklog Time Spent: 10m Work Description: yongzhi merged pull request #2639: URL: https://github.com/apache/hive/pull/2639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 662899) Time Spent: 1h 10m (was: 1h) > Data corruption when concatenating files with different compressions in same > table/partition > > > Key: HIVE-25521 > URL: https://issues.apache.org/jira/browse/HIVE-25521 > Project: Hive > Issue Type: Bug >Reporter: Harish JP >Assignee: Harish JP >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently if files of different compressions are in same directory then > concatenate can fail and cause data corruption. This happens because file can > be moved by one task as incompatible file and the other tasks will fail after > this. > > This issue is addressed in this Jira by only processing a file in one task > where offset 0 is process and ignoring the the file in all other tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25528) Avoid recalculating types after CBO on second AST pass
[ https://issues.apache.org/jira/browse/HIVE-25528?focusedWorklogId=662867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662867 ] ASF GitHub Bot logged work on HIVE-25528: - Author: ASF GitHub Bot Created on: 08/Oct/21 17:45 Start Date: 08/Oct/21 17:45 Worklog Time Spent: 10m Work Description: scarlin-cloudera closed pull request #2672: URL: https://github.com/apache/hive/pull/2672 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 662867) Time Spent: 1h 20m (was: 1h 10m) > Avoid recalculating types after CBO on second AST pass > -- > > Key: HIVE-25528 > URL: https://issues.apache.org/jira/browse/HIVE-25528 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > It should be possible to avoid recalculating and reevaluating types on the > second pass after going through CBO. CBO is making the effort to change the > types so to reassess them is a waste of time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25528) Avoid recalculating types after CBO on second AST pass
[ https://issues.apache.org/jira/browse/HIVE-25528?focusedWorklogId=662868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662868 ] ASF GitHub Bot logged work on HIVE-25528: - Author: ASF GitHub Bot Created on: 08/Oct/21 17:45 Start Date: 08/Oct/21 17:45 Worklog Time Spent: 10m Work Description: scarlin-cloudera opened a new pull request #2709: URL: https://github.com/apache/hive/pull/2709 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 662868) Time Spent: 1.5h (was: 1h 20m) > Avoid recalculating types after CBO on second AST pass > -- > > Key: HIVE-25528 > URL: https://issues.apache.org/jira/browse/HIVE-25528 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > It should be possible to avoid recalculating and reevaluating types on the > second pass after going through CBO. CBO is making the effort to change the > types so to reassess them is a waste of time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25479) Browser SSO auth may fail intermittently on chrome browser in virtual environments
[
https://issues.apache.org/jira/browse/HIVE-25479?focusedWorklogId=662866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662866
]
ASF GitHub Bot logged work on HIVE-25479:
-
Author: ASF GitHub Bot
Created on: 08/Oct/21 17:44
Start Date: 08/Oct/21 17:44
Worklog Time Spent: 10m
Work Description: vihangk1 commented on a change in pull request #2601:
URL: https://github.com/apache/hive/pull/2601#discussion_r725196715
##
File path: jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java
##
@@ -1067,33 +1069,58 @@ private void openSession() throws SQLException {
//TODO This is a bit hacky. We piggy back on a dummy OpenSession call
// to get the redirect response from the server. Instead its probably
cleaner to
// explicitly do a HTTP post request and get the response.
-int numRetry = isBrowserAuthMode() ? 2 : 1;
-for (int i=0; i Browser SSO auth may fail intermittently on chrome browser in virtual
> environments
> --
>
> Key: HIVE-25479
> URL: https://issues.apache.org/jira/browse/HIVE-25479
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When browser based SSO is enabled the Hive JDBC driver might miss the POST
> requests coming from the browser which provide the one-time token issued by
> HS2s after the SAML flow completes. The issue was observed mostly in virtual
> environments on Windows.
> The issue seems to be that when the driver binds to a port even though the
> port is in LISTEN state, if the browser issues posts request on the port
> before it goes into ACCEPT state the result is non-deterministic. On native
> OSes we observed that the connection is buffered and is received by the
> driver when it begins accepting the connections. In case of VMs it is
> observed that even though the connection is buffered and presented when the
> port goes into ACCEPT mode, the payload of the request or the connection
> itself is lost. This race condition causes the driver to wait for the browser
> until it timesout and the browser keeps waiting for a response from the
> driver.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25590) Able to create views referencing temporary tables and materialized views
[
https://issues.apache.org/jira/browse/HIVE-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426262#comment-17426262
]
Krisztian Kasa commented on HIVE-25590:
---
Pushed to master. Thanks [~kgyrtkirk], [~asolimando] for review.
> Able to create views referencing temporary tables and materialized views
>
>
> Key: HIVE-25590
> URL: https://issues.apache.org/jira/browse/HIVE-25590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Creating views/materialized views referencing temporary tables and
> materialized views are disabled in Hive. However the verification algorithm
> fails to recognize temporary tables and materialized views in subqueries.
> The verification also fails when the view definition contains joins because
> CBO transforms join branches to subqueries.
> Example1:
> {code}
> create temporary table tmp1 (c1 string, c2 string);
> create view tmp1_view as
> select subq.c1 from (select c1, c2 from tmp1) subq;
> {code}
> Example2:
> {code}
> create table t1 (a int) stored as orc tblproperties ('transactional'='true');
> create table t2 (a int) stored as orc tblproperties ('transactional'='true');
> create materialized view mv1 as
> select a from t1 where a = 10;
> create materialized view mv2 as
> select t2.a from mv1
> join t2 on (mv1.a = t2.a);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25590) Able to create views referencing temporary tables and materialized views
[
https://issues.apache.org/jira/browse/HIVE-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-25590:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Able to create views referencing temporary tables and materialized views
>
>
> Key: HIVE-25590
> URL: https://issues.apache.org/jira/browse/HIVE-25590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Creating views/materialized views referencing temporary tables and
> materialized views are disabled in Hive. However the verification algorithm
> fails to recognize temporary tables and materialized views in subqueries.
> The verification also fails when the view definition contains joins because
> CBO transforms join branches to subqueries.
> Example1:
> {code}
> create temporary table tmp1 (c1 string, c2 string);
> create view tmp1_view as
> select subq.c1 from (select c1, c2 from tmp1) subq;
> {code}
> Example2:
> {code}
> create table t1 (a int) stored as orc tblproperties ('transactional'='true');
> create table t2 (a int) stored as orc tblproperties ('transactional'='true');
> create materialized view mv1 as
> select a from t1 where a = 10;
> create materialized view mv2 as
> select t2.a from mv1
> join t2 on (mv1.a = t2.a);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25590) Able to create views referencing temporary tables and materialized views
[
https://issues.apache.org/jira/browse/HIVE-25590?focusedWorklogId=662826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662826
]
ASF GitHub Bot logged work on HIVE-25590:
-
Author: ASF GitHub Bot
Created on: 08/Oct/21 16:18
Start Date: 08/Oct/21 16:18
Worklog Time Spent: 10m
Work Description: kasakrisz merged pull request #2697:
URL: https://github.com/apache/hive/pull/2697
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 662826)
Time Spent: 1h (was: 50m)
> Able to create views referencing temporary tables and materialized views
>
>
> Key: HIVE-25590
> URL: https://issues.apache.org/jira/browse/HIVE-25590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Creating views/materialized views referencing temporary tables and
> materialized views are disabled in Hive. However the verification algorithm
> fails to recognize temporary tables and materialized views in subqueries.
> The verification also fails when the view definition contains joins because
> CBO transforms join branches to subqueries.
> Example1:
> {code}
> create temporary table tmp1 (c1 string, c2 string);
> create view tmp1_view as
> select subq.c1 from (select c1, c2 from tmp1) subq;
> {code}
> Example2:
> {code}
> create table t1 (a int) stored as orc tblproperties ('transactional'='true');
> create table t2 (a int) stored as orc tblproperties ('transactional'='true');
> create materialized view mv1 as
> select a from t1 where a = 10;
> create materialized view mv2 as
> select t2.a from mv1
> join t2 on (mv1.a = t2.a);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25505) Incorrect results with header. skip.header.line.count if first line is blank
[
https://issues.apache.org/jira/browse/HIVE-25505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated HIVE-25505:
--
Description:
aAtable with header. skip.header.line.count=1 does not skip the first line if
it is blank, except in a fetch task.
To reproduce, create a csv table, ans set header. skip.header.line.count=1 in
table properties.
In the table location, create a single file, with a blank (empty) first line,
and say 2 further lines.
If you do a select * on it, you see 2 rows (correct)
If you do select count(*) on it, you get 3 (incorrect)
{code:java}
CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.OpenCSVSerde'
LOCATION '${system:test.tmp.dir}/testcase1'
TBLPROPERTIES ("skip.header.line.count"="1");
SET hive.fetch.task.conversion = more;
select * from testcase1;
select count(*) from testcase1;
set hive.fetch.task.conversion=none;
select * from testcase1;
select count(*) from testcase1;
Test file:
1,2019-12-31
2,2019-12-31
3,2019-12-31
Should both yield (with the above test file):
A masked pattern was here
1 2019-12-31
2 2019-12-31
3 2019-12-31
3
{code}
was:
aAtable with header. skip.header.line.count=1 does not skip the first line if
it is blank, except in a fetch task.
To reproduce, create a csv table, ans set header. skip.header.line.count=1 in
table properties.
In the table location, create a single file, with a blank (empty) first line,
and say 2 further lines.
If you do a select * on it, you see 2 rows (correct)
If you do select count(\*) on it, you get 3 (incorrect)
{code:java}
// Some comments here
public String getFoo()
{
return foo;
}
{code}
> Incorrect results with header. skip.header.line.count if first line is blank
>
>
> Key: HIVE-25505
> URL: https://issues.apache.org/jira/browse/HIVE-25505
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Steve Carlin
>Assignee: Panagiotis Garefalakis
>Priority: Major
>
> aAtable with header. skip.header.line.count=1 does not skip the first line if
> it is blank, except in a fetch task.
> To reproduce, create a csv table, ans set header. skip.header.line.count=1 in
> table properties.
> In the table location, create a single file, with a blank (empty) first line,
> and say 2 further lines.
> If you do a select * on it, you see 2 rows (correct)
> If you do select count(*) on it, you get 3 (incorrect)
> {code:java}
> CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
> LOCATION '${system:test.tmp.dir}/testcase1'
> TBLPROPERTIES ("skip.header.line.count"="1");
> SET hive.fetch.task.conversion = more;
> select * from testcase1;
> select count(*) from testcase1;
> set hive.fetch.task.conversion=none;
> select * from testcase1;
> select count(*) from testcase1;
> Test file:
> 1,2019-12-31
> 2,2019-12-31
> 3,2019-12-31
> Should both yield (with the above test file):
> A masked pattern was here
> 1 2019-12-31
> 2 2019-12-31
> 3 2019-12-31
> 3
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25505) Incorrect results with header. skip.header.line.count if first line is blank
[
https://issues.apache.org/jira/browse/HIVE-25505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis reassigned HIVE-25505:
-
Assignee: Panagiotis Garefalakis
> Incorrect results with header. skip.header.line.count if first line is blank
>
>
> Key: HIVE-25505
> URL: https://issues.apache.org/jira/browse/HIVE-25505
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Steve Carlin
>Assignee: Panagiotis Garefalakis
>Priority: Major
>
> aAtable with header. skip.header.line.count=1 does not skip the first line if
> it is blank, except in a fetch task.
> To reproduce, create a csv table, ans set header. skip.header.line.count=1 in
> table properties.
> In the table location, create a single file, with a blank (empty) first line,
> and say 2 further lines.
> If you do a select * on it, you see 2 rows (correct)
> If you do select count(\*) on it, you get 3 (incorrect)
> {code:java}
> // Some comments here
> public String getFoo()
> {
> return foo;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25505) Incorrect results with header. skip.header.line.count if first line is blank
[
https://issues.apache.org/jira/browse/HIVE-25505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated HIVE-25505:
--
Description:
aAtable with header. skip.header.line.count=1 does not skip the first line if
it is blank, except in a fetch task.
To reproduce, create a csv table, ans set header. skip.header.line.count=1 in
table properties.
In the table location, create a single file, with a blank (empty) first line,
and say 2 further lines.
If you do a select * on it, you see 2 rows (correct)
If you do select count(\*) on it, you get 3 (incorrect)
{code:java}
// Some comments here
public String getFoo()
{
return foo;
}
{code}
was:
aAtable with header. skip.header.line.count=1 does not skip the first line if
it is blank, except in a fetch task.
To reproduce, create a csv table, ans set header. skip.header.line.count=1 in
table properties.
In the table location, create a single file, with a blank (empty) first line,
and say 2 further lines.
If you do a select * on it, you see 2 rows (correct)
If you do select count(\*) on it, you get 3 (incorrect)
> Incorrect results with header. skip.header.line.count if first line is blank
>
>
> Key: HIVE-25505
> URL: https://issues.apache.org/jira/browse/HIVE-25505
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Steve Carlin
>Priority: Major
>
> aAtable with header. skip.header.line.count=1 does not skip the first line if
> it is blank, except in a fetch task.
> To reproduce, create a csv table, ans set header. skip.header.line.count=1 in
> table properties.
> In the table location, create a single file, with a blank (empty) first line,
> and say 2 further lines.
> If you do a select * on it, you see 2 rows (correct)
> If you do select count(\*) on it, you get 3 (incorrect)
> {code:java}
> // Some comments here
> public String getFoo()
> {
> return foo;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25505) Incorrect results with header. skip.header.line.count if first line is blank
[ https://issues.apache.org/jira/browse/HIVE-25505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426212#comment-17426212 ] Panagiotis Garefalakis commented on HIVE-25505: --- I have a repro for this -- updating description and assigned to myself > Incorrect results with header. skip.header.line.count if first line is blank > > > Key: HIVE-25505 > URL: https://issues.apache.org/jira/browse/HIVE-25505 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Steve Carlin >Priority: Major > > aAtable with header. skip.header.line.count=1 does not skip the first line if > it is blank, except in a fetch task. > To reproduce, create a csv table, ans set header. skip.header.line.count=1 in > table properties. > In the table location, create a single file, with a blank (empty) first line, > and say 2 further lines. > If you do a select * on it, you see 2 rows (correct) > If you do select count(\*) on it, you get 3 (incorrect) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25448) Invalid partition columns when skew with distinct
[ https://issues.apache.org/jira/browse/HIVE-25448?focusedWorklogId=662753=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662753 ] ASF GitHub Bot logged work on HIVE-25448: - Author: ASF GitHub Bot Created on: 08/Oct/21 14:29 Start Date: 08/Oct/21 14:29 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2585: URL: https://github.com/apache/hive/pull/2585#issuecomment-938688401 Hey @pgaref, mind taking a look if have secs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 662753) Time Spent: 50m (was: 40m) > Invalid partition columns when skew with distinct > - > > Key: HIVE-25448 > URL: https://issues.apache.org/jira/browse/HIVE-25448 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > When hive.groupby.skewindata is enabled, we spray by the grouping key and > distinct key if distinct is present in the first reduce sink operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25479) Browser SSO auth may fail intermittently on chrome browser in virtual environments
[
https://issues.apache.org/jira/browse/HIVE-25479?focusedWorklogId=662751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662751
]
ASF GitHub Bot logged work on HIVE-25479:
-
Author: ASF GitHub Bot
Created on: 08/Oct/21 14:21
Start Date: 08/Oct/21 14:21
Worklog Time Spent: 10m
Work Description: nrg4878 commented on a change in pull request #2601:
URL: https://github.com/apache/hive/pull/2601#discussion_r724418606
##
File path: jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java
##
@@ -1067,33 +1069,58 @@ private void openSession() throws SQLException {
//TODO This is a bit hacky. We piggy back on a dummy OpenSession call
// to get the redirect response from the server. Instead its probably
cleaner to
// explicitly do a HTTP post request and get the response.
-int numRetry = isBrowserAuthMode() ? 2 : 1;
-for (int i=0; i Browser SSO auth may fail intermittently on chrome browser in virtual
> environments
> --
>
> Key: HIVE-25479
> URL: https://issues.apache.org/jira/browse/HIVE-25479
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When browser based SSO is enabled the Hive JDBC driver might miss the POST
> requests coming from the browser which provide the one-time token issued by
> HS2s after the SAML flow completes. The issue was observed mostly in virtual
> environments on Windows.
> The issue seems to be that when the driver binds to a port even though the
> port is in LISTEN state, if the browser issues posts request on the port
> before it goes into ACCEPT state the result is non-deterministic. On native
> OSes we observed that the connection is buffered and is received by the
> driver when it begins accepting the connections. In case of VMs it is
> observed that even though the connection is buffered and presented when the
> port goes into ACCEPT mode, the payload of the request or the connection
> itself is lost. This race condition causes the driver to wait for the browser
> until it timesout and the browser keeps waiting for a response from the
> driver.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25362) LLAP: ensure tasks with locality have a chance to adjust delay
[ https://issues.apache.org/jira/browse/HIVE-25362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-25362: -- Fix Version/s: 4.0.0 > LLAP: ensure tasks with locality have a chance to adjust delay > -- > > Key: HIVE-25362 > URL: https://issues.apache.org/jira/browse/HIVE-25362 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > HIVE-24914 introduced a short-circuit optimization when all nodes are busy > returning DELAYED_RESOURCES and reseting locality delay for a given tasks. > However, this may prevent tasks from adjusting their locality delay and being > added to the DelayQueue leading sometimes to missed locality chances when all > LLap resources are fully utilized. > To address the issue we should handle the two cases separately. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25362) LLAP: ensure tasks with locality have a chance to adjust delay
[ https://issues.apache.org/jira/browse/HIVE-25362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis resolved HIVE-25362. --- Resolution: Fixed > LLAP: ensure tasks with locality have a chance to adjust delay > -- > > Key: HIVE-25362 > URL: https://issues.apache.org/jira/browse/HIVE-25362 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > HIVE-24914 introduced a short-circuit optimization when all nodes are busy > returning DELAYED_RESOURCES and reseting locality delay for a given tasks. > However, this may prevent tasks from adjusting their locality delay and being > added to the DelayQueue leading sometimes to missed locality chances when all > LLap resources are fully utilized. > To address the issue we should handle the two cases separately. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25362) LLAP: ensure tasks with locality have a chance to adjust delay
[ https://issues.apache.org/jira/browse/HIVE-25362?focusedWorklogId=662732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662732 ] ASF GitHub Bot logged work on HIVE-25362: - Author: ASF GitHub Bot Created on: 08/Oct/21 13:45 Start Date: 08/Oct/21 13:45 Worklog Time Spent: 10m Work Description: pgaref merged pull request #2513: URL: https://github.com/apache/hive/pull/2513 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 662732) Time Spent: 1.5h (was: 1h 20m) > LLAP: ensure tasks with locality have a chance to adjust delay > -- > > Key: HIVE-25362 > URL: https://issues.apache.org/jira/browse/HIVE-25362 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > HIVE-24914 introduced a short-circuit optimization when all nodes are busy > returning DELAYED_RESOURCES and reseting locality delay for a given tasks. > However, this may prevent tasks from adjusting their locality delay and being > added to the DelayQueue leading sometimes to missed locality chances when all > LLap resources are fully utilized. > To address the issue we should handle the two cases separately. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25362) LLAP: ensure tasks with locality have a chance to adjust delay
[ https://issues.apache.org/jira/browse/HIVE-25362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426197#comment-17426197 ] Panagiotis Garefalakis commented on HIVE-25362: --- Resolved via https://github.com/apache/hive/pull/2513 > LLAP: ensure tasks with locality have a chance to adjust delay > -- > > Key: HIVE-25362 > URL: https://issues.apache.org/jira/browse/HIVE-25362 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > HIVE-24914 introduced a short-circuit optimization when all nodes are busy > returning DELAYED_RESOURCES and reseting locality delay for a given tasks. > However, this may prevent tasks from adjusting their locality delay and being > added to the DelayQueue leading sometimes to missed locality chances when all > LLap resources are fully utilized. > To address the issue we should handle the two cases separately. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?focusedWorklogId=662715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662715
]
ASF GitHub Bot logged work on HIVE-24797:
-
Author: ASF GitHub Bot
Created on: 08/Oct/21 13:26
Start Date: 08/Oct/21 13:26
Worklog Time Spent: 10m
Work Description: pvary merged pull request #2699:
URL: https://github.com/apache/hive/pull/2699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 662715)
Time Spent: 3h 40m (was: 3.5h)
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.9, 4.0.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field
> ARRAY_WITH_DEFAULT: null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
> at
> org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?focusedWorklogId=662676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662676
]
ASF GitHub Bot logged work on HIVE-24797:
-
Author: ASF GitHub Bot
Created on: 08/Oct/21 12:02
Start Date: 08/Oct/21 12:02
Worklog Time Spent: 10m
Work Description: findepi commented on pull request #2670:
URL: https://github.com/apache/hive/pull/2670#issuecomment-938586454
@pvary many thanks for the merge!
can you also please take a look at backport to 3.1
https://github.com/apache/hive/pull/2699? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 662676)
Time Spent: 3.5h (was: 3h 20m)
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.9, 4.0.0
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field
> ARRAY_WITH_DEFAULT: null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
> at
> org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25604) Iceberg should implement the authorization storage handler
[ https://issues.apache.org/jira/browse/HIVE-25604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426097#comment-17426097 ] Marton Bod commented on HIVE-25604: --- Pushed to master. Thanks [~pvary] for the review! > Iceberg should implement the authorization storage handler > -- > > Key: HIVE-25604 > URL: https://issues.apache.org/jira/browse/HIVE-25604 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > > Iceberg's StorageHandler should implement the HiveStorageAuthorizationHandler > interface for authorization purposes. We'll use the iceberg table root > location as the basis for permission handling. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25604) Iceberg should implement the authorization storage handler
[ https://issues.apache.org/jira/browse/HIVE-25604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod resolved HIVE-25604. --- Resolution: Fixed > Iceberg should implement the authorization storage handler > -- > > Key: HIVE-25604 > URL: https://issues.apache.org/jira/browse/HIVE-25604 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > > Iceberg's StorageHandler should implement the HiveStorageAuthorizationHandler > interface for authorization purposes. We'll use the iceberg table root > location as the basis for permission handling. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25604) Iceberg should implement the authorization storage handler
[ https://issues.apache.org/jira/browse/HIVE-25604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod reassigned HIVE-25604: - > Iceberg should implement the authorization storage handler > -- > > Key: HIVE-25604 > URL: https://issues.apache.org/jira/browse/HIVE-25604 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > > Iceberg's StorageHandler should implement the HiveStorageAuthorizationHandler > interface for authorization purposes. We'll use the iceberg table root > location as the basis for permission handling. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25540) Enable batch update of column stats only for MySql and Postgres
[ https://issues.apache.org/jira/browse/HIVE-25540?focusedWorklogId=662615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662615 ] ASF GitHub Bot logged work on HIVE-25540: - Author: ASF GitHub Bot Created on: 08/Oct/21 09:07 Start Date: 08/Oct/21 09:07 Worklog Time Spent: 10m Work Description: deniskuzZ commented on pull request #2657: URL: https://github.com/apache/hive/pull/2657#issuecomment-938477954 > I would guess that the Derby is also covered by the unit tests. > > Maybe it would be better to test with other databases unless we know there are some known issues which are hard to mitigate. The best would be to add an appropriate test to the [TestHiveMetastore.java](https://github.com/apache/hive/blob/07464b12ada011f13546b3d5d37276b76218416c/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java) > > I see a [testColumnStatistics](https://github.com/apache/hive/blob/07464b12ada011f13546b3d5d37276b76218416c/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L1674) method. This might be the one that we need, and we only have to run the tests against a cluster like described here: See: https://cwiki.apache.org/confluence/display/Hive/MetaStore+API+tests#MetaStoreAPITests-IMetaStoreClientTests > > You can use Docker images defined here to set up the db to test agains: https://github.com/apache/hive/blob/master/standalone-metastore/DEV-README > > @deniskuzZ: Is there a faster way to test out the changes against multiple databases? > > Thanks, Peter use ITestDbTxnManager: mvn test -Dtest=ITestDbTxnManager -Dtest.metastore.db=mysql -Ditest.jdbc.jars=mysql-connector-java-5.1.49.jar mvn test -Dtest=ITestDbTxnManager -Dtest.metastore.db=postgres -Ditest.jdbc.jars=postgresql-42.2.23.jar mvn test -Dtest=ITestDbTxnManager -Dtest.metastore.db=oracle -Ditest.jdbc.jars=ojdbc8.jar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 662615) Time Spent: 0.5h (was: 20m) > Enable batch update of column stats only for MySql and Postgres > > > Key: HIVE-25540 > URL: https://issues.apache.org/jira/browse/HIVE-25540 > Project: Hive > Issue Type: Sub-task >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The batch updation of partition column stats using direct sql is tested only > for MySql and Postgres. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25599) Addendum HIVE-25570 Hive should send full URL path for authorization for the command insert overwrite location
[ https://issues.apache.org/jira/browse/HIVE-25599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426052#comment-17426052 ] Panagiotis Garefalakis commented on HIVE-25599: --- Already resolved via https://github.com/apache/hive/commit/988be055289becbfc37b17264edafeca3edefbec > Addendum HIVE-25570 Hive should send full URL path for authorization for the > command insert overwrite location > -- > > Key: HIVE-25599 > URL: https://issues.apache.org/jira/browse/HIVE-25599 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25599) Addendum HIVE-25570 Hive should send full URL path for authorization for the command insert overwrite location
[ https://issues.apache.org/jira/browse/HIVE-25599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis resolved HIVE-25599. --- Resolution: Duplicate > Addendum HIVE-25570 Hive should send full URL path for authorization for the > command insert overwrite location > -- > > Key: HIVE-25599 > URL: https://issues.apache.org/jira/browse/HIVE-25599 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25603) Restore original method signature in VectorizedOrcAcidRowBatchReader.getDeleteDeltaDirsFromSplit
[ https://issues.apache.org/jira/browse/HIVE-25603?focusedWorklogId=662607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662607 ] ASF GitHub Bot logged work on HIVE-25603: - Author: ASF GitHub Bot Created on: 08/Oct/21 08:51 Start Date: 08/Oct/21 08:51 Worklog Time Spent: 10m Work Description: deniskuzZ opened a new pull request #2708: URL: https://github.com/apache/hive/pull/2708 …BatchReader.getDeleteDeltaDirsFromSplit ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 662607) Remaining Estimate: 0h Time Spent: 10m > Restore original method signature in > VectorizedOrcAcidRowBatchReader.getDeleteDeltaDirsFromSplit > > > Key: HIVE-25603 > URL: https://issues.apache.org/jira/browse/HIVE-25603 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > required for backward compatibility -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25603) Restore original method signature in VectorizedOrcAcidRowBatchReader.getDeleteDeltaDirsFromSplit
[ https://issues.apache.org/jira/browse/HIVE-25603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25603: -- Labels: pull-request-available (was: ) > Restore original method signature in > VectorizedOrcAcidRowBatchReader.getDeleteDeltaDirsFromSplit > > > Key: HIVE-25603 > URL: https://issues.apache.org/jira/browse/HIVE-25603 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > required for backward compatibility -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25603) Restore original method signature in VectorizedOrcAcidRowBatchReader.getDeleteDeltaDirsFromSplit
[ https://issues.apache.org/jira/browse/HIVE-25603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25603: -- Description: required for backward compatibility > Restore original method signature in > VectorizedOrcAcidRowBatchReader.getDeleteDeltaDirsFromSplit > > > Key: HIVE-25603 > URL: https://issues.apache.org/jira/browse/HIVE-25603 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > > required for backward compatibility -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25603) Restore original method signature in VectorizedOrcAcidRowBatchReader.getDeleteDeltaDirsFromSplit
[ https://issues.apache.org/jira/browse/HIVE-25603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reassigned HIVE-25603: - Assignee: Denys Kuzmenko > Restore original method signature in > VectorizedOrcAcidRowBatchReader.getDeleteDeltaDirsFromSplit > > > Key: HIVE-25603 > URL: https://issues.apache.org/jira/browse/HIVE-25603 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > > required for backward compatibility -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25602) Fix failover metadata file path in repl load execution.
[
https://issues.apache.org/jira/browse/HIVE-25602?focusedWorklogId=662553=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-662553
]
ASF GitHub Bot logged work on HIVE-25602:
-
Author: ASF GitHub Bot
Created on: 08/Oct/21 06:10
Start Date: 08/Oct/21 06:10
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #2707:
URL: https://github.com/apache/hive/pull/2707#discussion_r724726386
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java
##
@@ -101,7 +101,7 @@ public IncrementalLoadTasksBuilder(String dbName, String
loadPath, IncrementalLo
this.shouldFailover = shouldFailover;
if (shouldFailover) {
this.metricCollector.reportFailoverStart("REPL_LOAD", metricMap,
- new FailoverMetaData(new Path(dumpDirectory), conf));
+ new FailoverMetaData(new Path(dumpDirectory,
ReplUtils.REPL_HIVE_BASE_DIR), conf));
Review comment:
Wondering why doesn't the test fail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 662553)
Remaining Estimate: 0h
Time Spent: 10m
> Fix failover metadata file path in repl load execution.
> ---
>
> Key: HIVE-25602
> URL: https://issues.apache.org/jira/browse/HIVE-25602
> Project: Hive
> Issue Type: Bug
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When executed through scheduled queries, repl load fails with following error:
>
> {code:java}
> Reading failover metadata from file:
> 2021-10-08 02:02:51,824 ERROR org.apache.hadoop.hive.ql.Driver: [Scheduled
> Query Executor(schedule:repl_load_p1, execution_id:43)]: FAILED:
> SemanticException java.io.FileNotFoundException: File does not exist:
> /user/hive/repl/c291cmNl/36d04dfd-ee5d-4faf-bc0a-ae8d665f95f9/_failovermetadata
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:87)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:77)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:159)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2035)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:737)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:454)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25602) Fix failover metadata file path in repl load execution.
[
https://issues.apache.org/jira/browse/HIVE-25602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25602:
--
Labels: pull-request-available (was: )
> Fix failover metadata file path in repl load execution.
> ---
>
> Key: HIVE-25602
> URL: https://issues.apache.org/jira/browse/HIVE-25602
> Project: Hive
> Issue Type: Bug
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When executed through scheduled queries, repl load fails with following error:
>
> {code:java}
> Reading failover metadata from file:
> 2021-10-08 02:02:51,824 ERROR org.apache.hadoop.hive.ql.Driver: [Scheduled
> Query Executor(schedule:repl_load_p1, execution_id:43)]: FAILED:
> SemanticException java.io.FileNotFoundException: File does not exist:
> /user/hive/repl/c291cmNl/36d04dfd-ee5d-4faf-bc0a-ae8d665f95f9/_failovermetadata
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:87)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:77)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:159)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2035)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:737)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:454)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
