[jira] [Assigned] (HIVE-25545) Add/Drop constraints events on table should be authorized in HS2
[ https://issues.apache.org/jira/browse/HIVE-25545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala reassigned HIVE-25545: > Add/Drop constraints events on table should be authorized in HS2 > > > Key: HIVE-25545 > URL: https://issues.apache.org/jira/browse/HIVE-25545 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > > Alter table foo_tbl ADD constraint c1_unique UNIQUE(id1) disable novalidate; > Alter table foo_tbl DROP constraint c1_unique; > The above statements are currently not being authorized in Ranger/Sentry. > These should be authorized by creating authorizable events in Hive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25303) CTAS hive.create.as.external.legacy tries to place data files in managed WH path
[
https://issues.apache.org/jira/browse/HIVE-25303?focusedWorklogId=653839=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653839
]
ASF GitHub Bot logged work on HIVE-25303:
-
Author: ASF GitHub Bot
Created on: 21/Sep/21 21:53
Start Date: 21/Sep/21 21:53
Worklog Time Spent: 10m
Work Description: nrg4878 commented on pull request #2442:
URL: https://github.com/apache/hive/pull/2442#issuecomment-924414564
fix has been committed to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 653839)
Time Spent: 3h 20m (was: 3h 10m)
> CTAS hive.create.as.external.legacy tries to place data files in managed WH
> path
>
>
> Key: HIVE-25303
> URL: https://issues.apache.org/jira/browse/HIVE-25303
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Standalone Metastore
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Under legacy table creation mode (hive.create.as.external.legacy=true), when
> a database has been created in a specific LOCATION, in a session where that
> database is Used, tables are created using the following command:
> {code:java}
> CREATE TABLE AS SELECT {code}
> should inherit the HDFS path from the database's location. Instead, Hive is
> trying to write the table data into
> /warehouse/tablespace/managed/hive//
> +Design+:
> In the CTAS query, first data is written in the target directory (which
> happens in HS2) and then the table is created(This happens in HMS). So here
> two decisions are being made i) target directory location ii) how the table
> should be created (table type, sd e.t.c).
> When HS2 needs a target location that needs to be set, it'll make create
> table dry run call to HMS (where table translation happens) and i) and ii)
> decisions are made within HMS and returns table object. Then HS2 will use
> this location set by HMS for placing the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25517) Follow up on HIVE-24951: External Table created with Uppercase name using CTAS does not produce result for select queries
[
https://issues.apache.org/jira/browse/HIVE-25517?focusedWorklogId=653828=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653828
]
ASF GitHub Bot logged work on HIVE-25517:
-
Author: ASF GitHub Bot
Created on: 21/Sep/21 21:06
Start Date: 21/Sep/21 21:06
Worklog Time Spent: 10m
Work Description: nrg4878 commented on a change in pull request #2638:
URL: https://github.com/apache/hive/pull/2638#discussion_r713419101
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
##
@@ -355,7 +355,11 @@ public Path getDefaultTablePath(Database db, String
tableName, boolean isExterna
Path dbPath = null;
if (isExternal) {
dbPath = new Path(db.getLocationUri());
- if (FileUtils.isSubdirectory(getWhRoot().toString(), dbPath.toString() +
Path.SEPARATOR)) {
+ Path dbLocation = Path.getPathWithoutSchemeAndAuthority(dbPath);
Review comment:
This is a common API thats used by other areas of the code in Hive. I am
concerned that doing a scheme-less path comparison might cause issues.
dbPath is the location set on the Database object which has the scheme. for
ex: s3 or hdfs.
if we do a scheme-less comparison, s3://user/warehouse/hive/mydb.db/mytable
will be considered a managed location if the warehouse root is set to
"hdfs://user/warehouse/hive".
Do you see a downside with just toString comparison?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 653828)
Time Spent: 50m (was: 40m)
> Follow up on HIVE-24951: External Table created with Uppercase name using
> CTAS does not produce result for select queries
> -
>
> Key: HIVE-25517
> URL: https://issues.apache.org/jira/browse/HIVE-25517
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In [PR|https://github.com/apache/hive/pull/2125] for HIVE-24951, the
> recommendation was to use getDefaultTablePath() to set the location for an
> external table. This Jira addresses that and makes getDefaultTablePath() more
> generic.
>
> cc - [~ngangam]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25544) Remove Dependency of hive-meta-common From hive-common
[ https://issues.apache.org/jira/browse/HIVE-25544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-25544: - > Remove Dependency of hive-meta-common From hive-common > -- > > Key: HIVE-25544 > URL: https://issues.apache.org/jira/browse/HIVE-25544 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > > These two things should not be linked and it means any HS2 client libraries > pulling in hive-common library also has to pull in a ton of metastore code as > well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?focusedWorklogId=653744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653744
]
ASF GitHub Bot logged work on HIVE-25522:
-
Author: ASF GitHub Bot
Created on: 21/Sep/21 18:23
Start Date: 21/Sep/21 18:23
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #2647:
URL: https://github.com/apache/hive/pull/2647#discussion_r713306606
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -433,6 +433,7 @@ public void init() throws MetaException {
addAdminUsers();
currentUrl = MetaStoreInit.getConnectionURL(conf);
}
+ TxnUtils.initializeTxnStore(conf); // Must be before Metric services
start
Review comment:
Oh I see. Makes sense.
Strange that I don't see the latest test results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 653744)
Time Spent: 3h 50m (was: 3h 40m)
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
[jira] [Assigned] (HIVE-25543) Add Read-Only Capability to ObjectStore
[ https://issues.apache.org/jira/browse/HIVE-25543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-25543: - > Add Read-Only Capability to ObjectStore > --- > > Key: HIVE-25543 > URL: https://issues.apache.org/jira/browse/HIVE-25543 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > > Recently saw some stack-traces that shows that calling "commit" triggers > quite a bit of work within DataNucleus, as I understand it, to look for > changes in the transaction and to commit those changes. > Given that many of the RPCs within the Metastore are look-ups, Hive can avoid > all these needless work by making transaction read-only (rollbackOnly). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25532) Missing authorization info for KILL QUERY command
[ https://issues.apache.org/jira/browse/HIVE-25532?focusedWorklogId=653702=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653702 ] ASF GitHub Bot logged work on HIVE-25532: - Author: ASF GitHub Bot Created on: 21/Sep/21 17:07 Start Date: 21/Sep/21 17:07 Worklog Time Spent: 10m Work Description: achennagiri commented on pull request #2649: URL: https://github.com/apache/hive/pull/2649#issuecomment-924184200 Recheck -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 653702) Time Spent: 50m (was: 40m) > Missing authorization info for KILL QUERY command > - > > Key: HIVE-25532 > URL: https://issues.apache.org/jira/browse/HIVE-25532 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Abhay >Assignee: Abhay >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > We added authorization for Kill Query command some time back with the help of > Ranger. Below is the ticket https://issues.apache.org/jira/browse/RANGER-1851 > However, we have observed that this hasn't been working as expected. The > Ranger service expects Hive to send in a privilege object of the type > SERVICE_NAME but we can see below > > [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/server/KillQueryImpl.java#L131] > that it is sending an empty array list. > The Ranger service never throws an exception to this and this results in any > user being able to kill any query even though they don't have necessary > permissions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25542) Remove References to Index Configurations
[ https://issues.apache.org/jira/browse/HIVE-25542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-25542: -- Summary: Remove References to Index Configurations (was: Remove References to hive.optimize.index.filter) > Remove References to Index Configurations > - > > Key: HIVE-25542 > URL: https://issues.apache.org/jira/browse/HIVE-25542 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Priority: Minor > Labels: newbie, noob > > Hive indexes were removed from 4.x series. > Please remove all references to the Index configurations > For example: hive.optimize.index.filter > Also update the docs: > https://cwiki.apache.org/confluence/display/hive/configuration+properties -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25500) Switch back to alter_partition(s) in HMS client for Hive 2.3.x
[
https://issues.apache.org/jira/browse/HIVE-25500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-25500:
Fix Version/s: (was: 2.3.9)
2.3.10
> Switch back to alter_partition(s) in HMS client for Hive 2.3.x
> --
>
> Key: HIVE-25500
> URL: https://issues.apache.org/jira/browse/HIVE-25500
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 2.3.9
>Reporter: dzcxzl
>Assignee: dzcxzl
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.3.10
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> HIVE-12730 uses alter_partition_with_environment_context and
> alter_partitions_with_environment_context instead of alter_partition and
> alter_partitions when the EnvironmentContext is null.
> This causes the client version to be greater than 2.1 to connect to
> server<2.1, and this error will occur.
> {code:java}
> Caused by: org.apache.thrift.TApplicationException: Invalid method name:
> 'alter_partitions_with_environment_context'
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_partitions_with_environment_context(ThriftHiveMetastore.java:2843)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_partitions_with_environment_context(ThriftHiveMetastore.java:2827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1524)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25538) CommitTxn replay failing during incremental run
[
https://issues.apache.org/jira/browse/HIVE-25538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418161#comment-17418161
]
Ayush Saxena commented on HIVE-25538:
-
Hey [~zabetak]
I have updated the Jira with the details. Thanx
> CommitTxn replay failing during incremental run
> ---
>
> Key: HIVE-25538
> URL: https://issues.apache.org/jira/browse/HIVE-25538
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Critical
>
> CommitTxn Fails during incremental run, in case the source file is deleted
> post copy & before checksum validation.
> {noformat}
> 2021-09-21T07:53:40,898 ERROR [TThreadPoolServer WorkerProcess-%d]
> thrift.ProcessFunction: Internal error processing commit_txn
> org.apache.thrift.TException:
> /warehouse1/replicated_testreplcommittransactiononsourcedelete_1632235978675.db/testreplcommittransactiononsourcedelete/load_date=2016-03-01/delta_002_002_
> (is not a directory)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePath(FSDirectory.java:677)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:151)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1927)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:738)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:424)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at
> org.apache.hadoop.hive.metastore.HMSHandler.commit_txn(HMSHandler.java:8652)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_261]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy55.commit_txn(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$commit_txn.getResult(ThriftHiveMetastore.java:23159)
> ~[hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$commit_txn.getResult(ThriftHiveMetastore.java:23138)
> ~[hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_261]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_261]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_261]
> at
>
[jira] [Updated] (HIVE-25538) CommitTxn replay failing during incremental run
[
https://issues.apache.org/jira/browse/HIVE-25538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-25538:
Description:
CommitTxn Fails during incremental run, in case the source file is deleted post
copy & before checksum validation.
{noformat}
2021-09-21T07:53:40,898 ERROR [TThreadPoolServer WorkerProcess-%d]
thrift.ProcessFunction: Internal error processing commit_txn
org.apache.thrift.TException:
/warehouse1/replicated_testreplcommittransactiononsourcedelete_1632235978675.db/testreplcommittransactiononsourcedelete/load_date=2016-03-01/delta_002_002_
(is not a directory)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePath(FSDirectory.java:677)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:151)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1927)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:738)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:424)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
at
org.apache.hadoop.hive.metastore.HMSHandler.commit_txn(HMSHandler.java:8652)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_261]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy55.commit_txn(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$commit_txn.getResult(ThriftHiveMetastore.java:23159)
~[hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$commit_txn.getResult(ThriftHiveMetastore.java:23138)
~[hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_261]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_261]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
[hadoop-common-3.1.0.jar:?]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_261]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_261]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]
{noformat}
was:CommitTxn Fails during incremental run, in case the source file is
deleted post copy & before checksum validation.
> CommitTxn replay failing during incremental run
> ---
>
> Key: HIVE-25538
> URL: https://issues.apache.org/jira/browse/HIVE-25538
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Ayush Saxena
>
[jira] [Updated] (HIVE-25538) CommitTxn replay failing during incremental run
[ https://issues.apache.org/jira/browse/HIVE-25538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-25538: Affects Version/s: 4.0.0 > CommitTxn replay failing during incremental run > --- > > Key: HIVE-25538 > URL: https://issues.apache.org/jira/browse/HIVE-25538 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Critical > > CommitTxn Fails during incremental run, in case the source file is deleted > post copy & before checksum validation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25538) CommitTxn replay failing during incremental run
[
https://issues.apache.org/jira/browse/HIVE-25538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418138#comment-17418138
]
Stamatis Zampetakis commented on HIVE-25538:
Thanks for logging this [~ayushtkn]. Can you please fill in the "Affects
Version" field and also include the stack trace (inside \\{noformat\} tags)?
> CommitTxn replay failing during incremental run
> ---
>
> Key: HIVE-25538
> URL: https://issues.apache.org/jira/browse/HIVE-25538
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Critical
>
> CommitTxn Fails during incremental run, in case the source file is deleted
> post copy & before checksum validation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN
[
https://issues.apache.org/jira/browse/HIVE-25011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418133#comment-17418133
]
Denys Kuzmenko commented on HIVE-25011:
---
hi [~gopalv], sorry, I missed your reply. I have created PR with your fix +
added the test. Please check:
https://github.com/apache/hive/pull/2660
> Concurrency: Do not acquire locks for EXPLAIN
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN
[
https://issues.apache.org/jira/browse/HIVE-25011?focusedWorklogId=653565=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653565
]
ASF GitHub Bot logged work on HIVE-25011:
-
Author: ASF GitHub Bot
Created on: 21/Sep/21 13:51
Start Date: 21/Sep/21 13:51
Worklog Time Spent: 10m
Work Description: deniskuzZ opened a new pull request #2660:
URL: https://github.com/apache/hive/pull/2660
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 653565)
Remaining Estimate: 0h
Time Spent: 10m
> Concurrency: Do not acquire locks for EXPLAIN
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN
[
https://issues.apache.org/jira/browse/HIVE-25011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25011:
--
Labels: pull-request-available (was: )
> Concurrency: Do not acquire locks for EXPLAIN
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25512) Merge statement does not enforce check constraints
[
https://issues.apache.org/jira/browse/HIVE-25512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa resolved HIVE-25512.
---
Resolution: Fixed
Pushed to master. Thanks [~kgyrtkirk] for review.
> Merge statement does not enforce check constraints
> --
>
> Key: HIVE-25512
> URL: https://issues.apache.org/jira/browse/HIVE-25512
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> CREATE TABLE table_check_merge(
> name string CHECK (length(name)<=20),
> age int,
> gpa double CHECK (gpa BETWEEN 0.0 AND 4.0)
> ) stored as orc TBLPROPERTIES ('transactional'='true');
> CREATE TABLE table_source( name string, age int, gpa double);
> insert into table_source(name, age, gpa) values ('student1', 16, null),
> (null, 20, 4.0);
> insert into table_check_merge(name, age, gpa) values ('student1', 16, 2.0);
> merge into table_check_merge using (select age from table_source)source
> on source.age=table_check_merge.age
> when matched then update set gpa=6;
> {code}
> Merge statement tries to update gpa to 6 which is not between 0.0 and 4.0.
> However the update succeeds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25485) Transform selects of literals under a UNION ALL to inline table scan
[
https://issues.apache.org/jira/browse/HIVE-25485?focusedWorklogId=653483=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653483
]
ASF GitHub Bot logged work on HIVE-25485:
-
Author: ASF GitHub Bot
Created on: 21/Sep/21 10:52
Start Date: 21/Sep/21 10:52
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #2608:
URL: https://github.com/apache/hive/pull/2608#issuecomment-923865132
@kasakrisz could you please take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 653483)
Time Spent: 1h 20m (was: 1h 10m)
> Transform selects of literals under a UNION ALL to inline table scan
>
>
> Key: HIVE-25485
> URL: https://issues.apache.org/jira/browse/HIVE-25485
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> {code}
> select 1
> union all
> select 1
> union all
> [...]
> union all
> select 1
> {code}
> results in a very big plan; which will have vertexes proportional to the
> number of union all branch - hence it could be slow to execute it
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25485) Transform selects of literals under a UNION ALL to inline table scan
[
https://issues.apache.org/jira/browse/HIVE-25485?focusedWorklogId=653481=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653481
]
ASF GitHub Bot logged work on HIVE-25485:
-
Author: ASF GitHub Bot
Created on: 21/Sep/21 10:52
Start Date: 21/Sep/21 10:52
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #2608:
URL: https://github.com/apache/hive/pull/2608#discussion_r712923238
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveTransformSimpleSelectsToInlineTableInUnion.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException;
+import org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable;
+import org.apache.hadoop.hive.ql.optimizer.calcite.TraitsUtil;
+import
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableFunctionScan;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveUnion;
+import
org.apache.hadoop.hive.ql.optimizer.calcite.translator.SqlFunctionConverter;
+import org.apache.hadoop.hive.ql.parse.SemanticAnalyzer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.ImmutableList;
+
+/**
+ * Transforms SELECTS of literals under UNION ALL into inline table scans.
+ */
Review comment:
added some apidoc/etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 653481)
Time Spent: 1h 10m (was: 1h)
> Transform selects of literals under a UNION ALL to inline table scan
>
>
> Key: HIVE-25485
> URL: https://issues.apache.org/jira/browse/HIVE-25485
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> {code}
> select 1
> union all
> select 1
> union all
> [...]
> union all
> select 1
> {code}
> results in a very big plan; which will have vertexes proportional to the
> number of union all branch - hence it could be slow to execute it
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25536) Upgrade to Kafka 2.8
[ https://issues.apache.org/jira/browse/HIVE-25536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass updated HIVE-25536: --- Status: Patch Available (was: In Progress) > Upgrade to Kafka 2.8 > > > Key: HIVE-25536 > URL: https://issues.apache.org/jira/browse/HIVE-25536 > Project: Hive > Issue Type: Improvement > Components: kafka integration >Reporter: Viktor Somogyi-Vass >Assignee: Viktor Somogyi-Vass >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25536) Upgrade to Kafka 2.8
[ https://issues.apache.org/jira/browse/HIVE-25536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25536: -- Labels: pull-request-available (was: ) > Upgrade to Kafka 2.8 > > > Key: HIVE-25536 > URL: https://issues.apache.org/jira/browse/HIVE-25536 > Project: Hive > Issue Type: Improvement > Components: kafka integration >Reporter: Viktor Somogyi-Vass >Assignee: Viktor Somogyi-Vass >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25536) Upgrade to Kafka 2.8
[ https://issues.apache.org/jira/browse/HIVE-25536?focusedWorklogId=653467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653467 ] ASF GitHub Bot logged work on HIVE-25536: - Author: ASF GitHub Bot Created on: 21/Sep/21 10:28 Start Date: 21/Sep/21 10:28 Worklog Time Spent: 10m Work Description: viktorsomogyi opened a new pull request #2659: URL: https://github.com/apache/hive/pull/2659 This change upgrades the Kafka dependency in Hive to 2.8 (from 2.5). It also fixes some additional minor compilation issues that comes with upgrading the Kafka version. No user facing change is being introduced. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 653467) Remaining Estimate: 0h Time Spent: 10m > Upgrade to Kafka 2.8 > > > Key: HIVE-25536 > URL: https://issues.apache.org/jira/browse/HIVE-25536 > Project: Hive > Issue Type: Improvement > Components: kafka integration >Reporter: Viktor Somogyi-Vass >Assignee: Viktor Somogyi-Vass >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-19647) use bitvectors in IN operators
[ https://issues.apache.org/jira/browse/HIVE-19647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19647: Fix Version/s: 4.0.0 Assignee: Soumyakanti Das (was: Zoltan Haindrich) Resolution: Fixed Status: Resolved (was: Patch Available) merged into master. Thank you [~soumyakanti.das] for finishing this patch! > use bitvectors in IN operators > -- > > Key: HIVE-19647 > URL: https://issues.apache.org/jira/browse/HIVE-19647 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Soumyakanti Das >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19647.01.patch, HIVE-19647.02.patch, > HIVE-19647.03.patch, partial.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > for strings there is no easy way to filter things out ; even min/max is null; > but the bitvectors could be used if they are available -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-19647) use bitvectors in IN operators
[ https://issues.apache.org/jira/browse/HIVE-19647?focusedWorklogId=653464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653464 ] ASF GitHub Bot logged work on HIVE-19647: - Author: ASF GitHub Bot Created on: 21/Sep/21 10:23 Start Date: 21/Sep/21 10:23 Worklog Time Spent: 10m Work Description: kgyrtkirk merged pull request #2598: URL: https://github.com/apache/hive/pull/2598 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 653464) Time Spent: 0.5h (was: 20m) > use bitvectors in IN operators > -- > > Key: HIVE-19647 > URL: https://issues.apache.org/jira/browse/HIVE-19647 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-19647.01.patch, HIVE-19647.02.patch, > HIVE-19647.03.patch, partial.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > for strings there is no easy way to filter things out ; even min/max is null; > but the bitvectors could be used if they are available -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24776) Reduce HMS DB calls during stats updates
[
https://issues.apache.org/jira/browse/HIVE-24776?focusedWorklogId=653441=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653441
]
ASF GitHub Bot logged work on HIVE-24776:
-
Author: ASF GitHub Bot
Created on: 21/Sep/21 09:40
Start Date: 21/Sep/21 09:40
Worklog Time Spent: 10m
Work Description: HarshitGupta11 commented on a change in pull request
#2636:
URL: https://github.com/apache/hive/pull/2636#discussion_r712872960
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -2197,18 +2199,26 @@ private void
updatePartitionColumnStatisticsInCache(ColumnStatistics colStats, M
sharedCache.updatePartitionColStatsInCache(catName, dbName, tblName,
partVals, colStats.getStatsObj());
}
- @Override public Map
updatePartitionColumnStatistics(ColumnStatistics colStats, List
partVals,
+ @Override
+ public Map updatePartitionColumnStatistics(Table table,
+ ColumnStatistics colStats, List partVals,
String validWriteIds, long writeId)
throws NoSuchObjectException, MetaException, InvalidObjectException,
InvalidInputException {
Map newParams =
-rawStore.updatePartitionColumnStatistics(colStats, partVals,
validWriteIds, writeId);
+rawStore.updatePartitionColumnStatistics(table, colStats, partVals,
validWriteIds, writeId);
// in case of event based cache update, cache is updated during commit txn
if (newParams != null && !canUseEvents) {
updatePartitionColumnStatisticsInCache(colStats, newParams, partVals);
}
return newParams;
}
+ @Override public Map
updatePartitionColumnStatistics(ColumnStatistics statsObj, List
partVals,
+ String validWriteIds, long writeId)
+ throws NoSuchObjectException, MetaException, InvalidObjectException,
InvalidInputException {
+return updatePartitionColumnStatistics(null, statsObj, partVals,
validWriteIds, writeId);
Review comment:
There were some unit tests that were breaking that used the function
signature without the table. So, I added the additional function in the
interface and routed the original one back to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 653441)
Time Spent: 20m (was: 10m)
> Reduce HMS DB calls during stats updates
>
>
> Key: HIVE-24776
> URL: https://issues.apache.org/jira/browse/HIVE-24776
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When adding large number of partitions (100s/1000s) in a table, it ends up
> making lots of getTable calls which are not needed.
> Lines mentioned below may vary slightly in apache-master.
> {noformat}
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:620)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:637)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:646)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2112)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2150)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.ensureGetMTable(ObjectStore.java:4578)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.ensureGetTable(ObjectStore.java:4588)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:9264)
> at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy27.updatePartitionColumnStatistics(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartitonColStatsInternal(HiveMetaStore.java:6679)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:8655)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:8592)
> at
[jira] [Commented] (HIVE-23016) Extract JdbcConnectionParams from Utils Class
[ https://issues.apache.org/jira/browse/HIVE-23016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17417964#comment-17417964 ] Timur Malikin commented on HIVE-23016: -- [~belugabehr] Could you take a look at my PR? > Extract JdbcConnectionParams from Utils Class > - > > Key: HIVE-23016 > URL: https://issues.apache.org/jira/browse/HIVE-23016 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Priority: Minor > Labels: n00b, newbie, noob, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > And make it its own class. > https://github.com/apache/hive/blob/4700e210ef7945278c4eb313c9ebd810b0224da1/jdbc/src/java/org/apache/hive/jdbc/Utils.java#L72 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24776) Reduce HMS DB calls during stats updates
[
https://issues.apache.org/jira/browse/HIVE-24776?focusedWorklogId=653408=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653408
]
ASF GitHub Bot logged work on HIVE-24776:
-
Author: ASF GitHub Bot
Created on: 21/Sep/21 07:42
Start Date: 21/Sep/21 07:42
Worklog Time Spent: 10m
Work Description: maheshk114 commented on a change in pull request #2636:
URL: https://github.com/apache/hive/pull/2636#discussion_r712748333
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -9700,8 +9705,7 @@ private void writeMPartitionColumnStatistics(Table table,
Partition partition,
Map oldStats =
getPartitionColStats(table, statsDesc
.getPartName(), colNames, colStats.getEngine());
- MPartition mPartition = getMPartition(
- catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals,
mTable);
+ //MPartition mPartition = convertToMPart(partition,mTable,false);
if (partition == null) {
Review comment:
this check can be moved up.
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -9687,10 +9688,14 @@ private void writeMPartitionColumnStatistics(Table
table, Partition partition,
List statsObjs = colStats.getStatsObj();
ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() :
getDefaultCatalog(conf);
- MTable mTable = ensureGetMTable(catName, statsDesc.getDbName(),
statsDesc.getTableName());
- Table table = convertToTable(mTable);
- Partition partition = convertToPart(getMPartition(
- catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals,
mTable), false);
+ if(table == null) {
+MTable mTable = ensureGetMTable(catName, statsDesc.getDbName(),
statsDesc.getTableName());
+table = convertToTable(mTable);
+ }
+ //MTable mTable = ensureGetMTable(catName, statsDesc.getDbName(),
statsDesc.getTableName());
+ MTable mTable = convertToMTable(table);
Review comment:
convertToMTable is redundant incase table is null
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -75,6 +75,8 @@
import com.google.common.annotations.VisibleForTesting;
+import javax.servlet.http.Part;
Review comment:
why this import is required ?
##
File path:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
##
@@ -691,9 +691,9 @@ private void createPartitionedTable(boolean withPrivileges,
boolean withStatisti
ColumnStatisticsObj partStats = new
ColumnStatisticsObj("test_part_col", "int", data);
statsObjList.add(partStats);
-
+//Table tbl =
objectStore.getTable(tbl1.getCatName(),tbl1.getDbName(),tbl1.getTableName());
Review comment:
remove this line
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -2197,18 +2199,26 @@ private void
updatePartitionColumnStatisticsInCache(ColumnStatistics colStats, M
sharedCache.updatePartitionColStatsInCache(catName, dbName, tblName,
partVals, colStats.getStatsObj());
}
- @Override public Map
updatePartitionColumnStatistics(ColumnStatistics colStats, List
partVals,
+ @Override
+ public Map updatePartitionColumnStatistics(Table table,
+ ColumnStatistics colStats, List partVals,
String validWriteIds, long writeId)
throws NoSuchObjectException, MetaException, InvalidObjectException,
InvalidInputException {
Map newParams =
-rawStore.updatePartitionColumnStatistics(colStats, partVals,
validWriteIds, writeId);
+rawStore.updatePartitionColumnStatistics(table, colStats, partVals,
validWriteIds, writeId);
// in case of event based cache update, cache is updated during commit txn
if (newParams != null && !canUseEvents) {
updatePartitionColumnStatisticsInCache(colStats, newParams, partVals);
}
return newParams;
}
+ @Override public Map
updatePartitionColumnStatistics(ColumnStatistics statsObj, List
partVals,
+ String validWriteIds, long writeId)
+ throws NoSuchObjectException, MetaException, InvalidObjectException,
InvalidInputException {
+return updatePartitionColumnStatistics(null, statsObj, partVals,
validWriteIds, writeId);
Review comment:
why not call the rawstore API without table ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[jira] [Updated] (HIVE-24776) Reduce HMS DB calls during stats updates
[
https://issues.apache.org/jira/browse/HIVE-24776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24776:
--
Labels: pull-request-available (was: )
> Reduce HMS DB calls during stats updates
>
>
> Key: HIVE-24776
> URL: https://issues.apache.org/jira/browse/HIVE-24776
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When adding large number of partitions (100s/1000s) in a table, it ends up
> making lots of getTable calls which are not needed.
> Lines mentioned below may vary slightly in apache-master.
> {noformat}
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:620)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:637)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:646)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2112)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2150)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.ensureGetMTable(ObjectStore.java:4578)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.ensureGetTable(ObjectStore.java:4588)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:9264)
> at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy27.updatePartitionColumnStatistics(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartitonColStatsInternal(HiveMetaStore.java:6679)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:8655)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:8592)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.set_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:19060)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:19044)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25534) Error when executing DistCp on file system not supporting XAttrs
[
https://issues.apache.org/jira/browse/HIVE-25534?focusedWorklogId=653395=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-653395
]
ASF GitHub Bot logged work on HIVE-25534:
-
Author: ASF GitHub Bot
Created on: 21/Sep/21 07:08
Start Date: 21/Sep/21 07:08
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #2650:
URL: https://github.com/apache/hive/pull/2650#discussion_r712732512
##
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##
@@ -277,6 +277,16 @@ private boolean isFairScheduler (Configuration conf) {
equalsIgnoreCase(conf.get(YarnConfiguration.RM_SCHEDULER));
}
+ private boolean checkFileSystemXAttrSupport(FileSystem fs) throws
IOException {
+try {
+ fs.getXAttrs(new Path(Path.SEPARATOR));
+} catch (UnsupportedOperationException e) {
+ LOG.warn("XAttr won't be preserved since it is not supported for file
system: " + fs.getUri());
+ return false;
+}
+return true;
Review comment:
nit: You can move this to line number 283
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 653395)
Time Spent: 1.5h (was: 1h 20m)
> Error when executing DistCp on file system not supporting XAttrs
>
>
> Key: HIVE-25534
> URL: https://issues.apache.org/jira/browse/HIVE-25534
> Project: Hive
> Issue Type: Bug
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Remove the preserve xattr while calling distcp.
> {code:java}
> 2021-08-23 10:06:18,485 ERROR org.apache.hadoop.tools.DistCp:
> [HiveServer2-Background-Pool: Thread-73]: XAttrs not supported on at least
> one file system:
> org.apache.hadoop.tools.CopyListing$XAttrsNotSupportedException: XAttrs not
> supported for file system: s3a://hmangla1-dev
> at
> org.apache.hadoop.tools.util.DistCpUtils.checkFileSystemXAttrSupport(DistCpUtils.java:513)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:337)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:304)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:214)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
