[jira] [Created] (HIVE-28101) [Athena] Add connector for Amazon Athena

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-28101:


 Summary: [Athena] Add connector for Amazon Athena
 Key: HIVE-28101
 URL: https://issues.apache.org/jira/browse/HIVE-28101
 Project: Hive
  Issue Type: Sub-task
  Components: Standalone Metastore
Reporter: Naveen Gangam


Recent added a HIVEJDBC connector for Hive to Hive over JDBC. This seems to 
also work for Hive to EMR with a local catalog. Does not seem to work with EMR 
backed with a AWS Glue Catalog. 

Just filing this jira to assess the need for a connector implementation for 
Amazon Athena with Glue Catalog.

[~zhangbutao] What do you think? I do not have access to a test bed for Athena.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27952) Hive fails to create SslContextFactory when KeyStore has multiple certificates

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27952.
--
Resolution: Fixed

> Hive fails to create SslContextFactory when KeyStore has multiple certificates
> --
>
> Key: HIVE-27952
> URL: https://issues.apache.org/jira/browse/HIVE-27952
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0-beta-1
>Reporter: Seonggon Namgung
>Assignee: Seonggon Namgung
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>
> With Jetty 9.4.40, we should call SslContextFactory.Server(), instead of 
> SslContextFactory(), to create SslContextFactory. Otherwise we get the 
> following error when using a KeyStore with multiple certificates in it.
> {code:java}
> Caused by: java.lang.IllegalStateException: KeyStores with multiple 
> certificates are not supported on the base class 
> org.eclipse.jetty.util.ssl.SslContextFactory. (Use 
> org.eclipse.jetty.util.ssl.SslContextFactory$Server or 
> org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27952) Hive fails to create SslContextFactory when KeyStore has multiple certificates

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-27952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798295#comment-17798295
 ] 

Naveen Gangam commented on HIVE-27952:
--

Jetty's SslContextFactory has been split for Client/Server classes via 
https://github.com/jetty/jetty.project/pull/3480.

Fix has been merged to master. Thank you for the patch [~seonggon]

> Hive fails to create SslContextFactory when KeyStore has multiple certificates
> --
>
> Key: HIVE-27952
> URL: https://issues.apache.org/jira/browse/HIVE-27952
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0-beta-1
>Reporter: Seonggon Namgung
>Assignee: Seonggon Namgung
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>
> With Jetty 9.4.40, we should call SslContextFactory.Server(), instead of 
> SslContextFactory(), to create SslContextFactory. Otherwise we get the 
> following error when using a KeyStore with multiple certificates in it.
> {code:java}
> Caused by: java.lang.IllegalStateException: KeyStores with multiple 
> certificates are not supported on the base class 
> org.eclipse.jetty.util.ssl.SslContextFactory. (Use 
> org.eclipse.jetty.util.ssl.SslContextFactory$Server or 
> org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27749) SchemaTool initSchema fails on Mariadb 10.2

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-27749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796874#comment-17796874
 ] 

Naveen Gangam commented on HIVE-27749:
--

Thats might be fine. I do not expect that this table will have more than 1 row. 
In the prior version of the script, we did not have such constraints and we 
were ok. Also I do not think the hive code cares for additional rows where 
NNI_ID != 1.
So go for it.

> SchemaTool initSchema fails on Mariadb 10.2
> ---
>
> Key: HIVE-27749
> URL: https://issues.apache.org/jira/browse/HIVE-27749
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
>Affects Versions: 4.0.0-beta-1
>Reporter: Stamatis Zampetakis
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: mariadb-metastore-schema-tests.patch
>
>
> Schema initialization for 4.0.0-beta-1 fails when run on Mariadb 10.2.
> The problem is reproducible on current 
> (e5a7ce2f091da1f8a324da6e489cda59b9e4bfc6) master by applying the 
> [^mariadb-metastore-schema-tests.patch] and then running:
> {noformat}
> mvn test -Dtest=TestMariadb#install -Dtest.groups=""{noformat}
> The error is shown below:
> {noformat}
> 315/409  ALTER TABLE `NOTIFICATION_SEQUENCE` MODIFY COLUMN `NNI_ID` 
> BIGINT(20) GENERATED ALWAYS AS (1) STORED NOT NULL;
> Error: (conn=11) You have an error in your SQL syntax; check the manual that 
> corresponds to your MariaDB server version for the right syntax to use near 
> 'NOT NULL' at line 1 (state=42000,code=1064)
> Aborting command set because "force" is false and command failed: "ALTER 
> TABLE `NOTIFICATION_SEQUENCE` MODIFY COLUMN `NNI_ID` BIGINT(20) GENERATED 
> ALWAYS AS (1) STORED NOT NULL;"
> [ERROR] 2023-09-27 21:36:30.317 [main] MetastoreSchemaTool - Schema 
> initialization FAILED! Metastore state would be inconsistent!
> Schema initialization FAILED! Metastore state would be inconsistent!
> [ERROR] 2023-09-27 21:36:30.317 [main] MetastoreSchemaTool - Underlying 
> cause: java.io.IOException : Schema script failed, errorcode OTHER
> Underlying cause: java.io.IOException : Schema script failed, errorcode OTHER
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization 
> FAILED! Metastore state would be inconsistent!
> at 
> org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskInit.execute(SchemaToolTaskInit.java:66)
> at 
> org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:480)
> at 
> org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:425)
> at 
> org.apache.hadoop.hive.metastore.dbinstall.rules.DatabaseRule.installLatest(DatabaseRule.java:269)
> at 
> org.apache.hadoop.hive.metastore.dbinstall.DbInstallBase.install(DbInstallBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at 
> 

[jira] [Commented] (HIVE-27749) SchemaTool initSchema fails on Mariadb 10.2

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-27749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796487#comment-17796487
 ] 

Naveen Gangam commented on HIVE-27749:
--

[~sbadhya] Thank you for the research.

Does "ONE_ROW_CONSTRAINT" work for both MariaDB and MySQL ? Not sure how old 
MariaDB 10.2 is but mildly concerned that it may not work against all versions. 
Have you considered going back to what it was prior to this change? It seems 
like thats what we do for other DBs supported by HMS. 

> SchemaTool initSchema fails on Mariadb 10.2
> ---
>
> Key: HIVE-27749
> URL: https://issues.apache.org/jira/browse/HIVE-27749
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
>Affects Versions: 4.0.0-beta-1
>Reporter: Stamatis Zampetakis
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: mariadb-metastore-schema-tests.patch
>
>
> Schema initialization for 4.0.0-beta-1 fails when run on Mariadb 10.2.
> The problem is reproducible on current 
> (e5a7ce2f091da1f8a324da6e489cda59b9e4bfc6) master by applying the 
> [^mariadb-metastore-schema-tests.patch] and then running:
> {noformat}
> mvn test -Dtest=TestMariadb#install -Dtest.groups=""{noformat}
> The error is shown below:
> {noformat}
> 315/409  ALTER TABLE `NOTIFICATION_SEQUENCE` MODIFY COLUMN `NNI_ID` 
> BIGINT(20) GENERATED ALWAYS AS (1) STORED NOT NULL;
> Error: (conn=11) You have an error in your SQL syntax; check the manual that 
> corresponds to your MariaDB server version for the right syntax to use near 
> 'NOT NULL' at line 1 (state=42000,code=1064)
> Aborting command set because "force" is false and command failed: "ALTER 
> TABLE `NOTIFICATION_SEQUENCE` MODIFY COLUMN `NNI_ID` BIGINT(20) GENERATED 
> ALWAYS AS (1) STORED NOT NULL;"
> [ERROR] 2023-09-27 21:36:30.317 [main] MetastoreSchemaTool - Schema 
> initialization FAILED! Metastore state would be inconsistent!
> Schema initialization FAILED! Metastore state would be inconsistent!
> [ERROR] 2023-09-27 21:36:30.317 [main] MetastoreSchemaTool - Underlying 
> cause: java.io.IOException : Schema script failed, errorcode OTHER
> Underlying cause: java.io.IOException : Schema script failed, errorcode OTHER
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization 
> FAILED! Metastore state would be inconsistent!
> at 
> org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskInit.execute(SchemaToolTaskInit.java:66)
> at 
> org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:480)
> at 
> org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:425)
> at 
> org.apache.hadoop.hive.metastore.dbinstall.rules.DatabaseRule.installLatest(DatabaseRule.java:269)
> at 
> org.apache.hadoop.hive.metastore.dbinstall.DbInstallBase.install(DbInstallBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>  

[jira] [Commented] (HIVE-27749) SchemaTool initSchema fails on Mariadb 10.2

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-27749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793877#comment-17793877
 ] 

Naveen Gangam commented on HIVE-27749:
--

[~sbadhya] This seems to be broken since alpha-2 due to 
https://github.com/apache/hive/pull/3369

Is there a solution that works for both MySQL and MariaDB.
ALTER TABLE `NOTIFICATION_SEQUENCE` MODIFY COLUMN `NNI_ID` BIGINT(20) GENERATED 
ALWAYS AS (1) STORED NOT NULL;

According to MariaDB docs, it should work. 
https://mariadb.com/kb/en/generated-columns/#examples


I tried removing the NOT NULL keyword based on the error message but MariaDb 
does not seem to support generated columns for PK.
{noformat}
ALTER TABLE `NOTIFICATION_SEQUENCE` MODIFY COLUMN `NNI_ID` BIGINT(20) GENERATED 
ALWAYS AS (1) STORED NOT NULL;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that 
corresponds to your MariaDB server version for the right syntax to use near 
'NOT NULL' at line 1
MariaDB [naveen]> ALTER TABLE `NOTIFICATION_SEQUENCE` MODIFY COLUMN `NNI_ID` 
BIGINT(20) GENERATED ALWAYS AS (1) STORED;
ERROR 1903 (HY000): Primary key cannot be defined upon a generated column
{noformat}


> SchemaTool initSchema fails on Mariadb 10.2
> ---
>
> Key: HIVE-27749
> URL: https://issues.apache.org/jira/browse/HIVE-27749
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
>Affects Versions: 4.0.0-beta-1
>Reporter: Stamatis Zampetakis
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: mariadb-metastore-schema-tests.patch
>
>
> Schema initialization for 4.0.0-beta-1 fails when run on Mariadb 10.2.
> The problem is reproducible on current 
> (e5a7ce2f091da1f8a324da6e489cda59b9e4bfc6) master by applying the 
> [^mariadb-metastore-schema-tests.patch] and then running:
> {noformat}
> mvn test -Dtest=TestMariadb#install -Dtest.groups=""{noformat}
> The error is shown below:
> {noformat}
> 315/409  ALTER TABLE `NOTIFICATION_SEQUENCE` MODIFY COLUMN `NNI_ID` 
> BIGINT(20) GENERATED ALWAYS AS (1) STORED NOT NULL;
> Error: (conn=11) You have an error in your SQL syntax; check the manual that 
> corresponds to your MariaDB server version for the right syntax to use near 
> 'NOT NULL' at line 1 (state=42000,code=1064)
> Aborting command set because "force" is false and command failed: "ALTER 
> TABLE `NOTIFICATION_SEQUENCE` MODIFY COLUMN `NNI_ID` BIGINT(20) GENERATED 
> ALWAYS AS (1) STORED NOT NULL;"
> [ERROR] 2023-09-27 21:36:30.317 [main] MetastoreSchemaTool - Schema 
> initialization FAILED! Metastore state would be inconsistent!
> Schema initialization FAILED! Metastore state would be inconsistent!
> [ERROR] 2023-09-27 21:36:30.317 [main] MetastoreSchemaTool - Underlying 
> cause: java.io.IOException : Schema script failed, errorcode OTHER
> Underlying cause: java.io.IOException : Schema script failed, errorcode OTHER
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization 
> FAILED! Metastore state would be inconsistent!
> at 
> org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskInit.execute(SchemaToolTaskInit.java:66)
> at 
> org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:480)
> at 
> org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:425)
> at 
> org.apache.hadoop.hive.metastore.dbinstall.rules.DatabaseRule.installLatest(DatabaseRule.java:269)
> at 
> org.apache.hadoop.hive.metastore.dbinstall.DbInstallBase.install(DbInstallBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at 
> 

[jira] [Resolved] (HIVE-27885) Cast decimal from string with space without digits before dot returns NULL

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27885.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. Thank you [~nareshpr] for the patch.

> Cast decimal from string with space without digits before dot returns NULL
> --
>
> Key: HIVE-27885
> URL: https://issues.apache.org/jira/browse/HIVE-27885
> Project: Hive
>  Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>
> eg.,
> select cast(". " as decimal(8,4))
> {code:java}
> – Expected output
> 0.
> – Actual output
> NULL
> {code}
> select cast("0. " as decimal(8,4))
> {code:java}
> – Actual output
> 0.
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-24815) Remove "IDXS" Table from Metastore Schema

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-24815.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. [~rtrivedi12] do you plan on porting this change 
to branch-3? If so, please create a new jira as this change would not work 
as-is on branch-3 due to the schema file changes.

> Remove "IDXS" Table from Metastore Schema
> -
>
> Key: HIVE-24815
> URL: https://issues.apache.org/jira/browse/HIVE-24815
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore, Standalone Metastore
>Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2, 3.2.0, 4.0.0
>Reporter: Hunter Logan
>Assignee: Riju Trivedi
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In Hive 3 the rarely used "INDEXES" was removed from the DDL
> https://issues.apache.org/jira/browse/HIVE-18448
>  
> There are a few issues here:
>  # The Standalone-Metastore schema for Hive 3+ all include the "IDXS" table, 
> which has no function.
>  ** 
> [https://github.com/apache/hive/tree/master/standalone-metastore/metastore-server/src/main/sql/mysql]
>  # The upgrade schemas from 2.x -> 3.x do not do any cleanup of the IDXS table
>  ** If a user used the "INDEXES" feature in 2.x and then upgrades their 
> metastore to 3.x+ they cannot drop any table that has an index on it due to 
> "IDXS_FK1" constraint since the TBLS entry is referenced in the IDXS table
>  ** Since INDEX is no longer in the DDL they cannot run any command from Hive 
> to drop the index.
>  ** Users can manually connect to the metastore and either drop the IDXS 
> table or the foreign key constraint
>  
> Since indexes provide no benefits in Hive 3+ it should be fine to drop them 
> completely in the schema upgrade scripts. At the very least the 2.x -> 3.x+ 
> scripts should drop the fk constraint.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27834) [Postgres] Use schema names instead of db names

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27834:


 Summary: [Postgres] Use schema names instead of db names
 Key: HIVE-27834
 URL: https://issues.apache.org/jira/browse/HIVE-27834
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam


The Hiveserver2 side support for JDBC storage handler uses a select query to 
fetch the table/column metadata. something like this.
select * from  limit 1; --> where the tbl_name is 
schemaName.tableName where schemaName is the value if hive.sql.schema on the 
table.

https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java#L125-L129
https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java#L557-L563

So natively in Postgres, the following happens from PSQL (same is true for 
oracle where database and schemas are different, on MySQL seems to treat them 
as one and same)

{noformat}
hive_hms_testing=> select * from public."TXNS" limit 1;
 TXN_ID | TXN_STATE | TXN_STARTED | TXN_LAST_HEARTBEAT | TXN_USER | TXN_HOST | 
TXN_AGENT_INFO | TXN_META_INFO | TXN_HEARTBEAT_COUNT | TXN_TYPE 
+---+-++--+--++---+-+--
(0 rows)

hive_hms_testing=> select * from "TXNS" limit 1;
 TXN_ID | TXN_STATE | TXN_STARTED | TXN_LAST_HEARTBEAT | TXN_USER | TXN_HOST | 
TXN_AGENT_INFO | TXN_META_INFO | TXN_HEARTBEAT_COUNT | TXN_TYPE 
+---+-++--+--++---+-+--
(0 rows)

hive_hms_testing=> select * from hive_hms_testing."TXNS" limit 1;
ERROR:  relation "hive_hms_testing.TXNS" does not exist
LINE 1: select * from hive_hms_testing."TXNS" limit 1;
  ^
hive_hms_testing=> 
{noformat}

so schemaname cannot be the name of the database. if a schema name is not 
specified, then all tables in the database, across all schemas are listed. But 
if user wants to limit to a certain schema, they have to use the schemaname in 
"connector.remoteDbName" which then needs to be used as "hive.sql.schema" for 
the table.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-26192) JDBC data connector queries occur exception at cbo stage

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781386#comment-17781386
 ] 

Naveen Gangam commented on HIVE-26192:
--

[~zhangbutao] exactly. Also needs a null check when inserting this property. I 
have the fix I have been testing for a couple of days against other DBs. Should 
be done shortly. I can create a PR. Thanks again

> JDBC data connector queries  occur exception at cbo stage
> -
>
> Key: HIVE-26192
> URL: https://issues.apache.org/jira/browse/HIVE-26192
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0-alpha-2
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> If you do a select query qtest with jdbc data connector, you will  see 
> exception at cbo stage:
> {code:java}
> [ERROR] Failures:
> [ERROR]   TestMiniLlapCliDriver.testCliDriver:62 Client execution failed with 
> error code = 4
> running
> select * from country
> fname=dataconnector_mysql.qSee ./ql/target/tmp/log/hive.log or 
> ./itests/qtest/target/tmp/log/hive.log, or check ./ql/target/surefire-reports 
> or ./itests/qtest/target/surefire-reports/ for specific test cases logs.
>  org.apache.hadoop.hive.ql.parse.SemanticException: Table qtestDB.country was 
> not found in the database
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genTableLogicalPlan(CalcitePlanner.java:3078)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5048)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1665)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1605)
>         at 
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
>         at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
>         at 
> org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
>         at 
> org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1357)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:567)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12587)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
>         at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:317)
>         at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:106)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:500)
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:452)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:416)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:410)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:200)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:126)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:421)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:352)
>         at 
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:727)
>         at 
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:697)
>         at 
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:114)
>         at 
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
>         at 
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver(TestMiniLlapCliDriver.java:62)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>         at 
> 

[jira] [Commented] (HIVE-26192) JDBC data connector queries occur exception at cbo stage

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781133#comment-17781133
 ] 

Naveen Gangam commented on HIVE-26192:
--

[~zhangbutao] A part of this fix seems to be causing issues with postgres.

The Hiveserver2 side support for JDBC storage handler uses a select query to 
fetch the table/column metadata. something like this.
select * from  limit 1; --> where the tbl_name is 
schemaName.tableName where schemaName is the value if hive.sql.schema on the 
table.

https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java#L125-L129
https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java#L557-L563

So natively in Postgres, the following happens from PSQL (same is true for 
oracle where database and schemas are different, on MySQL seems to treat them 
as one and same)
{noformat}
hive_hms_testing=> select * from public."TXNS" limit 1;
 TXN_ID | TXN_STATE | TXN_STARTED | TXN_LAST_HEARTBEAT | TXN_USER | TXN_HOST | 
TXN_AGENT_INFO | TXN_META_INFO | TXN_HEARTBEAT_COUNT | TXN_TYPE 
+---+-++--+--++---+-+--
(0 rows)

hive_hms_testing=> select * from "TXNS" limit 1;
 TXN_ID | TXN_STATE | TXN_STARTED | TXN_LAST_HEARTBEAT | TXN_USER | TXN_HOST | 
TXN_AGENT_INFO | TXN_META_INFO | TXN_HEARTBEAT_COUNT | TXN_TYPE 
+---+-++--+--++---+-+--
(0 rows)

hive_hms_testing=> select * from hive_hms_testing."TXNS" limit 1;
ERROR:  relation "hive_hms_testing.TXNS" does not exist
LINE 1: select * from hive_hms_testing."TXNS" limit 1;
  ^
hive_hms_testing=> 
{noformat}

so schemaname cannot be the name of the database. if a schema name is not 
specified, then all tables in the database, across all schemas are listed. But 
if user wants to limit to a certain schema, they have to use the schemaname in 
"connector.remoteDbName" which then needs to be used as "hive.sql.schema" for 
the table.

So I am thinking we should only set when the remoteDbName is not null and flip 
the value of getCatalogName() and getDatabaseName() in the PG connector.
{code:java}
table.getParameters().put(JDBC_SCHEMA, scoped_db);
{code}


> JDBC data connector queries  occur exception at cbo stage
> -
>
> Key: HIVE-26192
> URL: https://issues.apache.org/jira/browse/HIVE-26192
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0-alpha-2
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> If you do a select query qtest with jdbc data connector, you will  see 
> exception at cbo stage:
> {code:java}
> [ERROR] Failures:
> [ERROR]   TestMiniLlapCliDriver.testCliDriver:62 Client execution failed with 
> error code = 4
> running
> select * from country
> fname=dataconnector_mysql.qSee ./ql/target/tmp/log/hive.log or 
> ./itests/qtest/target/tmp/log/hive.log, or check ./ql/target/surefire-reports 
> or ./itests/qtest/target/surefire-reports/ for specific test cases logs.
>  org.apache.hadoop.hive.ql.parse.SemanticException: Table qtestDB.country was 
> not found in the database
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genTableLogicalPlan(CalcitePlanner.java:3078)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5048)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1665)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1605)
>         at 
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
>         at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
>         at 
> org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
>         at 
> org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1357)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:567)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12587)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
>         at 
> 

[jira] [Commented] (HIVE-27814) Support VIEWs in the metadata federation

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-27814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1440#comment-1440
 ] 

Naveen Gangam commented on HIVE-27814:
--

[~zhangbutao] What do you think of this? Can you think of any datasources that 
we should exclude from such support?

> Support VIEWs in the metadata federation
> 
>
> Key: HIVE-27814
> URL: https://issues.apache.org/jira/browse/HIVE-27814
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Affects Versions: 4.0.0-beta-1
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>
> Currently, we only federate the TABLE type objects from the remote 
> datasource. We should be able to pull in VIEW type objects as well.
> It appears we can currently create a JDBC-storage handler based table in Hive 
> that points to a view in the remote DB server. I do not see a reason to not 
> include this in the list of federated objects we pull in.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27814) Support VIEWs in the metadata federation

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27814:


 Summary: Support VIEWs in the metadata federation
 Key: HIVE-27814
 URL: https://issues.apache.org/jira/browse/HIVE-27814
 Project: Hive
  Issue Type: Sub-task
  Components: Standalone Metastore
Affects Versions: 4.0.0-beta-1
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Currently, we only federate the TABLE type objects from the remote datasource. 
We should be able to pull in VIEW type objects as well.

It appears we can currently create a JDBC-storage handler based table in Hive 
that points to a view in the remote DB server. I do not see a reason to not 
include this in the list of federated objects we pull in.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27597) Implement JDBC Connector for HiveServer

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27597.
--
Resolution: Fixed

Fix has been merged to master. Thank you [~ayushtkn] and [~zhangbutao] for the 
review. Closing the jira.

[~zhangbutao] I have granted you write access to the Data connectors page. Hope 
it works. Thank you

> Implement JDBC Connector for HiveServer 
> 
>
> Key: HIVE-27597
> URL: https://issues.apache.org/jira/browse/HIVE-27597
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>  Labels: pull-request-available
>
> The initial idea of having a thrift based connector, that would enable Hive 
> Metastore to use thrift APIs to interact with another metastore from another 
> cluster, has some limitations. Features like column masking support become a 
> challenge as we may bypass the authz controls on the remote cluster.
> Instead if we could federate a query from one instance of HS2 to another 
> instance of HS2 over JDBC, we would address the above concerns. This will 
> atleast give us the ability to access tables across cluster boundaries.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27746) Hive Metastore should send single AlterPartitionEvent with list of partitions

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27746:


 Summary: Hive Metastore should send single AlterPartitionEvent 
with list of partitions
 Key: HIVE-27746
 URL: https://issues.apache.org/jira/browse/HIVE-27746
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Reporter: Naveen Gangam


In HIVE-3938, work was done to send single AddPartitionEvent for APIs that add 
partitions in bulk. Similarly, we have alter_partitions APIs that alter 
partitions in bulk via a single HMS call. For such events, we should also send 
a single AlterPartitionEvent with a list of partitions in it.
This would be way more efficient than having to send and process them 
individually.

This fix will be incompatible with the older clients that expect single 
partition.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27728) Changed behavior for alter table rename partition from legacy tables

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-27728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768963#comment-17768963
 ] 

Naveen Gangam commented on HIVE-27728:
--

[~dengzh] Would you be able to take a look at this? 
For tables that are converted/translated to EXTERNAL_TABLE, we add 
"TRANSLATED_TO_EXTERNAL" to the table properties. So any partitions (or even 
table) that are being renamed for such tables, we can treat them as the same 
for MANAGED tables and move the data as well.
https://issues.apache.org/jira/browse/HIVE-26158 might give some guidance.

> Changed behavior for alter table rename partition from legacy tables
> 
>
> Key: HIVE-27728
> URL: https://issues.apache.org/jira/browse/HIVE-27728
> Project: Hive
>  Issue Type: Bug
>Reporter: Naveen Gangam
>Assignee: Zhihua Deng
>Priority: Major
>
> set hive.create.as.external.legacy=true;
> CREATE TABLE default.metadata_test1(
> emp_number int,
> emp_name string,
> city string)
> PARTITIONED BY(state string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ‘,’;
> INSERT INTO default.metadata_test1 PARTITION(state=‘A’) VALUES (11, ‘ABC’, 
> ‘AA’);
> INSERT INTO default.metadata_test1 PARTITION(state=‘B’) VALUES (12, ‘XYZ’, 
> ‘BX’);
> INSERT INTO default.metadata_test1 PARTITION(state=‘B’) VALUES (13, ‘UVW’, 
> ‘BU’);
> from hdfs
> $ hdfs dfs -ls /warehouse/tablespace/external/hive/metadata_test1
> Found 2 items
> drwxr-xr-x - hive hive 0 2023-08-11 14:24 
> /warehouse/tablespace/external/hive/metadata_test1/state=A
> drwxr-xr-x - hive hive 0 2023-08-11 14:25 
> /warehouse/tablespace/external/hive/metadata_test1/state=B
> Now when we alter the partition.
> ALTER TABLE default.metadata_test1 PARTITION (state=‘A’) RENAME TO PARTITION 
> (state=‘C’);
> select * from default.metadata_test1;
> ++--+--+---+
> | metadata_test1.emp_number | metadata_test1.emp_name | metadata_test1.city | 
> metadata_test1.state |
> ++--+--+---+
> | 12 | XYZ | BX | B |
> | 13 | UVW | BU | B |
> | 11 | ABC | AA | C |
> ++--+--+---+
> show partitions default.metadata_test1;
> ++
> | partition |
> ++
> | state=B |
> | state=C |
> ++
> But from HDFS
> hdfs dfs -ls /warehouse/tablespace/external/hive/metadata_test1
> Found 2 items
> drwxr-xr-x - hive hive 0 2023-08-11 14:24 
> /warehouse/tablespace/external/hive/metadata_test1/state=A
> drwxr-xr-x - hive hive 0 2023-08-11 14:25 
> /warehouse/tablespace/external/hive/metadata_test1/state=B
> Hive only relocates/renames the partition locations to match the new 
> partition value if it is a MANAGED table. For external tables, we assume that 
> ETL pipelines are writing to the old location and thus will not rename the 
> dir.
> But for legacy tables, (create table) that would have created non-acid 
> managed tables, we convert this tables to EXTERNAL_TABLE but with auto-purge 
> enabled. For such tables, we should take the liberty to managed these 
> locations as well to match the legacy MANAGED table behavior.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-27728) Changed behavior for alter table rename partition from legacy tables

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-27728:


Assignee: Zhihua Deng

> Changed behavior for alter table rename partition from legacy tables
> 
>
> Key: HIVE-27728
> URL: https://issues.apache.org/jira/browse/HIVE-27728
> Project: Hive
>  Issue Type: Bug
>Reporter: Naveen Gangam
>Assignee: Zhihua Deng
>Priority: Major
>
> set hive.create.as.external.legacy=true;
> CREATE TABLE default.metadata_test1(
> emp_number int,
> emp_name string,
> city string)
> PARTITIONED BY(state string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ‘,’;
> INSERT INTO default.metadata_test1 PARTITION(state=‘A’) VALUES (11, ‘ABC’, 
> ‘AA’);
> INSERT INTO default.metadata_test1 PARTITION(state=‘B’) VALUES (12, ‘XYZ’, 
> ‘BX’);
> INSERT INTO default.metadata_test1 PARTITION(state=‘B’) VALUES (13, ‘UVW’, 
> ‘BU’);
> from hdfs
> $ hdfs dfs -ls /warehouse/tablespace/external/hive/metadata_test1
> Found 2 items
> drwxr-xr-x - hive hive 0 2023-08-11 14:24 
> /warehouse/tablespace/external/hive/metadata_test1/state=A
> drwxr-xr-x - hive hive 0 2023-08-11 14:25 
> /warehouse/tablespace/external/hive/metadata_test1/state=B
> Now when we alter the partition.
> ALTER TABLE default.metadata_test1 PARTITION (state=‘A’) RENAME TO PARTITION 
> (state=‘C’);
> select * from default.metadata_test1;
> ++--+--+---+
> | metadata_test1.emp_number | metadata_test1.emp_name | metadata_test1.city | 
> metadata_test1.state |
> ++--+--+---+
> | 12 | XYZ | BX | B |
> | 13 | UVW | BU | B |
> | 11 | ABC | AA | C |
> ++--+--+---+
> show partitions default.metadata_test1;
> ++
> | partition |
> ++
> | state=B |
> | state=C |
> ++
> But from HDFS
> hdfs dfs -ls /warehouse/tablespace/external/hive/metadata_test1
> Found 2 items
> drwxr-xr-x - hive hive 0 2023-08-11 14:24 
> /warehouse/tablespace/external/hive/metadata_test1/state=A
> drwxr-xr-x - hive hive 0 2023-08-11 14:25 
> /warehouse/tablespace/external/hive/metadata_test1/state=B
> Hive only relocates/renames the partition locations to match the new 
> partition value if it is a MANAGED table. For external tables, we assume that 
> ETL pipelines are writing to the old location and thus will not rename the 
> dir.
> But for legacy tables, (create table) that would have created non-acid 
> managed tables, we convert this tables to EXTERNAL_TABLE but with auto-purge 
> enabled. For such tables, we should take the liberty to managed these 
> locations as well to match the legacy MANAGED table behavior.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27728) Changed behavior for alter table rename partition from legacy tables

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27728:


 Summary: Changed behavior for alter table rename partition from 
legacy tables
 Key: HIVE-27728
 URL: https://issues.apache.org/jira/browse/HIVE-27728
 Project: Hive
  Issue Type: Bug
Reporter: Naveen Gangam


set hive.create.as.external.legacy=true;
CREATE TABLE default.metadata_test1(
emp_number int,
emp_name string,
city string)
PARTITIONED BY(state string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ‘,’;

INSERT INTO default.metadata_test1 PARTITION(state=‘A’) VALUES (11, ‘ABC’, 
‘AA’);
INSERT INTO default.metadata_test1 PARTITION(state=‘B’) VALUES (12, ‘XYZ’, 
‘BX’);
INSERT INTO default.metadata_test1 PARTITION(state=‘B’) VALUES (13, ‘UVW’, 
‘BU’);
from hdfs
$ hdfs dfs -ls /warehouse/tablespace/external/hive/metadata_test1
Found 2 items
drwxr-xr-x - hive hive 0 2023-08-11 14:24 
/warehouse/tablespace/external/hive/metadata_test1/state=A
drwxr-xr-x - hive hive 0 2023-08-11 14:25 
/warehouse/tablespace/external/hive/metadata_test1/state=B
Now when we alter the partition.
ALTER TABLE default.metadata_test1 PARTITION (state=‘A’) RENAME TO PARTITION 
(state=‘C’);

select * from default.metadata_test1;
++--+--+---+
| metadata_test1.emp_number | metadata_test1.emp_name | metadata_test1.city | 
metadata_test1.state |
++--+--+---+
| 12 | XYZ | BX | B |
| 13 | UVW | BU | B |
| 11 | ABC | AA | C |
++--+--+---+
show partitions default.metadata_test1;
++
| partition |
++
| state=B |
| state=C |
++
But from HDFS
hdfs dfs -ls /warehouse/tablespace/external/hive/metadata_test1
Found 2 items
drwxr-xr-x - hive hive 0 2023-08-11 14:24 
/warehouse/tablespace/external/hive/metadata_test1/state=A
drwxr-xr-x - hive hive 0 2023-08-11 14:25 
/warehouse/tablespace/external/hive/metadata_test1/state=B

Hive only relocates/renames the partition locations to match the new partition 
value if it is a MANAGED table. For external tables, we assume that ETL 
pipelines are writing to the old location and thus will not rename the dir.

But for legacy tables, (create table) that would have created non-acid managed 
tables, we convert this tables to EXTERNAL_TABLE but with auto-purge enabled. 
For such tables, we should take the liberty to managed these locations as well 
to match the legacy MANAGED table behavior.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27720) Optimize NotificationLog Cleaner

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27720:


 Summary: Optimize NotificationLog Cleaner 
 Key: HIVE-27720
 URL: https://issues.apache.org/jira/browse/HIVE-27720
 Project: Hive
  Issue Type: Improvement
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Currently this thread fetches the events first in batches of 10k (or whatever 
configured) and then deletes them. This is not very optimal. This table has an 
index on the EVENT_ID which are unique. So the records have high cardinality. 
The cleaner thread deletes using high EVENT_TIME which also has high 
cardinality. So the fetching of this events appears to be doing a full table 
scan each time even for a following batches.

Instead of fetching, we should be able to do a delete directly using JDO. This 
way we can delete them all in one go at the DB level.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27694) Include HiveIcebergSerDe in default list of serdes using HMS

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27694.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Thank you for the review [~ayushtkn] and [~dengzh]. Fix has been merged to 
master. Closing the jira.

> Include HiveIcebergSerDe in default list of serdes using HMS
> 
>
> Key: HIVE-27694
> URL: https://issues.apache.org/jira/browse/HIVE-27694
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>
> MetastoreConf has this default list of SerDes that use HMS to persist the 
> metadata. Iceberg tables also use have their metadata in HMS and hence can be 
> fetched from metastore. This SerDe needs to be added to this list as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27694) Include HiveIcebergSerDe in default list of serdes using HMS

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27694:


 Summary: Include HiveIcebergSerDe in default list of serdes using 
HMS
 Key: HIVE-27694
 URL: https://issues.apache.org/jira/browse/HIVE-27694
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Reporter: Naveen Gangam
Assignee: Naveen Gangam


MetastoreConf has this default list of SerDes that use HMS to persist the 
metadata. Iceberg tables also use have their metadata in HMS and hence can be 
fetched from metastore. This SerDe needs to be added to this list as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27675) Support keystore/truststore types for hive to zookeeper integration points

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27675.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. Thank you for the review

> Support keystore/truststore types for hive to zookeeper integration points
> --
>
> Key: HIVE-27675
> URL: https://issues.apache.org/jira/browse/HIVE-27675
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC, Standalone Metastore
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>
> In HIVE-24253, we added support for HS2/HMS/JDBC DRiver to support other 
> store types like BCFKS (other than JKS). This allows JDBC Clients to connect 
> to HS2 directly. However, with service discovery enabled, the clients have to 
> connect zookeeper to determine HS2 endpoints. This connectivity currently 
> does not support other store types. Similarly, HS2/HMS services also do not 
> provide ability to use different store types for the zk registration process.
> {noformat}
> $ beeline 
> Connecting to 
> jdbc:hive2://:2181/default;httpPath=cliservice;principal=hive/_HOST@;retries=5;serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;transportMode=http;trustStorePassword=RoeCFK11Pq54;trustStoreType=bcfks;zooKeeperNamespace=hiveserver2
> Error: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read 
> HiveServer2 configs from ZooKeeper (state=,code=0) 
> {noformat}
> {noformat}
> Opening socket connection to server :2182. Will attempt to 
> SASL-authenticate using Login Context section 'HiveZooKeeperClient'
> 2023-08-09 13:28:07,591 WARN  io.netty.channel.ChannelInitializer: 
> [nioEventLoopGroup-3-1]: Failed to initialize a channel. Closing: [id: 
> 0x0937583f]
> org.apache.zookeeper.common.X509Exception$SSLContextException: Failed to 
> create KeyManager
> at 
> org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:346)
>  ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
> at 
> org.apache.zookeeper.common.X509Util.createSSLContext(X509Util.java:278) 
> ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
> at 
> org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initSSL(ClientCnxnSocketNetty.java:454)
>  ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
> at 
> org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initChannel(ClientCnxnSocketNetty.java:444)
>  ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
> at 
> org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initChannel(ClientCnxnSocketNetty.java:429)
>  ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
> at 
> io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) 
> [netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at 
> io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) 
> [netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1114)
>  [netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at 
> io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
>  [netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at 
> io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
>  [netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at 
> io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
>  [netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at 
> io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
>  [netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at 
> io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
>  [netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
>  [netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
>  [netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at 
> io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
>  [netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at 
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
>  

[jira] [Created] (HIVE-27675) Support keystore/truststore types for hive to zookeeper integration points

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27675:


 Summary: Support keystore/truststore types for hive to zookeeper 
integration points
 Key: HIVE-27675
 URL: https://issues.apache.org/jira/browse/HIVE-27675
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC, Standalone Metastore
Affects Versions: 3.1.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


In HIVE-24253, we added support for HS2/HMS/JDBC DRiver to support other store 
types like BCFKS (other than JKS). This allows JDBC Clients to connect to HS2 
directly. However, with service discovery enabled, the clients have to connect 
zookeeper to determine HS2 endpoints. This connectivity currently does not 
support other store types. Similarly, HS2/HMS services also do not provide 
ability to use different store types for the zk registration process.
{noformat}
$ beeline 
Connecting to 
jdbc:hive2://:2181/default;httpPath=cliservice;principal=hive/_HOST@;retries=5;serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;transportMode=http;trustStorePassword=RoeCFK11Pq54;trustStoreType=bcfks;zooKeeperNamespace=hiveserver2
Error: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read 
HiveServer2 configs from ZooKeeper (state=,code=0) 
{noformat}


{noformat}
Opening socket connection to server :2182. Will attempt to 
SASL-authenticate using Login Context section 'HiveZooKeeperClient'
2023-08-09 13:28:07,591 WARN  io.netty.channel.ChannelInitializer: 
[nioEventLoopGroup-3-1]: Failed to initialize a channel. Closing: [id: 
0x0937583f]
org.apache.zookeeper.common.X509Exception$SSLContextException: Failed to create 
KeyManager
at 
org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:346)
 ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
at 
org.apache.zookeeper.common.X509Util.createSSLContext(X509Util.java:278) 
~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
at 
org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initSSL(ClientCnxnSocketNetty.java:454)
 ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
at 
org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initChannel(ClientCnxnSocketNetty.java:444)
 ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
at 
org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initChannel(ClientCnxnSocketNetty.java:429)
 ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
at 
io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) 
[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) 
[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1114)
 [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
 [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
 [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
 [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
 [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
 [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
 [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
 [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) 
[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
 [netty-common-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
 [netty-common-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
 [netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) 
[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
 

[jira] [Resolved] (HIVE-27595) Improve efficiency in the filtering hooks

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27595.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. Thank you for the patch [~henrib] and reviews 
[~hemanth619] and [~jfs]

> Improve efficiency in the filtering hooks
> -
>
> Key: HIVE-27595
> URL: https://issues.apache.org/jira/browse/HIVE-27595
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: Naveen Gangam
>Assignee: Henri Biestro
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>
> https://github.com/apache/hive/blob/a406d6d4417277e45b93f1733bed5201afdee29b/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/metastore/HiveMetaStoreAuthorizer.java#L353-L377
> In case where the tableList has large amounts of tables (tested with 200k in 
> my case), the hivePrivilegedObjects could just as big. So both these lists 
> are 200k. 
> Essentially. the code is trying to return a subset of tableList collection 
> that matches the objects returned in hivePrivilegedObjects. This results in a 
> N*N iteration that causes bad performance. (in my case, the HMS client 
> timeout expired and show tables failed). 
> This code needs to be optimized for performance. 
> we have a similar problem in this code as well.
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/AuthorizationMetaStoreFilterHook.java



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27586) Parse dates from strings ignoring trailing (potentialy) invalid chars

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27586.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Merged to master. Thank you [~zabetak] for the patch and [~amansinha] for the 
review.

> Parse dates from strings ignoring trailing (potentialy) invalid chars
> -
>
> Key: HIVE-27586
> URL: https://issues.apache.org/jira/browse/HIVE-27586
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 4.0.0-beta-1
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>  Labels: backwards-compatibility, pull-request-available
> Fix For: 4.0.0
>
>
> The goal of this ticket is to extract and return a valid date from a string 
> value when there is a valid date prefix in the string.
> The following table contains a few illustrative examples highlighting what 
> happens now and what will happen after the proposed changes to ignore 
> trailing characters. HIVE-20007 introduced some behavior changes around this 
> area so the table also displays what was the Hive behavior before that change.
> ||ID||String value||Before HIVE-20007||Current behavior||Ignore trailing 
> chars||
> |1|2023-08-03_16:02:00|2023-08-03|null|2023-08-03|
> |2|2023-08-03-16:02:00|2023-08-03|null|2023-08-03|
> |3|2023-08-0316:02:00|2024-06-11|null|2023-08-03|
> |4|03-08-2023|0009-02-12|null|0003-08-20|
> |5|2023-08-03 GARBAGE|2023-08-03|2023-08-03|2023-08-03|
> |6|2023-08-03TGARBAGE|2023-08-03|2023-08-03|2023-08-03|
> |7|2023-08-03_GARBAGE|2023-08-03|null|2023-08-03|
> This change partially (see example 3 and 4) restores the behavior changes 
> introduced by HIVE-20007 and at the same time makes the current behavior of 
> handling trailing invalid chars more uniform. 
> This change will have an impact on various Hive SQL functions and operators 
> (+/-) that accept dates from string values. A partial list of affected 
> functions is outlined below:
> * CAST (V AS DATE)
> * CAST (V AS TIMESTAMP)
> * TO_DATE
> * DATE_ADD
> * DATE_DIFF
> * WEEKOFYEAR
> * DAYOFWEEK
> * TRUNC



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27554) Validate URL used by SSO workflow for JDBC connection

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27554.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Patch has been merged to master. Thank you for the patch [~henrib]

> Validate URL used by SSO workflow for JDBC connection
> -
>
> Key: HIVE-27554
> URL: https://issues.apache.org/jira/browse/HIVE-27554
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Affects Versions: 3.1.3
>Reporter: Henri Biestro
>Assignee: Henri Biestro
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Add a validation to ensure the URL used during SSO workflow is proper 
> (http/https).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27597) Implement JDBC Connector for HiveServer

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27597:


 Summary: Implement JDBC Connector for HiveServer 
 Key: HIVE-27597
 URL: https://issues.apache.org/jira/browse/HIVE-27597
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Reporter: Naveen Gangam
Assignee: Naveen Gangam


The initial idea of having a thrift based connector, that would enable Hive 
Metastore to use thrift APIs to interact with another metastore from another 
cluster, has some limitations. Features like column masking support become a 
challenge as we may bypass the authz controls on the remote cluster.

Instead if we could federate a query from one instance of HS2 to another 
instance of HS2 over JDBC, we would address the above concerns. This will 
atleast give us the ability to access tables across cluster boundaries.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27595) Improve efficiency in the filtering hooks

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27595:


 Summary: Improve efficiency in the filtering hooks
 Key: HIVE-27595
 URL: https://issues.apache.org/jira/browse/HIVE-27595
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 4.0.0-alpha-2
Reporter: Naveen Gangam
Assignee: Henri Biestro


https://github.com/apache/hive/blob/a406d6d4417277e45b93f1733bed5201afdee29b/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/metastore/HiveMetaStoreAuthorizer.java#L353-L377

In case where the tableList has large amounts of tables (tested with 200k in my 
case), the hivePrivilegedObjects could just as big. So both these lists are 
200k. 

Essentially. the code is trying to return a subset of tableList collection that 
matches the objects returned in hivePrivilegedObjects. This results in a N*N 
iteration that causes bad performance. (in my case, the HMS client timeout 
expired and show tables failed). 

This code needs to be optimized for performance. 

we have a similar problem in this code as well.
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/AuthorizationMetaStoreFilterHook.java




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27195) Add database authorization for drop table command

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27195.
--
Fix Version/s: 4.0.0-beta-1
   Resolution: Fixed

Fix has been merged to master for beta1 release. Thank you for the patch.

> Add database authorization for drop table command
> -
>
> Key: HIVE-27195
> URL: https://issues.apache.org/jira/browse/HIVE-27195
> Project: Hive
>  Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-beta-1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Include authorization of the database object during the "drop table" command. 
> Similar to "Create table", DB permissions should be verified in the case of 
> "drop table" too. Add the database object along with the table object to the 
> list of output objects sent for verifying privileges. This change would 
> ensure that in case of a non-existent table or temporary table (skipped from 
> authorization after HIVE-20051), the authorizer will verify privileges for 
> the database object.
> This would also prevent DROP TABLE IF EXISTS command failure for temporary or 
> non-existing tables with `RangerHiveAuthorizer`. In case of 
> temporary/non-existing table, empty input and output HivePrivilege Objects 
> are sent to Ranger authorizer and after 
> https://issues.apache.org/jira/browse/RANGER-3407 authorization request is 
> built from command in case of empty objects. Hence, the drop table if Exists 
> command fails with  HiveAccessControlException.
> Steps to Repro:
> {code:java}
> use test; CREATE TEMPORARY TABLE temp_table (id int);
> drop table if exists test.temp_table;
> Error: Error while compiling statement: FAILED: HiveAccessControlException 
> Permission denied: user [rtrivedi] does not have [DROP] privilege on 
> [test/temp_table] (state=42000,code=4) {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27113) Increasing default for hive.thrift.client.max.message.size to 2 GB

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27113.
--
Fix Version/s: 4.0.0-beta-1
   Resolution: Fixed

Fix has been merged to master. Closing the jira. Thank you [~rtrivedi12]

> Increasing default for hive.thrift.client.max.message.size to 2 GB
> --
>
> Key: HIVE-27113
> URL: https://issues.apache.org/jira/browse/HIVE-27113
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-beta-1
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> HIVE_THRIFT_CLIENT_MAX_MESSAGE_SIZE("hive.thrift.client.max.message.size", 
> "1gb",
> new SizeValidator(-1L, true, (long) Integer.MAX_VALUE, true),
> "Thrift client configuration for max message size. 0 or -1 will use 
> the default defined in the Thrift " +
> "library. The upper limit is 2147483648 bytes (or 2gb).")
> Documentation on the help suggests setting 2147483648 while Integer Max is 
> 2147483647. So, it actually becomes -1 and gets set to thrift default limit 
> (100 MB)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27425) Upgrade Nimbus-JOSE-JWT to 9.24+ due to CVEs coming from json-smart

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27425.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

> Upgrade Nimbus-JOSE-JWT to 9.24+ due to CVEs coming from json-smart
> ---
>
> Key: HIVE-27425
> URL: https://issues.apache.org/jira/browse/HIVE-27425
> Project: Hive
>  Issue Type: Task
>Reporter: Devaspati Krishnatri
>Assignee: Devaspati Krishnatri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Nimbus-JOSE-JWT before 9.24 is using the vulnerable version of json-smart. 
> nimbus-jose-jwt has dropped the json-smart dependency completely with 
> nimbus-jose-jwt 9.24 and replaces it with *Gson 2.9.1 (shaded),* as seen in 
> the commit history here: 
> [https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/tag/9.24].
> Json-smart before 2.4.9 is affected by CVE-2023-1370
> CVE-2023-1370 - [Json-smart]([https://netplex.github.io/json-smart/]) is a 
> performance focused, JSON processor lib. When reaching a '[' or '{' character 
> in the JSON input, the code parses an array or an object respectively. It was 
> discovered that the code does not have any limit to the nesting of such 
> arrays or objects. Since the parsing of nested arrays and objects is done 
> recursively, nesting too many of them can cause a stack exhaustion (stack 
> overflow) and crash the software.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27425) Upgrade Nimbus-JOSE-JWT to 9.24+ due to CVEs coming from json-smart

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-27425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733217#comment-17733217
 ] 

Naveen Gangam commented on HIVE-27425:
--

Fix has been merged to master. Thank you for the contribution 
[~devaspatikrishnatri]. Closing the jira.

> Upgrade Nimbus-JOSE-JWT to 9.24+ due to CVEs coming from json-smart
> ---
>
> Key: HIVE-27425
> URL: https://issues.apache.org/jira/browse/HIVE-27425
> Project: Hive
>  Issue Type: Task
>Reporter: Devaspati Krishnatri
>Assignee: Devaspati Krishnatri
>Priority: Major
>  Labels: pull-request-available
>
> Nimbus-JOSE-JWT before 9.24 is using the vulnerable version of json-smart. 
> nimbus-jose-jwt has dropped the json-smart dependency completely with 
> nimbus-jose-jwt 9.24 and replaces it with *Gson 2.9.1 (shaded),* as seen in 
> the commit history here: 
> [https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/tag/9.24].
> Json-smart before 2.4.9 is affected by CVE-2023-1370
> CVE-2023-1370 - [Json-smart]([https://netplex.github.io/json-smart/]) is a 
> performance focused, JSON processor lib. When reaching a '[' or '{' character 
> in the JSON input, the code parses an array or an object respectively. It was 
> discovered that the code does not have any limit to the nesting of such 
> arrays or objects. Since the parsing of nested arrays and objects is done 
> recursively, nesting too many of them can cause a stack exhaustion (stack 
> overflow) and crash the software.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27398) SHOW CREATE TABLE doesn't output backticks for CLUSTERED by Col names

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27398.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. Thank you for the fix [~rtrivedi12]. Closing the 
jira.

> SHOW CREATE TABLE doesn't output backticks for CLUSTERED by Col names
> -
>
> Key: HIVE-27398
> URL: https://issues.apache.org/jira/browse/HIVE-27398
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>
> SHOW CREATE TABLE output uses backticks for all column names and partition 
> column names but does not include backticks for CLUSTERED BY column names. 
> This causes ParseException during table creation when any bucket column 
> identifier matches reserved keywords 
> {code:java}
> CREATE TABLE `test_ts_reserved_keyword7`(
> `member_id` varchar(8),
> `plan_nr` varchar(11),
> `timestamp` timestamp,
> `shared_ind` varchar(1),
> `user_id` varchar(8))
> CLUSTERED BY (
> member_nr,
> plan_nr,
> `timestamp`)
> INTO 4 BUCKETS;
> SHOW CREATE TABLE test_ts_reserved_keyword7;
> CREATE TABLE `test_ts_reserved_keyword7`(
> `member_id` varchar(8),
> `plan_nr` varchar(11),
> `timestamp` timestamp, 
> `shared_ind` varchar(1), 
> `user_id` varchar(8)) 
> CLUSTERED BY (
> member_id,
> plan_nr,
> timestamp)
> INTO 4 BUCKETS
> ROW FORMAT 
> SERDE'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'STORED AS 
> INPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';
> This fails with "Error while compiling statement: FAILED: ParseException line 
> 13:0 cannot recognize input near 'timestamp' ')' 'INTO' in column name"{code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27311) Improve LDAP auth to support generic search bind authentication

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27311.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Thank you for the review. Fix has been merged to master.

> Improve LDAP auth to support generic search bind authentication
> ---
>
> Key: HIVE-27311
> URL: https://issues.apache.org/jira/browse/HIVE-27311
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Hive's LDAP auth configuration is home-baked and a bit specific to hive. This 
> was by design intending to be as flexible as it can be for accommodating 
> various LDAP implementations. But this does not necessarily make it easy to 
> configure hive with such custom values for ldap filtering when most other 
> components accept generic ldap filters, for example: search bind filters.
> There has to be a layer of translation to have it configured. Instead we can 
> enhance Hive to support generic search bind filters.
> To support this, I am proposing adding NEW alternate configurations. 
> hive.server2.authentication.ldap.userSearchFilter
> hive.server2.authentication.ldap.groupSearchFilter
> hive.server2.authentication.ldap.groupBaseDN
> Search bind filtering will also use EXISTING config param
> hive.server2.authentication.ldap.baseDN
> This is alternate configuration and will be used first if specified. So users 
> can continue to use existing configuration as well. These changes should not 
> interfere with existing configurations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27138) MapJoinOperator throws NPE when computing OuterJoin with filter expressions on small table

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-27138:
-
Priority: Blocker  (was: Major)

> MapJoinOperator throws NPE when computing OuterJoin with filter expressions 
> on small table
> --
>
> Key: HIVE-27138
> URL: https://issues.apache.org/jira/browse/HIVE-27138
> Project: Hive
>  Issue Type: Bug
>Reporter: Seonggon Namgung
>Assignee: Seonggon Namgung
>Priority: Blocker
>  Labels: hive-4.0.0-must, pull-request-available
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Hive throws NPE when running mapjoin_filter_on_outerjoin.q using Tez engine. 
> (I used TestMiniLlapCliDriver.)
> The NPE is thrown by CommonJoinOperator.getFilterTag(), which just retreives 
> the last object from the given list.
> To the best of my knowledge, if Hive selects MapJoin to perform Join 
> operation, filterTag should be computed and appended to a row before the row 
> is passed to MapJoinOperator.
> In the case of MapReduce engine, this is done by HashTableSinkOperator.
> However, I cannot find any logic pareparing filterTag for small tables when 
> Hive uses Tez engine.
> I think there are 2 available options:
> 1. Don't use MapJoinOperator if a small table has filter expression.
> 2. Add a new logic that computes and passes filterTag to MapJoinOperator.
> I am working on the second option and ready to discuss about it.
> It would be grateful if you could give any opinion about this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27269) VectorizedMapJoin returns wrong result for TPC-DS query 97

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-27269:
-
Labels: hive-4.0.0-must  (was: )

> VectorizedMapJoin returns wrong result for TPC-DS query 97
> --
>
> Key: HIVE-27269
> URL: https://issues.apache.org/jira/browse/HIVE-27269
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Seonggon Namgung
>Priority: Major
>  Labels: hive-4.0.0-must
>
> TPC-DS query 97 returns wrong results when hive.auto.convert.join and 
> hive.vectorized.execution.enabled are set to true.
>  
> Result of query 97 on 1TB text dataset:
> CommonMergeJoinOperator(hive.auto.convert.join=false): 534151529, 
> 284185{*}746{*}, 84163
> MapJoinOperator(hive.auto.convert.join=true, 
> hive.vectorized.execution.enabled=false): 534151529, 284185{*}746{*}, 84163
> VectorMapJoinOperator(hive.auto.convert.join=true, 
> hive.vectorized.execution.enabled=true): 534151529, 284185{*}388{*}, 84163
>  
> Also I observed that VectorizedMapJoin returns different results for 100GB 
> dataset when I run query 97 twice, but I could not reproduce it since then.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27269) VectorizedMapJoin returns wrong result for TPC-DS query 97

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-27269:
-
Priority: Blocker  (was: Major)

> VectorizedMapJoin returns wrong result for TPC-DS query 97
> --
>
> Key: HIVE-27269
> URL: https://issues.apache.org/jira/browse/HIVE-27269
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Seonggon Namgung
>Priority: Blocker
>  Labels: hive-4.0.0-must
>
> TPC-DS query 97 returns wrong results when hive.auto.convert.join and 
> hive.vectorized.execution.enabled are set to true.
>  
> Result of query 97 on 1TB text dataset:
> CommonMergeJoinOperator(hive.auto.convert.join=false): 534151529, 
> 284185{*}746{*}, 84163
> MapJoinOperator(hive.auto.convert.join=true, 
> hive.vectorized.execution.enabled=false): 534151529, 284185{*}746{*}, 84163
> VectorMapJoinOperator(hive.auto.convert.join=true, 
> hive.vectorized.execution.enabled=true): 534151529, 284185{*}388{*}, 84163
>  
> Also I observed that VectorizedMapJoin returns different results for 100GB 
> dataset when I run query 97 twice, but I could not reproduce it since then.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27311) Improve LDAP auth to support generic search bind authentication

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27311:


 Summary: Improve LDAP auth to support generic search bind 
authentication
 Key: HIVE-27311
 URL: https://issues.apache.org/jira/browse/HIVE-27311
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 4.0.0-alpha-2
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Hive's LDAP auth configuration is home-baked and a bit specific to hive. This 
was by design intending to be as flexible as it can be for accommodating 
various LDAP implementations. But this does not necessarily make it easy to 
configure hive with such custom values for ldap filtering when most other 
components accept generic ldap filters, for example: search bind filters.

There has to be a layer of translation to have it configured. Instead we can 
enhance Hive to support generic search bind filters.

To support this, I am proposing adding NEW alternate configurations. 
hive.server2.authentication.ldap.userSearchFilter
hive.server2.authentication.ldap.groupSearchFilter
hive.server2.authentication.ldap.groupBaseDN

Search bind filtering will also use EXISTING config param
hive.server2.authentication.ldap.baseDN

This is alternate configuration and will be used first if specified. So users 
can continue to use existing configuration as well. These changes should not 
interfere with existing configurations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-25205) Reduce overhead of adding write notification log during batch loading of partition..

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-25205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-25205:
-
External issue URL: https://github.com/apache/hive/pull/2357

> Reduce overhead of adding write notification log during batch loading of 
> partition..
> 
>
> Key: HIVE-25205
> URL: https://issues.apache.org/jira/browse/HIVE-25205
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, HiveServer2
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: performance
> Fix For: 4.0.0
>
>
> During batch loading of partition the write notification logs are added for 
> each partition added. This is causing delay in execution as the call to HMS 
> is done for each partition. This can be optimised by adding a new API in HMS 
> to send a batch of partition and then this batch can be added together to the 
> backend database. Once we have a batch of notification log, at HMS side, code 
> can be optimised to add the logs using single call to backend RDBMS. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-25205) Reduce overhead of adding write notification log during batch loading of partition..

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-25205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17718289#comment-17718289
 ] 

Naveen Gangam commented on HIVE-25205:
--

Merged via https://github.com/apache/hive/pull/2357

> Reduce overhead of adding write notification log during batch loading of 
> partition..
> 
>
> Key: HIVE-25205
> URL: https://issues.apache.org/jira/browse/HIVE-25205
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, HiveServer2
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: performance
>
> During batch loading of partition the write notification logs are added for 
> each partition added. This is causing delay in execution as the call to HMS 
> is done for each partition. This can be optimised by adding a new API in HMS 
> to send a batch of partition and then this batch can be added together to the 
> backend database. Once we have a batch of notification log, at HMS side, code 
> can be optimised to add the logs using single call to backend RDBMS. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-25205) Reduce overhead of adding write notification log during batch loading of partition..

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-25205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-25205:
-
Fix Version/s: 4.0.0

> Reduce overhead of adding write notification log during batch loading of 
> partition..
> 
>
> Key: HIVE-25205
> URL: https://issues.apache.org/jira/browse/HIVE-25205
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, HiveServer2
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: performance
> Fix For: 4.0.0
>
>
> During batch loading of partition the write notification logs are added for 
> each partition added. This is causing delay in execution as the call to HMS 
> is done for each partition. This can be optimised by adding a new API in HMS 
> to send a batch of partition and then this batch can be added together to the 
> backend database. Once we have a batch of notification log, at HMS side, code 
> can be optimised to add the logs using single call to backend RDBMS. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27289) Check for proxy hosts with subnet in IP results in exception

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27289.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Merged to master.

> Check for proxy hosts with subnet in IP results in exception 
> -
>
> Key: HIVE-27289
> URL: https://issues.apache.org/jira/browse/HIVE-27289
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
>Affects Versions: 4.0.0-alpha-2
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> When running schematool for sysdb setup in a kerberized environment, the 
> check to see if the user is a super user fails when the 
> hadoop.proxyuser.hive.hosts contains a subnet in IPAddresses. (for example: 
> 192.168.0.3/23)
> {noformat}
> exec 
> /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p11.35002917/lib/hadoop/bin/hadoop 
> jar 
> /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p11.35002917/lib/hive/lib/hive-cli-3.1.3000.7.1.8.11-3.jar
>  org.apache.hive.beeline.schematool.HiveSchemaTool -verbose -dbType hive 
> -metaDbType mysql -initOrUpgradeSchema
> WARNING: Use "yarn jar" to launch YARN applications.
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p11.35002917/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p11.35002917/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = 93d863d8-cbe6-47fc-8817-49074841f9f1
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by 
> org.apache.hadoop.hive.common.StringInternUtils 
> (file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p11.35002917/jars/hive-common-3.1.3000.7.1.8.11-3.jar)
>  to field java.net.URI.string
> WARNING: Please consider reporting this to the maintainers of 
> org.apache.hadoop.hive.common.StringInternUtils
> WARNING: Use --illegal-access=warn to enable warnings of further illegal 
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> 23/04/22 12:01:38 ERROR metastore.HiveMetaStoreAuthorizer: [main]: 
> HiveMetaStoreAuthorizer.onEvent(): failed
> java.lang.IllegalArgumentException: Could not parse []
>   at 
> org.apache.commons.net.util.SubnetUtils.toInteger(SubnetUtils.java:287) 
> ~[commons-net-3.6.jar:3.6]
>   at 
> org.apache.commons.net.util.SubnetUtils.access$400(SubnetUtils.java:27) 
> ~[commons-net-3.6.jar:3.6]
>   at 
> org.apache.commons.net.util.SubnetUtils$SubnetInfo.isInRange(SubnetUtils.java:125)
>  ~[commons-net-3.6.jar:3.6]
>   at org.apache.hadoop.util.MachineList.includes(MachineList.java:155) 
> ~[hadoop-common-3.1.1.7.1.8.11-3.jar:?]
>   at 
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.checkUserHasHostProxyPrivileges(MetaStoreUtils.java:1347)
>  ~[hive-standalone-metastore-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
>   at 
> org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.isSuperUser(HiveMetaStoreAuthorizer.java:495)
>  ~[hive-exec-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
>   at 
> org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.skipAuthorization(HiveMetaStoreAuthorizer.java:558)
>  ~[hive-exec-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
>   at 
> org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.onEvent(HiveMetaStoreAuthorizer.java:104)
>  [hive-exec-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:4026)
>  [hive-standalone-metastore-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database_req(HiveMetaStore.java:1695)
>  [hive-standalone-metastore-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
>   at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:?]
>   at 
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  ~[?:?]
>   at 
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:?]
>   at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
>   at 
> 

[jira] [Created] (HIVE-27289) Check for proxy hosts with subnet in IP results in exception

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27289:


 Summary: Check for proxy hosts with subnet in IP results in 
exception 
 Key: HIVE-27289
 URL: https://issues.apache.org/jira/browse/HIVE-27289
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Affects Versions: 4.0.0-alpha-2
Reporter: Naveen Gangam
Assignee: Naveen Gangam


When running schematool for sysdb setup in a kerberized environment, the check 
to see if the user is a super user fails when the hadoop.proxyuser.hive.hosts 
contains a subnet in IPAddresses. (for example: 192.168.0.3/23)

{noformat}
exec 
/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p11.35002917/lib/hadoop/bin/hadoop 
jar 
/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p11.35002917/lib/hive/lib/hive-cli-3.1.3000.7.1.8.11-3.jar
 org.apache.hive.beeline.schematool.HiveSchemaTool -verbose -dbType hive 
-metaDbType mysql -initOrUpgradeSchema
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p11.35002917/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p11.35002917/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 93d863d8-cbe6-47fc-8817-49074841f9f1
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by 
org.apache.hadoop.hive.common.StringInternUtils 
(file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p11.35002917/jars/hive-common-3.1.3000.7.1.8.11-3.jar)
 to field java.net.URI.string
WARNING: Please consider reporting this to the maintainers of 
org.apache.hadoop.hive.common.StringInternUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal 
reflective access operations
WARNING: All illegal access operations will be denied in a future release
23/04/22 12:01:38 ERROR metastore.HiveMetaStoreAuthorizer: [main]: 
HiveMetaStoreAuthorizer.onEvent(): failed
java.lang.IllegalArgumentException: Could not parse []
at 
org.apache.commons.net.util.SubnetUtils.toInteger(SubnetUtils.java:287) 
~[commons-net-3.6.jar:3.6]
at 
org.apache.commons.net.util.SubnetUtils.access$400(SubnetUtils.java:27) 
~[commons-net-3.6.jar:3.6]
at 
org.apache.commons.net.util.SubnetUtils$SubnetInfo.isInRange(SubnetUtils.java:125)
 ~[commons-net-3.6.jar:3.6]
at org.apache.hadoop.util.MachineList.includes(MachineList.java:155) 
~[hadoop-common-3.1.1.7.1.8.11-3.jar:?]
at 
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.checkUserHasHostProxyPrivileges(MetaStoreUtils.java:1347)
 ~[hive-standalone-metastore-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.isSuperUser(HiveMetaStoreAuthorizer.java:495)
 ~[hive-exec-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.skipAuthorization(HiveMetaStoreAuthorizer.java:558)
 ~[hive-exec-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.onEvent(HiveMetaStoreAuthorizer.java:104)
 [hive-exec-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:4026)
 [hive-standalone-metastore-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database_req(HiveMetaStore.java:1695)
 [hive-standalone-metastore-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:?]
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 ~[?:?]
at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
 [hive-standalone-metastore-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
 [hive-standalone-metastore-3.1.3000.7.1.8.11-3.jar:3.1.3000.7.1.8.11-3]
at com.sun.proxy.$Proxy43.get_database_req(Unknown Source) [?:?]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1946)
 

[jira] [Commented] (HIVE-27262) Hive metastore changelog for Authorization operations

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-27262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715107#comment-17715107
 ] 

Naveen Gangam commented on HIVE-27262:
--

[~bharos92] HMS does not natively have a separate audit log. Because, this can 
be achieved by adding a custom NotificationListener that would get notified on 
DDL changes which it could then write out to a separate audit log.
I do not think we have notifications for grant/revoke operations though. Are 
you interested in audit the metadata changes or just grant/revoke specificallly.

> Hive metastore changelog for Authorization operations
> -
>
> Key: HIVE-27262
> URL: https://issues.apache.org/jira/browse/HIVE-27262
> Project: Hive
>  Issue Type: New Feature
>  Components: Standalone Metastore
>Affects Versions: 3.1.2, 4.0.0-alpha-2
>Reporter: Bharath Krishna
>Priority: Major
>
> IIUC, Hive metastore doesn't provide the changelog (NOTIFICATION_LOG) for the 
> authorization operations like GRANT , REVOKE etc..
> I also assume in this case the Hive Replication doesn't replicate these 
> events as they are not recorded as metastore events.
>  
> Is there any reason these events are not captured, or is it just a missing 
> feature ?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-25073) Optimise HiveAlterHandler::alterPartitions

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-25073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-25073.
--
Resolution: Cannot Reproduce

Closing based on the comment above.

> Optimise HiveAlterHandler::alterPartitions
> --
>
> Key: HIVE-25073
> URL: https://issues.apache.org/jira/browse/HIVE-25073
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Harshit Gupta
>Priority: Major
>
> Table details are populated again and again for each partition, which can be 
> avoided.
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L5892
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java#L808
> Following stacktrace may be relevant for apache master as well.
> {noformat}
>   at org.datanucleus.store.query.Query.executeWithArray(Query.java:1744)
>   at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:368)
>   at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2113)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2152)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4951)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitions(ObjectStore.java:5057)
>   at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
>   at com.sun.proxy.$Proxy27.alterPartitions(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:798)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:5695)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_req(HiveMetaStore.java:5647)
>   at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
>   at com.sun.proxy.$Proxy28.alter_partitions_req(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:18557)
>   at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:18541)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
>   at 
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-25707) SchemaTool may leave the metastore in-between upgrade steps

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-25707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-25707.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

A patch has been merged that turns off the autocommit when running the 
schematool.
Unfortunately, for Oracle and MySQL, there can be no transactional boundaries 
for DDLs. They are autocommitted as they are run. They cannot be rolled back.

So the patch works for postgres. It wasnt tested for MSSQL.

> SchemaTool may leave the metastore in-between upgrade steps
> ---
>
> Key: HIVE-25707
> URL: https://issues.apache.org/jira/browse/HIVE-25707
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zhihua Deng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> it seems like:
> * schematool runs the sql files via beeline
> * autocommit is turned on
> * pressing ctrl+c or killing the process will result in an invalid schema
> https://github.com/apache/hive/blob/6e02f6164385a370ee8014c795bee1fa423d7937/beeline/src/java/org/apache/hive/beeline/schematool/HiveSchemaTool.java#L79



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-25707) SchemaTool may leave the metastore in-between upgrade steps

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-25707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-25707:


Assignee: Zhihua Deng

> SchemaTool may leave the metastore in-between upgrade steps
> ---
>
> Key: HIVE-25707
> URL: https://issues.apache.org/jira/browse/HIVE-25707
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zhihua Deng
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> it seems like:
> * schematool runs the sql files via beeline
> * autocommit is turned on
> * pressing ctrl+c or killing the process will result in an invalid schema
> https://github.com/apache/hive/blob/6e02f6164385a370ee8014c795bee1fa423d7937/beeline/src/java/org/apache/hive/beeline/schematool/HiveSchemaTool.java#L79



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27173) Add method for Spark to be able to trigger DML events

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27173.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. 

> Add method for Spark to be able to trigger DML events
> -
>
> Key: HIVE-27173
> URL: https://issues.apache.org/jira/browse/HIVE-27173
> Project: Hive
>  Issue Type: Improvement
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Spark currently uses Hive.java from Hive as a convenient way to hide from the 
> having to deal with HMS Client and the thrift objects. Currently, Hive has 
> support for DML events (being able to generate events on DML operations but 
> does not expose a public method to do so). It has a private method that takes 
> in Hive objects like Table etc. Would be nice if we can have something with 
> more primitive datatypes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-27173) Add method for Spark to be able to trigger DML events

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-27173:


Assignee: Naveen Gangam

> Add method for Spark to be able to trigger DML events
> -
>
> Key: HIVE-27173
> URL: https://issues.apache.org/jira/browse/HIVE-27173
> Project: Hive
>  Issue Type: Improvement
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Spark currently uses Hive.java from Hive as a convenient way to hide from the 
> having to deal with HMS Client and the thrift objects. Currently, Hive has 
> support for DML events (being able to generate events on DML operations but 
> does not expose a public method to do so). It has a private method that takes 
> in Hive objects like Table etc. Would be nice if we can have something with 
> more primitive datatypes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27147) HS2 is not accessible to clients via zookeeper when hostname used is not FQDN

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-27147.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Merged to master. Thank you. for the fix [~VenuReddy] and [~dengzh] for the 
review.

> HS2 is not accessible to clients via zookeeper when hostname used is not FQDN
> -
>
> Key: HIVE-27147
> URL: https://issues.apache.org/jira/browse/HIVE-27147
> Project: Hive
>  Issue Type: Bug
>Reporter: Venugopal Reddy K
>Assignee: Venugopal Reddy K
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> HS2 is not accessible to clients via zookeeper when hostname used during 
> registration is InetAddress.getHostName() with JDK 11. This issue is 
> happening due to change in behavior on JDK 11 and it is OS specific - 
> [https://stackoverflow.com/questions/61898627/inetaddress-getlocalhost-gethostname-different-behavior-between-jdk-11-and-j|http://example.com/]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-27147) HS2 is not accessible to clients via zookeeper when hostname used is not FQDN

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-27147:


Assignee: Venugopal Reddy K

> HS2 is not accessible to clients via zookeeper when hostname used is not FQDN
> -
>
> Key: HIVE-27147
> URL: https://issues.apache.org/jira/browse/HIVE-27147
> Project: Hive
>  Issue Type: Bug
>Reporter: Venugopal Reddy K
>Assignee: Venugopal Reddy K
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> HS2 is not accessible to clients via zookeeper when hostname used during 
> registration is InetAddress.getHostName() with JDK 11. This issue is 
> happening due to change in behavior on JDK 11 and it is OS specific - 
> [https://stackoverflow.com/questions/61898627/inetaddress-getlocalhost-gethostname-different-behavior-between-jdk-11-and-j|http://example.com/]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-27163) Column stats are not getting published after an insert query into an external table with custom location

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-27163:


Assignee: Zhihua Deng

> Column stats are not getting published after an insert query into an external 
> table with custom location
> 
>
> Key: HIVE-27163
> URL: https://issues.apache.org/jira/browse/HIVE-27163
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Zhihua Deng
>Priority: Major
>
> Test case details are below
> *test.q*
> {noformat}
> set hive.stats.column.autogather=true;
> set hive.stats.autogather=true;
> dfs ${system:test.dfs.mkdir} ${system:test.tmp.dir}/test;
> create external table test_custom(age int, name string) stored as orc 
> location '/tmp/test';
> insert into test_custom select 1, 'test';
> desc formatted test_custom age;{noformat}
> *test.q.out*
>  
>  
> {noformat}
>  A masked pattern was here 
> PREHOOK: type: CREATETABLE
>  A masked pattern was here 
> PREHOOK: Output: database:default
> PREHOOK: Output: default@test_custom
>  A masked pattern was here 
> POSTHOOK: type: CREATETABLE
>  A masked pattern was here 
> POSTHOOK: Output: database:default
> POSTHOOK: Output: default@test_custom
> PREHOOK: query: insert into test_custom select 1, 'test'
> PREHOOK: type: QUERY
> PREHOOK: Input: _dummy_database@_dummy_table
> PREHOOK: Output: default@test_custom
> POSTHOOK: query: insert into test_custom select 1, 'test'
> POSTHOOK: type: QUERY
> POSTHOOK: Input: _dummy_database@_dummy_table
> POSTHOOK: Output: default@test_custom
> POSTHOOK: Lineage: test_custom.age SIMPLE []
> POSTHOOK: Lineage: test_custom.name SIMPLE []
> PREHOOK: query: desc formatted test_custom age
> PREHOOK: type: DESCTABLE
> PREHOOK: Input: default@test_custom
> POSTHOOK: query: desc formatted test_custom age
> POSTHOOK: type: DESCTABLE
> POSTHOOK: Input: default@test_custom
> col_name                age
> data_type               int
> min
> max
> num_nulls
> distinct_count
> avg_col_len
> max_col_len
> num_trues
> num_falses
> bit_vector
> comment                 from deserializer{noformat}
> As we can see from desc formatted output, column stats were not populated
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-26999) Upgrade MySQL Connector Java due to security CVEs

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702794#comment-17702794
 ] 

Naveen Gangam commented on HIVE-26999:
--

Merged to master. Closing the jira. THank you for the contribution 
[~devaspatikrishnatri]

> Upgrade MySQL Connector Java  due to security CVEs
> --
>
> Key: HIVE-26999
> URL: https://issues.apache.org/jira/browse/HIVE-26999
> Project: Hive
>  Issue Type: Task
>Reporter: Devaspati Krishnatri
>Assignee: Devaspati Krishnatri
>Priority: Major
>  Labels: pull-request-available
> Attachments: tree.txt
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> The following CVEs impact older versions of [MySQL Connector 
> Java|https://mvnrepository.com/artifact/mysql/mysql-connector-java]
>  * *CVE-2021-3711* : Critical  - Impacts all versions up to (including) 
> 8.0.27 (ref:  [https://nvd.nist.gov/vuln/detail/CVE-2021-3711])
>  * *CVE-2021-3712* - High - Impacts all versions up to (including) 8.0.27 
> (ref: 
> [https://nvd.nist.gov/vuln/detail/CVE-2021-37112)|https://nvd.nist.gov/vuln/detail/CVE-2021-3711]
>  * *CVE-2021-44531* - High - Impacts all versions up to (including) 8.0.28 
> (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-44531])
>  * *CVE-2022-21824* - High - Impacts all versions up to (including) 8.0.28 
> (ref:[https://nvd.nist.gov/vuln/detail/CVE-2022-21824)]
> Recommendation: *Upgrade* [*MySQL Connector 
> Java*|https://mvnrepository.com/artifact/mysql/mysql-connector-java]  *to*  
> [*8.0.31*|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.31]
>  *or above*



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26999) Upgrade MySQL Connector Java due to security CVEs

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26999.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

> Upgrade MySQL Connector Java  due to security CVEs
> --
>
> Key: HIVE-26999
> URL: https://issues.apache.org/jira/browse/HIVE-26999
> Project: Hive
>  Issue Type: Task
>Reporter: Devaspati Krishnatri
>Assignee: Devaspati Krishnatri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: tree.txt
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> The following CVEs impact older versions of [MySQL Connector 
> Java|https://mvnrepository.com/artifact/mysql/mysql-connector-java]
>  * *CVE-2021-3711* : Critical  - Impacts all versions up to (including) 
> 8.0.27 (ref:  [https://nvd.nist.gov/vuln/detail/CVE-2021-3711])
>  * *CVE-2021-3712* - High - Impacts all versions up to (including) 8.0.27 
> (ref: 
> [https://nvd.nist.gov/vuln/detail/CVE-2021-37112)|https://nvd.nist.gov/vuln/detail/CVE-2021-3711]
>  * *CVE-2021-44531* - High - Impacts all versions up to (including) 8.0.28 
> (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-44531])
>  * *CVE-2022-21824* - High - Impacts all versions up to (including) 8.0.28 
> (ref:[https://nvd.nist.gov/vuln/detail/CVE-2022-21824)]
> Recommendation: *Upgrade* [*MySQL Connector 
> Java*|https://mvnrepository.com/artifact/mysql/mysql-connector-java]  *to*  
> [*8.0.31*|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.31]
>  *or above*



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-25032) Optimise PartitionManagementTask

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-25032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-25032:


Assignee: Zhihua Deng

> Optimise PartitionManagementTask
> 
>
> Key: HIVE-25032
> URL: https://issues.apache.org/jira/browse/HIVE-25032
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Zhihua Deng
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> When large number of tables and dbs are present, it takes lot of time to 
> iterate over all tables in PartitionManagementTask.
> If table parameters are populated in TableMeta, it will cut down lots of 
> interactions with HMS in PartitionManagementTask.
>  
> Code Ref:
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java#L126]
>  
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java#L132]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27063) LDAP+JWT auth forms not supported

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-27063:
-
Description: 
In HIVE-25875, support for multiple authentication forms was added for Hive 
Server. In HIVE-25575, support for JWT authentication was added. However, 
setting hive.server2.authentication="JWT,LDAP" will fail with the following 
validation error.


{noformat}
<12>1 2023-02-03T09:32:11.018Z hiveserver2-0 hiveserver2 1 
0393cf91-48f7-49e3-b2b1-b983000d4cd6 [mdc@18060 class="server.HiveServer2" 
level="WARN" thread="main"] Error starting HiveServer2 on attempt 2, will retry 
in 6ms\rorg.apache.hive.service.ServiceException: Failed to Start 
HiveServer2
at org.apache.hive.service.CompositeService.start(CompositeService.java:80) 
at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:692) 
at 
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1154)
 
at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:145) 
at 
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1503)
 
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1316) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) 
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 
at java.base/java.lang.reflect.Method.invoke(Method.java:566)\r at 
org.apache.hadoop.util.RunJar.run(RunJar.java:318) 
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)\rCaused by: 
java.lang.RuntimeException: Failed to init HttpServer 
at 
org.apache.hive.service.cli.thrift.ThriftHttpCLIService.initServer(ThriftHttpCLIService.java:239)
 
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.start(ThriftCLIService.java:235)
 
at org.apache.hive.service.CompositeService.start(CompositeService.java:70) ... 
11 more
Caused by: java.lang.Exception: The authentication types have conflicts: 
LDAP,JWT 
at org.apache.hive.service.auth.AuthType.verifyTypes(AuthType.java:69) 
at org.apache.hive.service.auth.AuthType.(AuthType.java:43) 
org.apache.hive.service.cli.thrift.ThriftHttpServlet.(ThriftHttpServlet.java:124)
 
at 
org.apache.hive.service.cli.thrift.ThriftHttpCLIService.initServer(ThriftHttpCLIService.java:197)
 ... 13 more
{noformat}

We never fixed the AuthType.validateTypes() to support this.

  was:
In HIVE-25875, support for multiple authentication forms was added for Hive 
Server. In HIVE-25575, support for JWT authentication was added. However, 
setting hive.server2.authentication="JWT,LDAP" will fail with the following 
validation error.


{noformat}
<12>1 2023-02-03T09:32:11.018Z hiveserver2-0 hiveserver2 1 
0393cf91-48f7-49e3-b2b1-b983000d4cd6 [mdc@18060 class="server.HiveServer2" 
level="WARN" thread="main"] Error starting HiveServer2 on attempt 2, will retry 
in 6ms\rorg.apache.hive.service.ServiceException: Failed to Start 
HiveServer2\r at 
org.apache.hive.service.CompositeService.start(CompositeService.java:80)\r at 
org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:692)\r at 
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1154)\r
 at 
org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:145)\r 
at 
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1503)\r
 at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1316)\r at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)\r at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r
 at java.base/java.lang.reflect.Method.invoke(Method.java:566)\r at 
org.apache.hadoop.util.RunJar.run(RunJar.java:318)\r at 
org.apache.hadoop.util.RunJar.main(RunJar.java:232)\rCaused by: 
java.lang.RuntimeException: Failed to init HttpServer\r at 
org.apache.hive.service.cli.thrift.ThriftHttpCLIService.initServer(ThriftHttpCLIService.java:239)\r
 at 
org.apache.hive.service.cli.thrift.ThriftCLIService.start(ThriftCLIService.java:235)\r
 at org.apache.hive.service.CompositeService.start(CompositeService.java:70)\r 
... 11 more\rCaused by: java.lang.Exception: The authentication types have 
conflicts: LDAP,JWT\r at 
org.apache.hive.service.auth.AuthType.verifyTypes(AuthType.java:69)\r at 
org.apache.hive.service.auth.AuthType.(AuthType.java:43)\r at 
org.apache.hive.service.cli.thrift.ThriftHttpServlet.(ThriftHttpServlet.java:124)\r
 at 
org.apache.hive.service.cli.thrift.ThriftHttpCLIService.initServer(ThriftHttpCLIService.java:197)\r
 ... 13 more\r
{noformat}

We never fixed the AuthType.validateTypes() to support 

[jira] [Assigned] (HIVE-27063) LDAP+JWT auth forms not supported

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-27063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-27063:


Assignee: Naveen Gangam

> LDAP+JWT auth forms not supported
> -
>
> Key: HIVE-27063
> URL: https://issues.apache.org/jira/browse/HIVE-27063
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>
> In HIVE-25875, support for multiple authentication forms was added for Hive 
> Server. In HIVE-25575, support for JWT authentication was added. However, 
> setting hive.server2.authentication="JWT,LDAP" will fail with the following 
> validation error.
> {noformat}
> <12>1 2023-02-03T09:32:11.018Z hiveserver2-0 hiveserver2 1 
> 0393cf91-48f7-49e3-b2b1-b983000d4cd6 [mdc@18060 class="server.HiveServer2" 
> level="WARN" thread="main"] Error starting HiveServer2 on attempt 2, will 
> retry in 6ms\rorg.apache.hive.service.ServiceException: Failed to Start 
> HiveServer2\r at 
> org.apache.hive.service.CompositeService.start(CompositeService.java:80)\r at 
> org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:692)\r at 
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1154)\r
>  at 
> org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:145)\r
>  at 
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1503)\r
>  at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1316)\r 
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)\r at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r
>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r
>  at java.base/java.lang.reflect.Method.invoke(Method.java:566)\r at 
> org.apache.hadoop.util.RunJar.run(RunJar.java:318)\r at 
> org.apache.hadoop.util.RunJar.main(RunJar.java:232)\rCaused by: 
> java.lang.RuntimeException: Failed to init HttpServer\r at 
> org.apache.hive.service.cli.thrift.ThriftHttpCLIService.initServer(ThriftHttpCLIService.java:239)\r
>  at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.start(ThriftCLIService.java:235)\r
>  at 
> org.apache.hive.service.CompositeService.start(CompositeService.java:70)\r 
> ... 11 more\rCaused by: java.lang.Exception: The authentication types have 
> conflicts: LDAP,JWT\r at 
> org.apache.hive.service.auth.AuthType.verifyTypes(AuthType.java:69)\r at 
> org.apache.hive.service.auth.AuthType.(AuthType.java:43)\r at 
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.(ThriftHttpServlet.java:124)\r
>  at 
> org.apache.hive.service.cli.thrift.ThriftHttpCLIService.initServer(ThriftHttpCLIService.java:197)\r
>  ... 13 more\r
> {noformat}
> We never fixed the AuthType.validateTypes() to support this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26035.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

> Explore moving to directsql for ObjectStore::addPartitions
> --
>
> Key: HIVE-26035
> URL: https://issues.apache.org/jira/browse/HIVE-26035
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Venugopal Reddy K
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> Currently {{addPartitions}} uses datanuclues and is super slow for large 
> number of partitions. It will be good to move to direct sql. Lots of repeated 
> SQLs can be avoided as well (e.g SDS, SERDE, TABLE_PARAMS)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-26035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683595#comment-17683595
 ] 

Naveen Gangam commented on HIVE-26035:
--

[~VenuReddy] The patch has been merged to master. Could you please post the 
performance improvements from the microbenchmarks that were done to this jira. 
It would be useful for future. Thank you

> Explore moving to directsql for ObjectStore::addPartitions
> --
>
> Key: HIVE-26035
> URL: https://issues.apache.org/jira/browse/HIVE-26035
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Venugopal Reddy K
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> Currently {{addPartitions}} uses datanuclues and is super slow for large 
> number of partitions. It will be good to move to direct sql. Lots of repeated 
> SQLs can be avoided as well (e.g SDS, SERDE, TABLE_PARAMS)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26681) Upgrade dom4j: flexible XML framework for Java to safe version due to critical CVEs

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26681.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. Thank you for the patch [~devaspatikrishnatri] 
and review [~cnauroth]

> Upgrade dom4j: flexible XML framework for Java to safe version due to 
> critical CVEs
> ---
>
> Key: HIVE-26681
> URL: https://issues.apache.org/jira/browse/HIVE-26681
> Project: Hive
>  Issue Type: Task
>Reporter: Devaspati Krishnatri
>Assignee: Devaspati Krishnatri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26860) Appropriate rows in HMS datastore tables(SDS, SERDES, SERDE_PARAMS, SKEWED_COL_NAMES, SKEWED_COL_VALUE_LOC_MAP, SKEWED_VALUES) are not deleted upon drop partition table

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26860.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. Closing the jira. Thank you for the patch 
[~VenuReddy] and review [~dengzh]

> Appropriate rows in HMS datastore tables(SDS, SERDES, SERDE_PARAMS, 
> SKEWED_COL_NAMES, SKEWED_COL_VALUE_LOC_MAP, SKEWED_VALUES) are not deleted 
> upon drop partition table with skewed columns
> 
>
> Key: HIVE-26860
> URL: https://issues.apache.org/jira/browse/HIVE-26860
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Metastore
>Reporter: Venugopal Reddy K
>Assignee: Venugopal Reddy K
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: partdata3
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> *[Description]*
> Appropriate rows in HMS backing datastore tables(SDS, SERDES, SERDE_PARAMS, 
> SKEWED_COL_NAMES, SKEWED_COL_VALUE_LOC_MAP, SKEWED_VALUES) are not deleted 
> upon drop partition table with skewed columns due to class cast exception.
> Exception Stack:
> {code:java}
> MetaStoreDirectSql.java:2690) at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$2000(MetaStoreDirectSql.java:118)
>  at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql$9.run(MetaStoreDirectSql.java:2608)
>  at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:74) 
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.dropPartitionsViaSqlFilter(MetaStoreDirectSql.java:2598)
>  at 
> org.apache.hadoop.hive.metastore.ObjectStore$6.getSqlResult(ObjectStore.java:3053)
> 2022-12-30T10:29:12,490 DEBUG [HiveServer2-Background-Pool: Thread-292] 
> metastore.ObjectStore: Full DirectSQL callstack for debugging (not an error)
> java.lang.ClassCastException: java.lang.Long cannot be cast to 
> [Ljava.lang.Object;
>     at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.dropStorageDescriptors(MetaStoreDirectSql.java:2724)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.dropPartitionsByPartitionIds(MetaStoreDirectSql.java:2690)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$2000(MetaStoreDirectSql.java:118)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql$9.run(MetaStoreDirectSql.java:2608)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:74) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.dropPartitionsViaSqlFilter(MetaStoreDirectSql.java:2598)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.ObjectStore$6.getSqlResult(ObjectStore.java:3053)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.ObjectStore$6.getSqlResult(ObjectStore.java:3050)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4352)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsInternal(ObjectStore.java:3061)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitions(ObjectStore.java:3040)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_292]
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_292]
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_292]
>     at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
>     at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at com.sun.proxy.$Proxy29.dropPartitions(Unknown Source) ~[?:?]
>     at 
> org.apache.hadoop.hive.metastore.HMSHandler.dropPartitionsAndGetLocations(HMSHandler.java:3174)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.HMSHandler.drop_table_core(HMSHandler.java:2951)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>     at 
> org.apache.hadoop.hive.metastore.HMSHandler.drop_table_with_environment_context(HMSHandler.java:3199)
>  

[jira] [Updated] (HIVE-26537) Deprecate older APIs in the HMS

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26537:
-
Priority: Critical  (was: Major)

> Deprecate older APIs in the HMS
> ---
>
> Key: HIVE-26537
> URL: https://issues.apache.org/jira/browse/HIVE-26537
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Critical
>  Labels: pull-request-available
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> This Jira is to track the clean-up(deprecate older APIs and point the HMS 
> client to the newer APIs) work in the hive metastore server.
> More details will be added here soon.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26537) Deprecate older APIs in the HMS

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26537:
-
Labels: hive-4.0.0-must pull-request-available  (was: 
pull-request-available)

> Deprecate older APIs in the HMS
> ---
>
> Key: HIVE-26537
> URL: https://issues.apache.org/jira/browse/HIVE-26537
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Critical
>  Labels: hive-4.0.0-must, pull-request-available
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> This Jira is to track the clean-up(deprecate older APIs and point the HMS 
> client to the newer APIs) work in the hive metastore server.
> More details will be added here soon.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26154) Upgrade cron-utils to 9.1.6 for branch-3

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26154:
-
Labels: hive-3.2.0-must pull-request-available  (was: 
pull-request-available)

> Upgrade cron-utils to 9.1.6 for branch-3
> 
>
> Key: HIVE-26154
> URL: https://issues.apache.org/jira/browse/HIVE-26154
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Affects Versions: 3.1.3
>Reporter: Asif Saleh
>Priority: Major
>  Labels: hive-3.2.0-must, pull-request-available
> Fix For: 3.1.3
>
>
> To fix [CVE-2021-41269|https://nvd.nist.gov/vuln/detail/CVE-2021-41269] issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-26154) Upgrade cron-utils to 9.1.6 for branch-3

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-26154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644912#comment-17644912
 ] 

Naveen Gangam commented on HIVE-26154:
--

I think we should

> Upgrade cron-utils to 9.1.6 for branch-3
> 
>
> Key: HIVE-26154
> URL: https://issues.apache.org/jira/browse/HIVE-26154
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Affects Versions: 3.1.3
>Reporter: Asif Saleh
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.1.3
>
>
> To fix [CVE-2021-41269|https://nvd.nist.gov/vuln/detail/CVE-2021-41269] issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26753) Upgrade Apache Ivy to 2.5.1 due to CVE-2022-37865, CVE-2022-37866

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26753.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. Thank you for the patch [~devaspatikrishnatri]

> Upgrade Apache Ivy to 2.5.1 due to CVE-2022-37865, CVE-2022-37866
> -
>
> Key: HIVE-26753
> URL: https://issues.apache.org/jira/browse/HIVE-26753
> Project: Hive
>  Issue Type: Task
>Reporter: Devaspati Krishnatri
>Assignee: Devaspati Krishnatri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26701) Enable metrics for Database connection pools(1 & 2) used by ObjectStore in HMS

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26701.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. Thank you for the patch [~dengzh]

> Enable metrics for Database connection pools(1 & 2) used by ObjectStore in HMS
> --
>
> Key: HIVE-26701
> URL: https://issues.apache.org/jira/browse/HIVE-26701
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Zhihua Deng
>Priority: Major
>  Labels: hive-4.0.0-must, pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> We have metrics enabled for database connection pools(3 & 4) used in 
> TxnHandler. We don't have the same for pools(1 & 2) used by ObjectStore



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26751) Bug Fixes and Improvements for 3.2.0 release

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26751:
-
Labels: hive-3.2.0-must release-3.2.0  (was: release-3.2.0)

> Bug Fixes and Improvements for 3.2.0 release
> 
>
> Key: HIVE-26751
> URL: https://issues.apache.org/jira/browse/HIVE-26751
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Major
>  Labels: hive-3.2.0-must, release-3.2.0
>
> Creating subtask to track the bug fixes that will go as part of 3.2.0 release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26751) Bug Fixes and Improvements for 3.2.0 release

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26751:
-
Priority: Blocker  (was: Major)

> Bug Fixes and Improvements for 3.2.0 release
> 
>
> Key: HIVE-26751
> URL: https://issues.apache.org/jira/browse/HIVE-26751
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Blocker
>  Labels: hive-3.2.0-must, release-3.2.0
>
> Creating subtask to track the bug fixes that will go as part of 3.2.0 release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-26751) Bug Fixes and Improvements for 3.2.0 release

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-26751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17635363#comment-17635363
 ] 

Naveen Gangam commented on HIVE-26751:
--

[~amanraj2520] How are we tracking the sub-jiras for this release? I noticed 
the label release-3.2.0 in this jira but the sub-jiras do not contain the same 
label.
I created this jira board for Hive Releases. Would be nice if we could track 
them from a common dashboard. You should have permissions to edit this but I 
can do so as well, if you want me to. I just need to know what labels you want 
to use for all items for this release.
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=564=2571

> Bug Fixes and Improvements for 3.2.0 release
> 
>
> Key: HIVE-26751
> URL: https://issues.apache.org/jira/browse/HIVE-26751
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Major
>  Labels: release-3.2.0
>
> Creating subtask to track the bug fixes that will go as part of 3.2.0 release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26701) Enable metrics for Database connection pools(1 & 2) used by ObjectStore in HMS

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26701:
-
Labels: hive-4.0.0-must  (was: )

> Enable metrics for Database connection pools(1 & 2) used by ObjectStore in HMS
> --
>
> Key: HIVE-26701
> URL: https://issues.apache.org/jira/browse/HIVE-26701
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Zhihua Deng
>Priority: Major
>  Labels: hive-4.0.0-must
>
> We have metrics enabled for database connection pools(3 & 4) used in 
> TxnHandler. We don't have the same for pools(1 & 2) used by ObjectStore



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-26220) Shade & relocate dependencies in hive-exec to avoid conflicting with downstream projects

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-26220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631412#comment-17631412
 ] 

Naveen Gangam commented on HIVE-26220:
--

[~ayushtkn] I have marked this jira a must-fix for 4.0.0. Would you have cycles 
to take a look at this? Thank you

> Shade & relocate dependencies in hive-exec to avoid conflicting with 
> downstream projects
> 
>
> Key: HIVE-26220
> URL: https://issues.apache.org/jira/browse/HIVE-26220
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 4.0.0, 4.0.0-alpha-1
>Reporter: Chao Sun
>Priority: Blocker
>  Labels: hive-4.0.0-must
>
> Currently projects like Spark, Trino/Presto, Iceberg, etc, are depending on 
> {{hive-exec:core}} which was removed in HIVE-25531. The reason these projects 
> use {{hive-exec:core}} is because they have the flexibility to exclude, shade 
> & relocate dependencies in {{hive-exec}} that conflict with the ones they 
> brought in by themselves. However, with {{hive-exec}} this is no longer 
> possible, since it is a fat jar that shade those dependencies but do not 
> relocate many of them.
> In order for the downstream projects to consume {{hive-exec}}, we will need 
> to make sure all the dependencies in {{hive-exec}} are properly shaded and 
> relocated, so they won't cause conflicts with those from the downstream.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26220) Shade & relocate dependencies in hive-exec to avoid conflicting with downstream projects

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26220:
-
Target Version/s: 4.0.0

> Shade & relocate dependencies in hive-exec to avoid conflicting with 
> downstream projects
> 
>
> Key: HIVE-26220
> URL: https://issues.apache.org/jira/browse/HIVE-26220
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 4.0.0, 4.0.0-alpha-1
>Reporter: Chao Sun
>Priority: Blocker
>  Labels: hive-4.0.0-must
>
> Currently projects like Spark, Trino/Presto, Iceberg, etc, are depending on 
> {{hive-exec:core}} which was removed in HIVE-25531. The reason these projects 
> use {{hive-exec:core}} is because they have the flexibility to exclude, shade 
> & relocate dependencies in {{hive-exec}} that conflict with the ones they 
> brought in by themselves. However, with {{hive-exec}} this is no longer 
> possible, since it is a fat jar that shade those dependencies but do not 
> relocate many of them.
> In order for the downstream projects to consume {{hive-exec}}, we will need 
> to make sure all the dependencies in {{hive-exec}} are properly shaded and 
> relocated, so they won't cause conflicts with those from the downstream.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26220) Shade & relocate dependencies in hive-exec to avoid conflicting with downstream projects

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26220:
-
Labels: hive-4.0.0-must  (was: )

> Shade & relocate dependencies in hive-exec to avoid conflicting with 
> downstream projects
> 
>
> Key: HIVE-26220
> URL: https://issues.apache.org/jira/browse/HIVE-26220
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 4.0.0, 4.0.0-alpha-1
>Reporter: Chao Sun
>Priority: Blocker
>  Labels: hive-4.0.0-must
>
> Currently projects like Spark, Trino/Presto, Iceberg, etc, are depending on 
> {{hive-exec:core}} which was removed in HIVE-25531. The reason these projects 
> use {{hive-exec:core}} is because they have the flexibility to exclude, shade 
> & relocate dependencies in {{hive-exec}} that conflict with the ones they 
> brought in by themselves. However, with {{hive-exec}} this is no longer 
> possible, since it is a fat jar that shade those dependencies but do not 
> relocate many of them.
> In order for the downstream projects to consume {{hive-exec}}, we will need 
> to make sure all the dependencies in {{hive-exec}} are properly shaded and 
> relocated, so they won't cause conflicts with those from the downstream.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26640) Upgrade JUnit to 4.13.2 due to medium CVEs

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26640.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Thanks for the patch [~devaspatikrishnatri]. Fix has been merged to master.

> Upgrade JUnit to 4.13.2 due to medium CVEs
> --
>
> Key: HIVE-26640
> URL: https://issues.apache.org/jira/browse/HIVE-26640
> Project: Hive
>  Issue Type: Task
>Reporter: Devaspati Krishnatri
>Assignee: Devaspati Krishnatri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26400) Provide docker images for Hive

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26400:
-
Labels: hive-4.0.0-must pull-request-available  (was: 
pull-request-available)

> Provide docker images for Hive
> --
>
> Key: HIVE-26400
> URL: https://issues.apache.org/jira/browse/HIVE-26400
> Project: Hive
>  Issue Type: Improvement
>  Components: Build Infrastructure
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
>  Labels: hive-4.0.0-must, pull-request-available
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Make Apache Hive be able to run inside docker container in pseudo-distributed 
> mode, with MySQL/Derby as its back database, provide the following:
>  * Quick-start/Debugging/Prepare a test env for Hive;
>  * Tools to build target image with specified version of Hive and its 
> dependencies;
>  * Images can be used as the basis for the Kubernetes operator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26400) Provide docker images for Hive

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26400:
-
Priority: Blocker  (was: Major)

> Provide docker images for Hive
> --
>
> Key: HIVE-26400
> URL: https://issues.apache.org/jira/browse/HIVE-26400
> Project: Hive
>  Issue Type: Improvement
>  Components: Build Infrastructure
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Blocker
>  Labels: hive-4.0.0-must, pull-request-available
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Make Apache Hive be able to run inside docker container in pseudo-distributed 
> mode, with MySQL/Derby as its back database, provide the following:
>  * Quick-start/Debugging/Prepare a test env for Hive;
>  * Tools to build target image with specified version of Hive and its 
> dependencies;
>  * Images can be used as the basis for the Kubernetes operator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26400) Provide docker images for Hive

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26400:
-
Target Version/s: 4.0.0

> Provide docker images for Hive
> --
>
> Key: HIVE-26400
> URL: https://issues.apache.org/jira/browse/HIVE-26400
> Project: Hive
>  Issue Type: Improvement
>  Components: Build Infrastructure
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
>  Labels: hive-4.0.0-must, pull-request-available
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Make Apache Hive be able to run inside docker container in pseudo-distributed 
> mode, with MySQL/Derby as its back database, provide the following:
>  * Quick-start/Debugging/Prepare a test env for Hive;
>  * Tools to build target image with specified version of Hive and its 
> dependencies;
>  * Images can be used as the basis for the Kubernetes operator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26644) Introduce auto sizing in HMS

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26644:
-
Labels: hive-4.0.0-must pull-request-available  (was: 
pull-request-available)

> Introduce auto sizing in HMS
> 
>
> Key: HIVE-26644
> URL: https://issues.apache.org/jira/browse/HIVE-26644
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Blocker
>  Labels: hive-4.0.0-must, pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HMS should have some ability to auto-size itself based on enabled features. 
> Server thread pool sizes-to-HMS connection pool sizes, larger pool sizes on 
> compaction-disabled-instances for better performance etc. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26644) Introduce auto sizing in HMS

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26644:
-
Target Version/s: 4.0.0

> Introduce auto sizing in HMS
> 
>
> Key: HIVE-26644
> URL: https://issues.apache.org/jira/browse/HIVE-26644
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Blocker
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HMS should have some ability to auto-size itself based on enabled features. 
> Server thread pool sizes-to-HMS connection pool sizes, larger pool sizes on 
> compaction-disabled-instances for better performance etc. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26644) Introduce auto sizing in HMS

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26644:
-
Priority: Blocker  (was: Major)

> Introduce auto sizing in HMS
> 
>
> Key: HIVE-26644
> URL: https://issues.apache.org/jira/browse/HIVE-26644
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Blocker
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HMS should have some ability to auto-size itself based on enabled features. 
> Server thread pool sizes-to-HMS connection pool sizes, larger pool sizes on 
> compaction-disabled-instances for better performance etc. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26045) Detect timed out connections for providers and auto-reconnect

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26045.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. Thank you for the contribution [~zhangbutao].

> Detect timed out connections for providers and auto-reconnect
> -
>
> Key: HIVE-26045
> URL: https://issues.apache.org/jira/browse/HIVE-26045
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: zhangbutao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> For the connectors, we use single connection, no pooling. But when the 
> connection is idle for an extended period, the JDBC connection times out. We 
> need to check for closed connections (Connection.isClosed()?) and 
> re-establish the connection. Otherwise it renders the connector fairly 
> useless.
> {noformat}
> 2022-03-17T13:02:16,635  WARN [HiveServer2-Handler-Pool: Thread-116] 
> thrift.ThriftCLIService: Error executing statement: 
> org.apache.hive.service.cli.HiveSQLException: Error while compiling 
> statement: FAILED: SemanticException Unable to fetch table temp_dbs. Error 
> retrieving remote 
> table:com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: 
> No operations allowed after connection closed.
>   at 
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:373)
>  ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:211)
>  ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:265)
>  ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at 
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:285) 
> ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:576)
>  ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:562)
>  ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) ~[?:?]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_231]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_231]
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
>  ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
>  ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
>  ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at java.security.AccessController.doPrivileged(Native Method) 
> ~[?:1.8.0_231]
>   at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_231]
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
>  ~[hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
>  ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at com.sun.proxy.$Proxy44.executeStatementAsync(Unknown Source) ~[?:?]
>   at 
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
>  ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:567)
>  ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at 
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1550)
>  ~[hive-exec-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at 
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1530)
>  ~[hive-exec-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) 
> 

[jira] [Commented] (HIVE-26611) add HiveServer2 History Server?

[Naveen Gangam (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-26611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615887#comment-17615887
 ] 

Naveen Gangam commented on HIVE-26611:
--

[~yigress] We have not invested much lately in the HS2 webui, mostly because of 
what Peter said. So what would the design look like ? Something like this, 
could help HS2 run with lower memory footprint as I assume the operation logs 
aren't retained in HS2 anymore. If you have a 1-pager for this feature, please 
post it to this jira. We can have the community chime in or put it up for a 
vote for PMC.


> add HiveServer2 History Server?
> ---
>
> Key: HIVE-26611
> URL: https://issues.apache.org/jira/browse/HIVE-26611
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Yi Zhang
>Priority: Major
>
> HiveServer2 Web UI provides query profile and optional operation log, however 
> these are gone when hs2 server exits. 
> Was there discussion of add a hs2 history server before?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26594) Upgrade netty to 4.1.77 due to CVE-2022-24823

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26594.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. Closing the jira. Thank you for the patch 
[~devaspatikrishnatri]

> Upgrade netty to 4.1.77 due to CVE-2022-24823
> -
>
> Key: HIVE-26594
> URL: https://issues.apache.org/jira/browse/HIVE-26594
> Project: Hive
>  Issue Type: Task
>Reporter: Devaspati Krishnatri
>Assignee: Devaspati Krishnatri
>Priority: Major
> Fix For: 4.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26502) Improve LDAP auth to support include generic user filters

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26502.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged to master. Thank you for the review [~dengzh]

> Improve LDAP auth to support include generic user filters
> -
>
> Key: HIVE-26502
> URL: https://issues.apache.org/jira/browse/HIVE-26502
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 4.0.0-alpha-1
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Fix For: 4.0.0
>
>
> Currently, Hive's ldap userfiltering is based on configuring a set of 
> patterns in which wild cards are replaced by usernames and searched for. 
> While this model supports advanced filtering options where a corporate ldap 
> can have users in different orgs and trees, it does not quite support generic 
> ldap searches like this.
> (&(uid={0})(objectClass=person))
> To be able to support this without making changes to the semantics of 
> existing configuration params, and to be backward compatible, we can enhance 
> the existing custom query functionality to support this.
> For with a configuration like this, we should be able to perform a search for 
> user who uid matches the username being authenticated.
> {noformat}
>   
> hive.server2.authentication.ldap.baseDN
> dc=apache,dc=org
>   
>   
> hive.server2.authentication.ldap.customLDAPQuery
> (&(uid={0})(objectClass=person))
>   
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26568) Upgrade Log4j2 to 2.18.0 due to CVEs

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26568.
--
Fix Version/s: 4.0.0
   Resolution: Duplicate

> Upgrade Log4j2 to 2.18.0 due to CVEs
> 
>
> Key: HIVE-26568
> URL: https://issues.apache.org/jira/browse/HIVE-26568
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26567) Hive Security - Upgrade Apache Log4j to 2.18.0 due to critical CVEs

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26567.
--
Fix Version/s: 4.0.0
 Assignee: Hongdan Zhu
   Resolution: Fixed

Merged via https://github.com/apache/hive/pull/3624
[~danielzhu] Please include the jira number in the commit message. This is very 
helpful in tracking jiras with the commits. Thank you

> Hive Security - Upgrade Apache Log4j to 2.18.0 due to critical CVEs
> ---
>
> Key: HIVE-26567
> URL: https://issues.apache.org/jira/browse/HIVE-26567
> Project: Hive
>  Issue Type: Improvement
>Reporter: Hongdan Zhu
>Assignee: Hongdan Zhu
>Priority: Major
> Fix For: 4.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26566) Upgrade H2 database version to 2.1.214

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam resolved HIVE-26566.
--
Fix Version/s: 4.0.0
   Resolution: Fixed

Fix has been merged. Thank you [~zabetak] for the review.

> Upgrade H2 database version to 2.1.214
> --
>
> Key: HIVE-26566
> URL: https://issues.apache.org/jira/browse/HIVE-26566
> Project: Hive
>  Issue Type: Task
>  Components: Testing Infrastructure
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>
> CVE-2021-23463 (CRITICAL severity) - The package com.h2database:h2 from 
> 1.4.198 and before 2.0.202 are vulnerable to XML External Entity (XXE) 
> Injection via the org.h2.jdbc.JdbcSQLXML class object, when it receives 
> parsed string data from org.h2.jdbc.JdbcResultSet.getSQLXML() method. If it 
> executes the getSource() method when the parameter is DOMSource.class it will 
> trigger the vulnerability.
> CVE-2021-42392 (CRITICAL severity) - The org.h2.util.JdbcUtils.getConnection 
> method of the H2 database takes as parameters the class name of the driver 
> and URL of the database. An attacker may pass a JNDI driver name and a URL 
> leading to a LDAP or RMI servers, causing remote code execution. This can be 
> exploited through various attack vectors, most notably through the H2 Console 
> which leads to unauthenticated remote code execution.
> CVE-2022-23221 (CRITICAL severity) - H2 Console before 2.1.210 allows remote 
> attackers to execute arbitrary code via a jdbc:h2:mem JDBC URL containing the 
> IGNORE_UNKNOWN_SETTINGS=TRUE;FORBID_CREATION=FALSE;INIT=RUNSCRIPT substring, 
> a different vulnerability than CVE-2021-42392.
> these have been addressed in 2.1.214



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26568) Upgrade Log4j2 to 2.18.0 due to CVEs

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26568:
-
Labels:   (was: pull-request-available)

> Upgrade Log4j2 to 2.18.0 due to CVEs
> 
>
> Key: HIVE-26568
> URL: https://issues.apache.org/jira/browse/HIVE-26568
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>
> High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26568) Upgrade Log4j2 to 2.18.0 due to CVEs

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26568:
-
Reporter: Naveen Gangam  (was: weidong)

> Upgrade Log4j2 to 2.18.0 due to CVEs
> 
>
> Key: HIVE-26568
> URL: https://issues.apache.org/jira/browse/HIVE-26568
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
>
> High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-26568) Upgrade Log4j2 to 2.18.0 due to CVEs

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-26568:



> Upgrade Log4j2 to 2.18.0 due to CVEs
> 
>
> Key: HIVE-26568
> URL: https://issues.apache.org/jira/browse/HIVE-26568
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: weidong
>Assignee: Hankó Gergely
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
>
> High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-26568) Upgrade Log4j2 to 2.18.0 due to CVEs

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-26568:


Assignee: Naveen Gangam  (was: Hankó Gergely)

> Upgrade Log4j2 to 2.18.0 due to CVEs
> 
>
> Key: HIVE-26568
> URL: https://issues.apache.org/jira/browse/HIVE-26568
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: weidong
>Assignee: Naveen Gangam
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
>
> High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26568) Upgrade Log4j2 to 2.18.0 due to CVEs

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26568:
-
Fix Version/s: (was: 4.0.0)
   (was: 4.0.0-alpha-1)

> Upgrade Log4j2 to 2.18.0 due to CVEs
> 
>
> Key: HIVE-26568
> URL: https://issues.apache.org/jira/browse/HIVE-26568
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>  Labels: pull-request-available
>
> High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26566) Upgrade H2 database version to 2.1.214

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26566:
-
Reporter: Naveen Gangam  (was: Stamatis Zampetakis)

> Upgrade H2 database version to 2.1.214
> --
>
> Key: HIVE-26566
> URL: https://issues.apache.org/jira/browse/HIVE-26566
> Project: Hive
>  Issue Type: Task
>  Components: Testing Infrastructure
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
>
> CVE-2021-23463 (CRITICAL severity) - The package com.h2database:h2 from 
> 1.4.198 and before 2.0.202 are vulnerable to XML External Entity (XXE) 
> Injection via the org.h2.jdbc.JdbcSQLXML class object, when it receives 
> parsed string data from org.h2.jdbc.JdbcResultSet.getSQLXML() method. If it 
> executes the getSource() method when the parameter is DOMSource.class it will 
> trigger the vulnerability.
> CVE-2021-42392 (CRITICAL severity) - The org.h2.util.JdbcUtils.getConnection 
> method of the H2 database takes as parameters the class name of the driver 
> and URL of the database. An attacker may pass a JNDI driver name and a URL 
> leading to a LDAP or RMI servers, causing remote code execution. This can be 
> exploited through various attack vectors, most notably through the H2 Console 
> which leads to unauthenticated remote code execution.
> CVE-2022-23221 (CRITICAL severity) - H2 Console before 2.1.210 allows remote 
> attackers to execute arbitrary code via a jdbc:h2:mem JDBC URL containing the 
> IGNORE_UNKNOWN_SETTINGS=TRUE;FORBID_CREATION=FALSE;INIT=RUNSCRIPT substring, 
> a different vulnerability than CVE-2021-42392.
> these have been addressed in 2.1.214



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26566) Upgrade H2 database version to 2.1.214

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26566:
-
Description: 
CVE-2021-23463 (CRITICAL severity) - The package com.h2database:h2 from 1.4.198 
and before 2.0.202 are vulnerable to XML External Entity (XXE) Injection via 
the org.h2.jdbc.JdbcSQLXML class object, when it receives parsed string data 
from org.h2.jdbc.JdbcResultSet.getSQLXML() method. If it executes the 
getSource() method when the parameter is DOMSource.class it will trigger the 
vulnerability.

CVE-2021-42392 (CRITICAL severity) - The org.h2.util.JdbcUtils.getConnection 
method of the H2 database takes as parameters the class name of the driver and 
URL of the database. An attacker may pass a JNDI driver name and a URL leading 
to a LDAP or RMI servers, causing remote code execution. This can be exploited 
through various attack vectors, most notably through the H2 Console which leads 
to unauthenticated remote code execution.

CVE-2022-23221 (CRITICAL severity) - H2 Console before 2.1.210 allows remote 
attackers to execute arbitrary code via a jdbc:h2:mem JDBC URL containing the 
IGNORE_UNKNOWN_SETTINGS=TRUE;FORBID_CREATION=FALSE;INIT=RUNSCRIPT substring, a 
different vulnerability than CVE-2021-42392.

these have been addressed in 2.1.214

  was:
The 1.3.166 version, which is in use in Hive, suffers from the following 
security vulnerabilities:
https://nvd.nist.gov/vuln/detail/CVE-2021-42392
https://nvd.nist.gov/vuln/detail/CVE-2022-23221

In the project, we use H2 only for testing purposes (inside the jdbc-handler 
module) thus the H2 binaries are not present in the runtime classpath thus 
these CVEs do not pose a problem for Hive or its users. Nevertheless, it would 
be good to upgrade to a more recent version to avoid Hive coming up in 
vulnerability scans due to this.


> Upgrade H2 database version to 2.1.214
> --
>
> Key: HIVE-26566
> URL: https://issues.apache.org/jira/browse/HIVE-26566
> Project: Hive
>  Issue Type: Task
>  Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
>
> CVE-2021-23463 (CRITICAL severity) - The package com.h2database:h2 from 
> 1.4.198 and before 2.0.202 are vulnerable to XML External Entity (XXE) 
> Injection via the org.h2.jdbc.JdbcSQLXML class object, when it receives 
> parsed string data from org.h2.jdbc.JdbcResultSet.getSQLXML() method. If it 
> executes the getSource() method when the parameter is DOMSource.class it will 
> trigger the vulnerability.
> CVE-2021-42392 (CRITICAL severity) - The org.h2.util.JdbcUtils.getConnection 
> method of the H2 database takes as parameters the class name of the driver 
> and URL of the database. An attacker may pass a JNDI driver name and a URL 
> leading to a LDAP or RMI servers, causing remote code execution. This can be 
> exploited through various attack vectors, most notably through the H2 Console 
> which leads to unauthenticated remote code execution.
> CVE-2022-23221 (CRITICAL severity) - H2 Console before 2.1.210 allows remote 
> attackers to execute arbitrary code via a jdbc:h2:mem JDBC URL containing the 
> IGNORE_UNKNOWN_SETTINGS=TRUE;FORBID_CREATION=FALSE;INIT=RUNSCRIPT substring, 
> a different vulnerability than CVE-2021-42392.
> these have been addressed in 2.1.214



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-26566) Upgrade H2 database version to 2.1.214

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-26566:


Assignee: Naveen Gangam  (was: Stamatis Zampetakis)

> Upgrade H2 database version to 2.1.214
> --
>
> Key: HIVE-26566
> URL: https://issues.apache.org/jira/browse/HIVE-26566
> Project: Hive
>  Issue Type: Task
>  Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Naveen Gangam
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
>
> CVE-2021-23463 (CRITICAL severity) - The package com.h2database:h2 from 
> 1.4.198 and before 2.0.202 are vulnerable to XML External Entity (XXE) 
> Injection via the org.h2.jdbc.JdbcSQLXML class object, when it receives 
> parsed string data from org.h2.jdbc.JdbcResultSet.getSQLXML() method. If it 
> executes the getSource() method when the parameter is DOMSource.class it will 
> trigger the vulnerability.
> CVE-2021-42392 (CRITICAL severity) - The org.h2.util.JdbcUtils.getConnection 
> method of the H2 database takes as parameters the class name of the driver 
> and URL of the database. An attacker may pass a JNDI driver name and a URL 
> leading to a LDAP or RMI servers, causing remote code execution. This can be 
> exploited through various attack vectors, most notably through the H2 Console 
> which leads to unauthenticated remote code execution.
> CVE-2022-23221 (CRITICAL severity) - H2 Console before 2.1.210 allows remote 
> attackers to execute arbitrary code via a jdbc:h2:mem JDBC URL containing the 
> IGNORE_UNKNOWN_SETTINGS=TRUE;FORBID_CREATION=FALSE;INIT=RUNSCRIPT substring, 
> a different vulnerability than CVE-2021-42392.
> these have been addressed in 2.1.214



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26566) Upgrade H2 database version to 2.1.214

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26566:
-
Fix Version/s: (was: 4.0.0)
   (was: 4.0.0-alpha-1)

> Upgrade H2 database version to 2.1.214
> --
>
> Key: HIVE-26566
> URL: https://issues.apache.org/jira/browse/HIVE-26566
> Project: Hive
>  Issue Type: Task
>  Components: Testing Infrastructure
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
>  Labels: pull-request-available
>
> CVE-2021-23463 (CRITICAL severity) - The package com.h2database:h2 from 
> 1.4.198 and before 2.0.202 are vulnerable to XML External Entity (XXE) 
> Injection via the org.h2.jdbc.JdbcSQLXML class object, when it receives 
> parsed string data from org.h2.jdbc.JdbcResultSet.getSQLXML() method. If it 
> executes the getSource() method when the parameter is DOMSource.class it will 
> trigger the vulnerability.
> CVE-2021-42392 (CRITICAL severity) - The org.h2.util.JdbcUtils.getConnection 
> method of the H2 database takes as parameters the class name of the driver 
> and URL of the database. An attacker may pass a JNDI driver name and a URL 
> leading to a LDAP or RMI servers, causing remote code execution. This can be 
> exploited through various attack vectors, most notably through the H2 Console 
> which leads to unauthenticated remote code execution.
> CVE-2022-23221 (CRITICAL severity) - H2 Console before 2.1.210 allows remote 
> attackers to execute arbitrary code via a jdbc:h2:mem JDBC URL containing the 
> IGNORE_UNKNOWN_SETTINGS=TRUE;FORBID_CREATION=FALSE;INIT=RUNSCRIPT substring, 
> a different vulnerability than CVE-2021-42392.
> these have been addressed in 2.1.214



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-26566) Upgrade H2 database version to 2.1.214

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-26566:



> Upgrade H2 database version to 2.1.214
> --
>
> Key: HIVE-26566
> URL: https://issues.apache.org/jira/browse/HIVE-26566
> Project: Hive
>  Issue Type: Task
>  Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
>
> The 1.3.166 version, which is in use in Hive, suffers from the following 
> security vulnerabilities:
> https://nvd.nist.gov/vuln/detail/CVE-2021-42392
> https://nvd.nist.gov/vuln/detail/CVE-2022-23221
> In the project, we use H2 only for testing purposes (inside the jdbc-handler 
> module) thus the H2 binaries are not present in the runtime classpath thus 
> these CVEs do not pose a problem for Hive or its users. Nevertheless, it 
> would be good to upgrade to a more recent version to avoid Hive coming up in 
> vulnerability scans due to this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-26032) Upgrade cron-utils to 9.1.6

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-26032:


Assignee: Yuming Wang

> Upgrade cron-utils to 9.1.6
> ---
>
> Key: HIVE-26032
> URL: https://issues.apache.org/jira/browse/HIVE-26032
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Affects Versions: 3.1.3, 4.0.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> To fix [CVE-2021-41269|https://nvd.nist.gov/vuln/detail/CVE-2021-41269] issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-26502) Improve LDAP auth to support include generic user filters

[Naveen Gangam (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-26502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-26502:
-
Description: 
Currently, Hive's ldap userfiltering is based on configuring a set of patterns 
in which wild cards are replaced by usernames and searched for. While this 
model supports advanced filtering options where a corporate ldap can have users 
in different orgs and trees, it does not quite support generic ldap searches 
like this.
(&(uid={0})(objectClass=person))

To be able to support this without making changes to the semantics of existing 
configuration params, and to be backward compatible, we can enhance the 
existing custom query functionality to support this.

For with a configuration like this, we should be able to perform a search for 
user who uid matches the username being authenticated.

{noformat}
  
hive.server2.authentication.ldap.baseDN
dc=apache,dc=org
  
  
hive.server2.authentication.ldap.customLDAPQuery
(&(uid={0})(objectClass=person))
  

{noformat}


  was:
Currently, Hive's ldap userfiltering is based on configuring a set of patterns 
in which wild cards are replaced by usernames and searched for. While this 
model supports advanced filtering options where a corporate ldap can have users 
in different orgs and trees, it does not quite support generic ldap searches 
like this.
(&(uid={0})(objectClass=person))

To be able to support this without making changes to the semantics of existing 
configuration params, and to be backward compatible, we can enhance the 
existing custom query functionality to support this.

For with a configuration like this, we should be able to perform a search for 
user who uid matches the username being authenticated.
  
hive.server2.authentication.ldap.baseDN
dc=apache,dc=org
  
  
hive.server2.authentication.ldap.customLDAPQuery
(&(uid={0})(objectClass=person))
  



> Improve LDAP auth to support include generic user filters
> -
>
> Key: HIVE-26502
> URL: https://issues.apache.org/jira/browse/HIVE-26502
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 4.0.0-alpha-1
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
>
> Currently, Hive's ldap userfiltering is based on configuring a set of 
> patterns in which wild cards are replaced by usernames and searched for. 
> While this model supports advanced filtering options where a corporate ldap 
> can have users in different orgs and trees, it does not quite support generic 
> ldap searches like this.
> (&(uid={0})(objectClass=person))
> To be able to support this without making changes to the semantics of 
> existing configuration params, and to be backward compatible, we can enhance 
> the existing custom query functionality to support this.
> For with a configuration like this, we should be able to perform a search for 
> user who uid matches the username being authenticated.
> {noformat}
>   
> hive.server2.authentication.ldap.baseDN
> dc=apache,dc=org
>   
>   
> hive.server2.authentication.ldap.customLDAPQuery
> (&(uid={0})(objectClass=person))
>   
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)