[jira] [Created] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
Laszlo Pinter created HIVE-22818: Summary: Preparation for jetty 9.4.26 upgrade Key: HIVE-22818 URL: https://issues.apache.org/jira/browse/HIVE-22818 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Laszlo Pinter Assignee: Laszlo Pinter Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22793) Update default settings in HMS Benchmarking tool
Laszlo Pinter created HIVE-22793: Summary: Update default settings in HMS Benchmarking tool Key: HIVE-22793 URL: https://issues.apache.org/jira/browse/HIVE-22793 Project: Hive Issue Type: Bug Components: Standalone Metastore Reporter: Laszlo Pinter Assignee: Laszlo Pinter HMS Benchmarking tool has invalid default setting values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22765) Quote table names in CompactionTxnHandler
Laszlo Pinter created HIVE-22765:
Summary: Quote table names in CompactionTxnHandler
Key: HIVE-22765
URL: https://issues.apache.org/jira/browse/HIVE-22765
Project: Hive
Issue Type: Improvement
Affects Versions: 4.0.0
Reporter: Laszlo Pinter
Assignee: Zoltan Chovan
When I run a metastore backed up by a postgres db, I get the following
exception:
{code:java}
MetaException(message:Unable to connect to transaction database
org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does not
existMetaException(message:Unable to connect to transaction database
org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does not
exist Position: 83 at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309) at
org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at
org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at
org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at
org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at
org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at
org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at
org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
at
org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
at
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at
org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code}
This is due to missing quotes around table names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22729) Provide a failure reason for failed compactions
Laszlo Pinter created HIVE-22729:
Summary: Provide a failure reason for failed compactions
Key: HIVE-22729
URL: https://issues.apache.org/jira/browse/HIVE-22729
Project: Hive
Issue Type: Improvement
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
We should provide a compaction failure reason as easily accessible as possible.
Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22703) Compaction configuration check when starting HMS/HS2
Laszlo Pinter created HIVE-22703: Summary: Compaction configuration check when starting HMS/HS2 Key: HIVE-22703 URL: https://issues.apache.org/jira/browse/HIVE-22703 Project: Hive Issue Type: Improvement Components: Hive Reporter: Laszlo Pinter Assignee: Laszlo Pinter Currently when starting HMS we can have bugous configuration which prevents compatction to work. We should find a way to inform the admin about the configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22700) Compactions may leak memory when unauthorized
Laszlo Pinter created HIVE-22700: Summary: Compactions may leak memory when unauthorized Key: HIVE-22700 URL: https://issues.apache.org/jira/browse/HIVE-22700 Project: Hive Issue Type: Bug Components: Hive Reporter: Laszlo Pinter Assignee: Laszlo Pinter Initiator class determines compaction type periodically. Initiator either runs as hive user or impersonates the owner of the table. When impersonation is used, Initiator#checkForCompaction may leak memory. If impersonation (ugi.doAs()) call fails, FileSystem.closeAllForUGI does not run, therefore does not clean the file system cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22586) Temporary table delete fails with "SemanticException No valid privileges"
Laszlo Pinter created HIVE-22586: Summary: Temporary table delete fails with "SemanticException No valid privileges" Key: HIVE-22586 URL: https://issues.apache.org/jira/browse/HIVE-22586 Project: Hive Issue Type: Improvement Components: Hive Reporter: Laszlo Pinter When using Hive with Sentry, at the moment is not possible to delete a temporary tables without granting ALL privileges on database/table level. Temporary tables are strongly bound to the session, and it is only visible/accessible to the user who owns the session, and the table exists until the session is not closed. Since temporary tables only exists on session level, and they are not connected to any database, checking the privileges with Sentry is unnecessary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
Laszlo Pinter created HIVE-22554:
Summary: ACID: Wait timeout for blocking compaction should be
configurable
Key: HIVE-22554
URL: https://issues.apache.org/jira/browse/HIVE-22554
Project: Hive
Issue Type: Task
Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
The wait timeout for blocking compaction is hardcoded to 5 minutes.
{code:java}
public class AlterTableCompactOperation extends
DDLOperation {
private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000;
...
}{code}
This should be configurable via a Hive Configuration parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22542) Update orc version to 1.5.8
Laszlo Pinter created HIVE-22542: Summary: Update orc version to 1.5.8 Key: HIVE-22542 URL: https://issues.apache.org/jira/browse/HIVE-22542 Project: Hive Issue Type: Task Reporter: Laszlo Pinter Assignee: Laszlo Pinter Update orc to 1.5.8 in root pom -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22535) ACID: Check for already running compactions in Worker
Laszlo Pinter created HIVE-22535: Summary: ACID: Check for already running compactions in Worker Key: HIVE-22535 URL: https://issues.apache.org/jira/browse/HIVE-22535 Project: Hive Issue Type: Bug Components: Hive Reporter: Laszlo Pinter Assignee: Laszlo Pinter Compactor worker doesn't check if a compaction is already running for a given resource. Initiator does this, but in case of manual compaction, this check is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22534) ACID: Improve Compactor thread logging
Laszlo Pinter created HIVE-22534: Summary: ACID: Improve Compactor thread logging Key: HIVE-22534 URL: https://issues.apache.org/jira/browse/HIVE-22534 Project: Hive Issue Type: Bug Reporter: Laszlo Pinter Assignee: Laszlo Pinter Make sure that it is easy to find issues when one of the compactor thread fails. Maybe: * MDC - with iteration / threadname - so we can easily grep the logs for a given run * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22474) Query based major compaction always creates only one bucket file
Laszlo Pinter created HIVE-22474:
Summary: Query based major compaction always creates only one
bucket file
Key: HIVE-22474
URL: https://issues.apache.org/jira/browse/HIVE-22474
Project: Hive
Issue Type: Sub-task
Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
{code:sql}
set hive.execution.engine=mr;
drop table if exists tbl2;
create table tbl2 (a int, b int) clustered by (a) into 2 buckets stored as ORC
TBLPROPERTIES('bucketing_version'='2', 'transactional'='true',
'compactorthreshold.hive.compactor.delta.num.threshold'='3');
insert into tbl2 values(1,2),(1,3),(1,4),(2,2),(2,3),(2,4);
insert into tbl2 values(3,2),(3,3),(3,4),(4,2),(4,3),(4,4);
delete from tbl2 where b = 2;
insert into tbl2 values(5,2),(5,3),(5,4),(6,2),(6,3),(6,4);
delete from tbl2 where a = 1;
{code}
Having the above use case, at the end of the major compaction the base
directory contains only one bucket file, although the table is bucketed in 2
buckets. Before running the compaction, the delta directories contains the
right amount of bucket files, and the data is split accordingly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22473) Avoid reading table as ACID when table name is starting with "delta"
Laszlo Pinter created HIVE-22473:
Summary: Avoid reading table as ACID when table name is starting
with "delta"
Key: HIVE-22473
URL: https://issues.apache.org/jira/browse/HIVE-22473
Project: Hive
Issue Type: Bug
Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
Fix For: 4.0.0
{code:sql}
create table delta_result (a int) stored as orc
tblproperties('transactional'='false');
insert into delta_result select 1;
select * from delta_result;
{code}
The above query will result in the following exception:
2019-11-08T13:49:05,780 WARN [HiveServer2-Handler-Pool: Thread-7906]
thrift.ThriftCLIService: Error fetching results:2019-11-08T13:49:05,780 WARN
[HiveServer2-Handler-Pool: Thread-7906] thrift.ThriftCLIService: Error fetching
results:org.apache.hive.service.cli.HiveSQLException: java.io.IOException:
java.lang.RuntimeException: ORC split generation failed with exception:
java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:481)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:331)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:946)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:567)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:801)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_211] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]Caused by:
java.io.IOException: java.lang.RuntimeException: ORC split generation failed
with exception: java.lang.StringIndexOutOfBoundsException: String index out of
range: -1 at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:638)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:151)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2142)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:241)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:476)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] ... 13 moreCaused by:
java.lang.RuntimeException: ORC split generation failed with exception:
java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1929)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2016)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:461)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:430)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:336)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:576)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545)
[jira] [Created] (HIVE-22401) ACID: Refactor CompactorMR
Laszlo Pinter created HIVE-22401: Summary: ACID: Refactor CompactorMR Key: HIVE-22401 URL: https://issues.apache.org/jira/browse/HIVE-22401 Project: Hive Issue Type: Sub-task Components: Hive Reporter: Laszlo Pinter Assignee: Laszlo Pinter At the moment CompactorMR contains all the methods which handles compaction. Since compaction can also be triggered as hive queries not only as MR jobs, it makes sense to move out these methods to a separate class, to avoid confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22280) Q tests for partitioned temporary tables
Laszlo Pinter created HIVE-22280: Summary: Q tests for partitioned temporary tables Key: HIVE-22280 URL: https://issues.apache.org/jira/browse/HIVE-22280 Project: Hive Issue Type: Sub-task Components: Hive Reporter: Laszlo Pinter Assignee: Laszlo Pinter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22279) Enable temporary table partitioning
Laszlo Pinter created HIVE-22279: Summary: Enable temporary table partitioning Key: HIVE-22279 URL: https://issues.apache.org/jira/browse/HIVE-22279 Project: Hive Issue Type: Sub-task Components: Hive Reporter: Laszlo Pinter Assignee: Laszlo Pinter In order to have full support for partitioned temporary tables, some checks from the SemanticAnalyzer must be removed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22242) Move TempTable and PartitionTree out of SessionHiveMetastoreClient
Laszlo Pinter created HIVE-22242: Summary: Move TempTable and PartitionTree out of SessionHiveMetastoreClient Key: HIVE-22242 URL: https://issues.apache.org/jira/browse/HIVE-22242 Project: Hive Issue Type: Sub-task Components: Hive Reporter: Laszlo Pinter Assignee: Laszlo Pinter SessionHiveMetastoreClient is getting too crowded and unreadable. TempTable and PartitionTree should moved into a separate class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22230) Add support for filtering partitions on temporary tables
Laszlo Pinter created HIVE-22230:
Summary: Add support for filtering partitions on temporary tables
Key: HIVE-22230
URL: https://issues.apache.org/jira/browse/HIVE-22230
Project: Hive
Issue Type: Task
Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
We need support for filtering partitions on temporary tables. In order to
achieve this, SessionHiveMetastoreClient must implement the following methods:
{code:java}
public List listPartitionsByFilter(String catName, String dbName,
String tableName,String filter, int maxParts)
public int getNumPartitionsByFilter(String catName, String dbName, String
tableName, String filter)
public PartitionSpecProxy listPartitionSpecsByFilter(String catName, String
dbName, String tblName, String filter, int maxParts)
public PartitionValuesResponse listPartitionValues(PartitionValuesRequest
request)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22212) implement append partition related methods on temporary tables
Laszlo Pinter created HIVE-22212:
Summary: implement append partition related methods on temporary
tables
Key: HIVE-22212
URL: https://issues.apache.org/jira/browse/HIVE-22212
Project: Hive
Issue Type: Sub-task
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
The following methods must be implemented in SessionHiveMetastoreClient, in
order to support partition append on temporary tables:
{code:java}
Partition appendPartition(String dbName, String tableName, List
partVals)
throws InvalidObjectException, AlreadyExistsException, MetaException,
TException;
Partition appendPartition(String catName, String dbName, String tableName,
List partVals)
throws InvalidObjectException, AlreadyExistsException, MetaException,
TException;
Partition appendPartition(String dbName, String tableName, String name)
throws InvalidObjectException, AlreadyExistsException, MetaException,
TException;
Partition appendPartition(String catName, String dbName, String tableName,
String name)
throws InvalidObjectException, AlreadyExistsException, MetaException,
TException;
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (HIVE-22137) Implement alter/rename partition related methods on temporary tables
Laszlo Pinter created HIVE-22137:
Summary: Implement alter/rename partition related methods on
temporary tables
Key: HIVE-22137
URL: https://issues.apache.org/jira/browse/HIVE-22137
Project: Hive
Issue Type: Sub-task
Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
IMetaStoreClient exposes the following methods related to altering of
partitions:
{code:java}
void alter_partition(String dbName, String tblName, Partition newPart);
void alter_partition(String catName, String dbName, String tblName, Partition
newPart);
void alter_partition(String dbName, String tblName, Partition newPart,
EnvironmentContext environmentContext);
void alter_partition(String catName, String dbName, String tblName, Partition
newPart, EnvironmentContext environmentContext, String writeIdList);
void alter_partition(String catName, String dbName, String tblName, Partition
newPart, EnvironmentContext environmentContext);
void alter_partitions(String dbName, String tblName, List newParts);
void alter_partitions(String dbName, String tblName, List newParts,
EnvironmentContext environmentContext);
void alter_partitions(String dbName, String tblName, List newParts,
EnvironmentContext environmentContext,String writeIdList, long writeId);
void alter_partitions(String catName, String dbName, String tblName,
List newParts);
void alter_partitions(String catName, String dbName, String tblName,
List newParts, EnvironmentContext environmentContext, String
writeIdList, long writeId);
void renamePartition(final String dbname, final String tableName, final
List part_vals, final Partition newPart);
void renamePartition(String catName, String dbname, String tableName,
List part_vals, Partition newPart, String validWriteIds){code}
These should be implemented, in order to completely support partition on
temporary tables.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (HIVE-22084) Implement exchange partitions related methods on SessionHiveMetastoreClient
Laszlo Pinter created HIVE-22084:
Summary: Implement exchange partitions related methods on
SessionHiveMetastoreClient
Key: HIVE-22084
URL: https://issues.apache.org/jira/browse/HIVE-22084
Project: Hive
Issue Type: Sub-task
Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
Fix For: 4.0.0
IMetaStoreClient exposes the following methods related to exchanging partitions:
{code:java}
Partition exchange_partition(Map partitionSpecs, String
sourceDb, String sourceTable, String destdb, String destTableName);
Partition exchange_partition(Map partitionSpecs, String
sourceCat, String sourceDb, String sourceTable, String destCat, String destdb,
String destTableName);
List exchange_partitions(Map partitionSpecs, String
sourceDb, String sourceTable, String destdb, String destTableName);
List exchange_partitions(Map partitionSpecs, String
sourceCat, String sourceDb, String sourceTable, String destCat, String destdb,
String destTableName);{code}
In order to support partitions on temporary tables, these methods must be
implemented.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (HIVE-21957) Create temporary table like should omit transactional properties
Laszlo Pinter created HIVE-21957: Summary: Create temporary table like should omit transactional properties Key: HIVE-21957 URL: https://issues.apache.org/jira/browse/HIVE-21957 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 4.0.0 Reporter: Laszlo Pinter Assignee: Laszlo Pinter In case of create temporary table like queries, where the source table is transactional, the transactional properties should not be copied over to the new table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21875) Implement drop partition related methods on temporary tables
Laszlo Pinter created HIVE-21875:
Summary: Implement drop partition related methods on temporary
tables
Key: HIVE-21875
URL: https://issues.apache.org/jira/browse/HIVE-21875
Project: Hive
Issue Type: Sub-task
Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
IMetaStoreClient exposes the following methods related to dropping partitions:
{code:java}
boolean dropPartition(String db_name, String tbl_name, List part_vals,
boolean deleteData);
boolean dropPartition(String catName, String db_name, String tbl_name,
List part_vals, boolean deleteData);
boolean dropPartition(String db_name, String tbl_name, List part_vals,
PartitionDropOptions options);
boolean dropPartition(String catName, String db_name, String tbl_name,
List part_vals, PartitionDropOptions options);
List dropPartitions(String dbName, String tblName,
List> partExprs, boolean deleteData, boolean
ifExists);
List dropPartitions(String catName, String dbName, String tblName,
List> partExprs, boolean deleteData, boolean
ifExists);
List dropPartitions(String dbName, String tblName,
List> partExprs, boolean deleteData, boolean
ifExists, boolean needResults);
List dropPartitions(String catName, String dbName, String tblName,
List> partExprs, boolean deleteData, boolean
ifExists, boolean needResults);
List dropPartitions(String dbName, String tblName,
List> partExprs, PartitionDropOptions options);
List dropPartitions(String catName, String dbName, String tblName,
List> partExprs, PartitionDropOptions options);
boolean dropPartition(String db_name, String tbl_name, String name, boolean
deleteData);
boolean dropPartition(String catName, String db_name, String tbl_name, String
name, boolean deleteData){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-21874) Implement add partitions related methods on temporary table
Laszlo Pinter created HIVE-21874:
Summary: Implement add partitions related methods on temporary
table
Key: HIVE-21874
URL: https://issues.apache.org/jira/browse/HIVE-21874
Project: Hive
Issue Type: Sub-task
Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
IMetaStoreClient exposes the following add partition related methods:
{code:java}
Partition add_partition(Partition partition);
int add_partitions(List partitions);
int add_partitions_pspec(PartitionSpecProxy partitionSpec);
List add_partitions(List partitions, boolean ifNotExists,
boolean needResults);
{code}
These methods should be implemented in order to handle addition of partitions
to temporary tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-21814) Implement list partitions related methods on temporary tables
Laszlo Pinter created HIVE-21814:
Summary: Implement list partitions related methods on temporary
tables
Key: HIVE-21814
URL: https://issues.apache.org/jira/browse/HIVE-21814
Project: Hive
Issue Type: Sub-task
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
IMetaStoreClient exposes the following methods related to listing partitions:
{code:java}
List listPartitionNames(String db_name, String tbl_name, List
part_vals, short max_parts);
List listPartitionNames(String catName, String db_name, String
tbl_name, List part_vals, int max_parts);
List listPartitions(String db_name, String tbl_name, List
part_vals, short max_parts);
List listPartitions(String catName, String db_name, String tbl_name,
List part_vals, int max_parts);
List listPartitionNames(String db_name, String tbl_name, short
max_parts);
List listPartitionNames(String catName, String db_name, String
tbl_name, int max_parts);
List listPartitionNames(String db_name, String tbl_name, List
part_vals, short max_parts);
List listPartitionNames(String catName, String db_name, String
tbl_name, List part_vals, int max_parts);
PartitionSpecProxy listPartitionSpecs(String dbName, String tableName, int
maxParts);
PartitionSpecProxy listPartitionSpecs(String catName, String dbName, String
tableName,int maxParts);
List listPartitionsWithAuthInfo(String dbName, String tableName,
List partialPvals, short maxParts, String userName, List
groupNames);
List listPartitionsWithAuthInfo(String catName, String dbName,
String tableName, List partialPvals, int maxParts, String userName,
List groupNames);
{code}
In order to support partitions on temporary tables, the majority of these
methods must be implemented in SessionHiveMetastoreClient.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-21812) Implement get partition related methods on temporary tables
Laszlo Pinter created HIVE-21812:
Summary: Implement get partition related methods on temporary
tables
Key: HIVE-21812
URL: https://issues.apache.org/jira/browse/HIVE-21812
Project: Hive
Issue Type: Sub-task
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
IMetastoreClient exposes several APIs related to getting partitions. These
definition of the APIs are:
{code:java}
Partition getPartition(String dbName, String tblName, List partVals);
Partition getPartition(String catName, String dbName, String tblName,
List partVals);
Partition getPartition(String dbName, String tblName, String name);
Partition getPartition(String catName, String dbName, String tblName, String
name);
Partition getPartitionWithAuthInfo(String dbName, String tableName,
List pvals, String userName, List groupNames);
Partition getPartitionWithAuthInfo(String catName, String dbName, String
tableName, List pvals, String userName, List groupNames);
List getPartitionsByNames(String db_name, String tbl_name,
List part_names);
List getPartitionsByNames(String catName, String db_name, String
tbl_name, List part_names);
Partition getPartitionWithAuthInfo(String dbName, String tableName,
List pvals, String userName, List groupNames);
Partition getPartitionWithAuthInfo(String catName, String dbName, String
tableName, List pvals, String userName, List groupNames);
{code}
In order to support partitions on temporary tables, the majority of these
methods must be implemented in SessionHiveMetastoreClient.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-21765) Support partitions on temporary tables
Laszlo Pinter created HIVE-21765: Summary: Support partitions on temporary tables Key: HIVE-21765 URL: https://issues.apache.org/jira/browse/HIVE-21765 Project: Hive Issue Type: New Feature Reporter: Laszlo Pinter Assignee: Laszlo Pinter HIVE-18739 and HIVE-20661 added partial support for partition columns on temporary tables, but it was not complete and it was available only for internal usage. The scope of this jira, is to implement the missing parts and expose this functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21603) Java 11 preparation: update powermock version
Laszlo Pinter created HIVE-21603: Summary: Java 11 preparation: update powermock version Key: HIVE-21603 URL: https://issues.apache.org/jira/browse/HIVE-21603 Project: Hive Issue Type: Task Components: Hive Affects Versions: 4.0.0 Reporter: Laszlo Pinter Assignee: Laszlo Pinter PowerMock1 has no support for Java11, therefore we need to bump its version to 2.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21569) Preparation to bump guava version
Laszlo Pinter created HIVE-21569: Summary: Preparation to bump guava version Key: HIVE-21569 URL: https://issues.apache.org/jira/browse/HIVE-21569 Project: Hive Issue Type: Task Components: Hive Reporter: Laszlo Pinter Assignee: Laszlo Pinter Fix For: 4.0.0 This is a preparation work in hive, before updating guava version to 27.0-jre. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21086) Fetch task and MR job give back different results when multiple partitions point to the same location
Laszlo Pinter created HIVE-21086:
Summary: Fetch task and MR job give back different results when
multiple partitions point to the same location
Key: HIVE-21086
URL: https://issues.apache.org/jira/browse/HIVE-21086
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 4.0.0
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
Fix For: 4.0.0
Create a table with 2 partitions where both partitions share the same location
and inserting a single line to one of them.
{code:sql}
create table test (i int) partitioned by (j int);
alter table test add partition (j=1);
alter table test add partition (j=2) location '<>';
insert into table test partition (j=1) values (1);{code}
Make sure *hive.fetch.task.conversion=more*.
{code:sql}
set hive.fetch.task.conversion=more;{code}
The result of a simple select * test:
+---+---+
|test.i |test.j |
+---+---+
|1 |1 |
|1 |2 |
+---+---+
Change the *hive.fetch.task.conversion=none;*
{code:sql}
set hive.fetch.task.conversion=none;
{code}
The result of a simple select * test:
+---+---+
|test.i |test.j |
+---+---+
|1 |2 |
+---+---+
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20891) Call alter_partition in batch when dynamically loading partitions
Laszlo Pinter created HIVE-20891: Summary: Call alter_partition in batch when dynamically loading partitions Key: HIVE-20891 URL: https://issues.apache.org/jira/browse/HIVE-20891 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0 Reporter: Laszlo Pinter Assignee: Laszlo Pinter When dynamically loading partitions, the setStatsPropAndAlterPartition() is called for each partition one by one, resulting in unnecessary calls to the metastore client. This whole logic can be changed to just one call. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20796) jdbc URL can contain sensitive information that should not be logged
Laszlo Pinter created HIVE-20796: Summary: jdbc URL can contain sensitive information that should not be logged Key: HIVE-20796 URL: https://issues.apache.org/jira/browse/HIVE-20796 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0 Reporter: Laszlo Pinter Assignee: Laszlo Pinter It is possible to put passwords in the jdbc connection url and some jdbc drivers will supposedly use that. (derby, mysql). This information is considered sensitive, and should be masked out, while logging the connection url. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1
Laszlo Pinter created HIVE-20661: Summary: Dynamic partitions loading calls add partition for every partition 1-by-1 Key: HIVE-20661 URL: https://issues.apache.org/jira/browse/HIVE-20661 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0 Reporter: Laszlo Pinter Assignee: Laszlo Pinter Hive.loadDynamicPartitions creates partitions using a threadpool, but the update of the Metastore via the MetastoreClient is done 1-by-1. This adds unnecessary extra calls. The partitions should be created in one batch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20595) Add findbugs-exclude.xml to metastore-server
Laszlo Pinter created HIVE-20595: Summary: Add findbugs-exclude.xml to metastore-server Key: HIVE-20595 URL: https://issues.apache.org/jira/browse/HIVE-20595 Project: Hive Issue Type: Bug Components: Hive, Standalone Metastore Affects Versions: 4.0.0 Reporter: Laszlo Pinter Assignee: Laszlo Pinter The findbugs-exclude.xml is missing from standalone-metastore/metastore-server/findbugs. This should be added, otherwise the findbugs check will fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20575) Fix flaky connection metric tests
Laszlo Pinter created HIVE-20575: Summary: Fix flaky connection metric tests Key: HIVE-20575 URL: https://issues.apache.org/jira/browse/HIVE-20575 Project: Hive Issue Type: Test Components: Hive, Test Affects Versions: 4.0.0 Reporter: Laszlo Pinter Assignee: Laszlo Pinter Fix For: 4.0.0 TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics() is flaky. We need to fix it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20551) Create PreparedStatement query dynamically when IN clause is used
Laszlo Pinter created HIVE-20551: Summary: Create PreparedStatement query dynamically when IN clause is used Key: HIVE-20551 URL: https://issues.apache.org/jira/browse/HIVE-20551 Project: Hive Issue Type: Bug Reporter: Laszlo Pinter Assignee: Laszlo Pinter In the MetaStoreDirectSql class when IN clause is used, the query statement is created via string concatenation, meaning that an attacker could change the statement meaning or insert arbitrary SQL commands. Since JDBC API allows only one literal for one “?” parameter, PreparedStatement doesn’t work for IN clause queries. To create the PreparedStatement query dynamically based on the size of the elements in IN clause, the makeParams() should be used instead of concatenation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20505) upgrade org.openjdk.jmh:jmh-core to 1.21
Laszlo Pinter created HIVE-20505: Summary: upgrade org.openjdk.jmh:jmh-core to 1.21 Key: HIVE-20505 URL: https://issues.apache.org/jira/browse/HIVE-20505 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 4.0.0 Reporter: Laszlo Pinter Assignee: Laszlo Pinter Fix For: 4.0.0 This ticket tracks the following CVE(s) that were found in the direct dependency *org.openjdk.jmh:jmh-core:1.19*: [CVE-2009-1896|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1896], [CVE-2009-2689|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-2689], [CVE-2009-3879|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3879], [CVE-2009-0733|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-0733], [CVE-2009-2475|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-2475], [CVE-2009-3883|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3883], [CVE-2009-2476|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-2476], [CVE-2009-3884|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3884], [CVE-2013-0169|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-0169], [CVE-2012-5373|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5373], [CVE-2009-3880|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3880], [CVE-2009-3881|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3881], [CVE-2009-3882|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3882], [CVE-2009-0581|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-0581], [CVE-2009-2690|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-2690], [CVE-2012-2739|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-2739], [CVE-2009-0723|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-0723], [CVE-2009-3728|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3728] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20396) Test HS2 open_connection metrics
Laszlo Pinter created HIVE-20396: Summary: Test HS2 open_connection metrics Key: HIVE-20396 URL: https://issues.apache.org/jira/browse/HIVE-20396 Project: Hive Issue Type: Test Components: HiveServer2 Reporter: Laszlo Pinter Assignee: Laszlo Pinter Fix For: 4.0.0 HiveServer2 is emitting metrics _default.General.open_connections_ in both binary and http mode. These metrics should be tested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20350) Unnecessary value assignment
Laszlo Pinter created HIVE-20350:
Summary: Unnecessary value assignment
Key: HIVE-20350
URL: https://issues.apache.org/jira/browse/HIVE-20350
Project: Hive
Issue Type: Improvement
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
There is an unnecessary value assignment in Hive.java
{code:java}
newPartPath = oldPartPath == null ? newPartPath = genPartPathFromTable(tbl,
partSpec,
tblDataLocationPath) : oldPartPath;{code}
This can be changed to
{code:java}
newPartPath = oldPartPath == null ? genPartPathFromTable(tbl, partSpec,
tblDataLocationPath) : oldPartPath;{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
