[jira] [Resolved] (HIVE-27668) Backport of HIVE-21126: Allow session level queries in LlapBaseInputFormat#getSplits() before actual get_splits() call
[ https://issues.apache.org/jira/browse/HIVE-27668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-27668. - Fix Version/s: 3.2.0 Resolution: Fixed > Backport of HIVE-21126: Allow session level queries in > LlapBaseInputFormat#getSplits() before actual get_splits() call > -- > > Key: HIVE-27668 > URL: https://issues.apache.org/jira/browse/HIVE-27668 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27679) Ranger Yarn Queue policies are not applying correctly, rework done for HIVE-26352
[
https://issues.apache.org/jira/browse/HIVE-27679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-27679:
--
Labels: pull-request-available (was: )
> Ranger Yarn Queue policies are not applying correctly, rework done for
> HIVE-26352
> -
>
> Key: HIVE-27679
> URL: https://issues.apache.org/jira/browse/HIVE-27679
> Project: Hive
> Issue Type: Bug
>Reporter: Mahesh Raju Somalaraju
>Assignee: Mahesh Raju Somalaraju
>Priority: Major
> Labels: pull-request-available
>
> This Jira is raised to modify/fix the code which is done in part of
> *HIVE-26352.*
> Versions which have {*}HIVE-26352{*}/HIVE-27029 are not able to enforce Yarn
> Ranger queue policies, because the change made in {*}HIVE-26352{*}/HIVE-27029
> catches ALL exceptions, so exceptions normally thrown is ignored and the user
> is allowed to run a job in that queue.
> Allowing user to run jobs is not expecting behaviour.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27679) Ranger Yarn Queue policies are not applying correctly, rework done for HIVE-26352
[
https://issues.apache.org/jira/browse/HIVE-27679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Raju Somalaraju reassigned HIVE-27679:
-
Assignee: Mahesh Raju Somalaraju
> Ranger Yarn Queue policies are not applying correctly, rework done for
> HIVE-26352
> -
>
> Key: HIVE-27679
> URL: https://issues.apache.org/jira/browse/HIVE-27679
> Project: Hive
> Issue Type: Bug
>Reporter: Mahesh Raju Somalaraju
>Assignee: Mahesh Raju Somalaraju
>Priority: Major
>
> This Jira is raised to modify/fix the code which is done in part of
> *HIVE-26352.*
> Versions which have {*}HIVE-26352{*}/HIVE-27029 are not able to enforce Yarn
> Ranger queue policies, because the change made in {*}HIVE-26352{*}/HIVE-27029
> catches ALL exceptions, so exceptions normally thrown is ignored and the user
> is allowed to run a job in that queue.
> Allowing user to run jobs is not expecting behaviour.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27683) Incorrect result when filtering the table with default partition
[
https://issues.apache.org/jira/browse/HIVE-27683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng updated HIVE-27683:
---
Description:
Steps to repro:
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
int, partcol2 int);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select '2', 2, NULL;
select intcol from pt.alterdynamic_part_table where (partcol1=2 and partcol2=1)
or (partcol1=2 and partcol2='__HIVE_DEFAULT_PARTITION__');
select intcol from pt.alterdynamic_part_table where
partcol2='__HIVE_DEFAULT_PARTITION__';{noformat}
The last two queries should return one line row instead of an empty result.
was:
Steps to repro:
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
int, partcol2 int);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select '2', 2, NULL;
select intcol from pt.alterdynamic_part_table where (partcol1=2 and partcol2=1)
or (partcol1=2 and partcol2='__HIVE_DEFAULT_PARTITION__');
select intcol from pt.alterdynamic_part_table where
partcol2='__HIVE_DEFAULT_PARTITION__';{noformat}
The last two queries should return one line row instead of empty result.
> Incorrect result when filtering the table with default partition
>
>
> Key: HIVE-27683
> URL: https://issues.apache.org/jira/browse/HIVE-27683
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
>
> Steps to repro:
> {noformat}
> create database pt;
> create table pt.alterdynamic_part_table(intcol string) partitioned by
> (partcol1 int, partcol2 int);
> insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
> select '2', 2, NULL;
> select intcol from pt.alterdynamic_part_table where (partcol1=2 and
> partcol2=1) or (partcol1=2 and partcol2='__HIVE_DEFAULT_PARTITION__');
> select intcol from pt.alterdynamic_part_table where
> partcol2='__HIVE_DEFAULT_PARTITION__';{noformat}
> The last two queries should return one line row instead of an empty result.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27683) Incorrect result when filtering the table with default partition
[
https://issues.apache.org/jira/browse/HIVE-27683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng updated HIVE-27683:
---
Description:
Steps to repro:
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
int, partcol2 int);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select '2', 2, NULL;
select intcol from pt.alterdynamic_part_table where (partcol1=2 and partcol2=1)
or (partcol1=2 and partcol2='__HIVE_DEFAULT_PARTITION__');
select intcol from pt.alterdynamic_part_table where
partcol2='__HIVE_DEFAULT_PARTITION__';{noformat}
The last two queries should return one line row instead of empty result.
was:
Steps to repro:
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
int, partcol2 int);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select '2', 2, NULL;
select intcol from pt.alterdynamic_part_table where (partcol1=2 and partcol2=1)
or (partcol1=2 and partcol2='__HIVE_DEFAULT_PARTITION__');
set hive.cbo.enable=false;
select intcol from pt.alterdynamic_part_table where
partcol2='__HIVE_DEFAULT_PARTITION__';{noformat}
The last two queries should return one line row instead of empty result.
> Incorrect result when filtering the table with default partition
>
>
> Key: HIVE-27683
> URL: https://issues.apache.org/jira/browse/HIVE-27683
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
>
> Steps to repro:
> {noformat}
> create database pt;
> create table pt.alterdynamic_part_table(intcol string) partitioned by
> (partcol1 int, partcol2 int);
> insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
> select '2', 2, NULL;
> select intcol from pt.alterdynamic_part_table where (partcol1=2 and
> partcol2=1) or (partcol1=2 and partcol2='__HIVE_DEFAULT_PARTITION__');
> select intcol from pt.alterdynamic_part_table where
> partcol2='__HIVE_DEFAULT_PARTITION__';{noformat}
> The last two queries should return one line row instead of empty result.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27683) Incorrect result when filtering the table with default partition
[
https://issues.apache.org/jira/browse/HIVE-27683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng updated HIVE-27683:
---
Description:
Steps to repro:
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
int, partcol2 int);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select '2', 2, NULL;
select intcol from pt.alterdynamic_part_table where (partcol1=2 and partcol2=1)
or (partcol1=2 and partcol2='__HIVE_DEFAULT_PARTITION__');
set hive.cbo.enable=false;
select intcol from pt.alterdynamic_part_table where
partcol2='__HIVE_DEFAULT_PARTITION__';{noformat}
The last two queries should return one line row instead of empty result.
was:
Steps to repro:
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
string, partcol2 string);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select 2, '2', NULL;
select intcol from pt.alterdynamic_part_table where
partcol2='__HIVE_DEFAULT_PARTITION__';{noformat}
The last query should return 2 instead of empty result.
> Incorrect result when filtering the table with default partition
>
>
> Key: HIVE-27683
> URL: https://issues.apache.org/jira/browse/HIVE-27683
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Priority: Major
>
> Steps to repro:
> {noformat}
> create database pt;
> create table pt.alterdynamic_part_table(intcol string) partitioned by
> (partcol1 int, partcol2 int);
> insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
> select '2', 2, NULL;
> select intcol from pt.alterdynamic_part_table where (partcol1=2 and
> partcol2=1) or (partcol1=2 and partcol2='__HIVE_DEFAULT_PARTITION__');
> set hive.cbo.enable=false;
> select intcol from pt.alterdynamic_part_table where
> partcol2='__HIVE_DEFAULT_PARTITION__';{noformat}
> The last two queries should return one line row instead of empty result.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27683) Incorrect result when filtering the table with default partition
Zhihua Deng created HIVE-27683:
--
Summary: Incorrect result when filtering the table with default
partition
Key: HIVE-27683
URL: https://issues.apache.org/jira/browse/HIVE-27683
Project: Hive
Issue Type: Bug
Reporter: Zhihua Deng
Steps to repro:
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
string, partcol2 string);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select 2, '2', NULL;
select intcol from pt.alterdynamic_part_table where
partcol2='__HIVE_DEFAULT_PARTITION__';{noformat}
The last query should return 2 instead of empty result.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27682) AlterTableAlterPartitionOperation cannot change the column type if the table has default partition
[
https://issues.apache.org/jira/browse/HIVE-27682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng updated HIVE-27682:
---
Description:
Steps to repro the case :
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
string, partcol2 string);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select NULL, '2', NULL;
alter table pt.alterdynamic_part_table partition column (partcol2
int);{noformat}
Exception is thrown:
{noformat}
org.apache.hadoop.hive.ql.metadata.HiveException: Exception while checking type
conversion of existing partition values to FieldSchema(name:partcol2, type:int,
comment:null) : Exception while converting string to int for value : NULL
at
org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.check(AlterTableAlterPartitionOperation.java:69)
at
org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.execute(AlterTableAlterPartitionOperation.java:55){noformat}
was:
Steps to repro the case :
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
string, partcol2 string);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select NULL, '2', 'NULL';
alter table pt.alterdynamic_part_table partition column (partcol2
int);{noformat}
Exception is thrown:
{noformat}
org.apache.hadoop.hive.ql.metadata.HiveException: Exception while checking type
conversion of existing partition values to FieldSchema(name:partcol2, type:int,
comment:null) : Exception while converting string to int for value : NULL
at
org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.check(AlterTableAlterPartitionOperation.java:69)
at
org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.execute(AlterTableAlterPartitionOperation.java:55){noformat}
> AlterTableAlterPartitionOperation cannot change the column type if the table
> has default partition
> --
>
> Key: HIVE-27682
> URL: https://issues.apache.org/jira/browse/HIVE-27682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
>
> Steps to repro the case :
> {noformat}
> create database pt;
> create table pt.alterdynamic_part_table(intcol string) partitioned by
> (partcol1 string, partcol2 string);
> insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
> select NULL, '2', NULL;
> alter table pt.alterdynamic_part_table partition column (partcol2
> int);{noformat}
> Exception is thrown:
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException: Exception while checking
> type conversion of existing partition values to FieldSchema(name:partcol2,
> type:int, comment:null) : Exception while converting string to int for value
> : NULL
> at
> org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.check(AlterTableAlterPartitionOperation.java:69)
> at
> org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.execute(AlterTableAlterPartitionOperation.java:55){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27682) AlterTableAlterPartitionOperation cannot change the column type if the table has default partition
[
https://issues.apache.org/jira/browse/HIVE-27682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng updated HIVE-27682:
---
Description:
Steps to repro the case :
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
string, partcol2 string);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select NULL, '2', 'NULL';
alter table pt.alterdynamic_part_table partition column (partcol2
int);{noformat}
Exception is thrown:
{noformat}
org.apache.hadoop.hive.ql.metadata.HiveException: Exception while checking type
conversion of existing partition values to FieldSchema(name:partcol2, type:int,
comment:null) : Exception while converting string to int for value : NULL
at
org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.check(AlterTableAlterPartitionOperation.java:69)
at
org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.execute(AlterTableAlterPartitionOperation.java:55){noformat}
was:
Steps to repro the case :
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
string, partcol2 string);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select NULL, '2', 'NULL';
alter table pt.alterdynamic_part_table partition column (partcol2
int);{noformat}
Exception is thrown:
{noformat}
org.apache.hadoop.hive.ql.metadata.HiveException: Exception while checking type
conversion of existing partition values to FieldSchema(name:partcol2, type:int,
comment:null) : Exception while converting string to int for value : NULL
at
org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.check(AlterTableAlterPartitionOperation.java:69)
at
org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.execute(AlterTableAlterPartitionOperation.java:55){noformat}
> AlterTableAlterPartitionOperation cannot change the column type if the table
> has default partition
> --
>
> Key: HIVE-27682
> URL: https://issues.apache.org/jira/browse/HIVE-27682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
>
> Steps to repro the case :
> {noformat}
> create database pt;
> create table pt.alterdynamic_part_table(intcol string) partitioned by
> (partcol1 string, partcol2 string);
> insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
> select NULL, '2', 'NULL';
> alter table pt.alterdynamic_part_table partition column (partcol2
> int);{noformat}
> Exception is thrown:
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException: Exception while checking
> type conversion of existing partition values to FieldSchema(name:partcol2,
> type:int, comment:null) : Exception while converting string to int for value
> : NULL
> at
> org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.check(AlterTableAlterPartitionOperation.java:69)
> at
> org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.execute(AlterTableAlterPartitionOperation.java:55){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27682) AlterTableAlterPartitionOperation cannot change the column type if the table has default partition
Zhihua Deng created HIVE-27682:
--
Summary: AlterTableAlterPartitionOperation cannot change the
column type if the table has default partition
Key: HIVE-27682
URL: https://issues.apache.org/jira/browse/HIVE-27682
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Zhihua Deng
Assignee: Zhihua Deng
Steps to repro the case :
{noformat}
create database pt;
create table pt.alterdynamic_part_table(intcol string) partitioned by (partcol1
string, partcol2 string);
insert into table pt.alterdynamic_part_table partition(partcol1, partcol2)
select NULL, '2', 'NULL';
alter table pt.alterdynamic_part_table partition column (partcol2
int);{noformat}
Exception is thrown:
{noformat}
org.apache.hadoop.hive.ql.metadata.HiveException: Exception while checking type
conversion of existing partition values to FieldSchema(name:partcol2, type:int,
comment:null) : Exception while converting string to int for value : NULL
at
org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.check(AlterTableAlterPartitionOperation.java:69)
at
org.apache.hadoop.hive.ql.ddl.table.partition.alter.AlterTableAlterPartitionOperation.execute(AlterTableAlterPartitionOperation.java:55){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27670) Failed to build the image locally on Apple silicon
[
https://issues.apache.org/jira/browse/HIVE-27670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng reassigned HIVE-27670:
--
Assignee: Zhihua Deng
> Failed to build the image locally on Apple silicon
> --
>
> Key: HIVE-27670
> URL: https://issues.apache.org/jira/browse/HIVE-27670
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
>
> When using build.sh to build the image, the arg HADOOP_VERSION is empty, and
> the build is failed as a consequence.
>
> {noformat}
> => ERROR [env 5/2] RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/
> && rm -rf /opt/hadoop-$HADOOP_VERSION 0.1s
> --
> > [env 5/2] RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/ &&
> rm -rf /opt/hadoop-$HADOOP_VERSION/share/doc/* && tar -xzvf
> /opt/apache-hive-$HIVE_VERSION-bin.tar.gz -C /opt/ && rm -rf
> /opt/apache-hive-$HIVE_VERSION-bin/jdbc/* && tar -xzvf
> /opt/apache-tez-$TEZ_VERSION-bin.tar.gz -C /opt && rm -rf
> /opt/apache-tez-$TEZ_VERSION-bin/share/*:
> #0 0.135 tar (child): /opt/hadoop-.tar.gz: Cannot open: No such file or
> directory{noformat}
> This is caused by the "--build-arg"s following --build-arg
> "BUILD_ENV=unarchive" weren't passed to the build command, we'd better add a
> '\' at the end of "BUILD_ENV=unarchive" to get rid of the problem.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27114) Provide a configurable filter for removing useless properties in Partition objects from listPartitions HMS Calls
[
https://issues.apache.org/jira/browse/HIVE-27114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng reassigned HIVE-27114:
--
Assignee: Zhihua Deng (was: Henri Biestro)
> Provide a configurable filter for removing useless properties in Partition
> objects from listPartitions HMS Calls
>
>
> Key: HIVE-27114
> URL: https://issues.apache.org/jira/browse/HIVE-27114
> Project: Hive
> Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Zhihua Deng
>Priority: Major
>
> HMS API calls are throwing following exception because of thrift upgrade
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
>
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:417)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1286)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java)
>
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:3290)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:3275)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1782)
>
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitions(SessionHiveMetaStoreClient.java:1134)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1775)
>
> at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source)
> ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_311]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:213)
>
> at com.sun.proxy.$Proxy52.listPartitions(Unknown Source) ~[?:?]
> at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source)
> ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_311]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3550)
>
> at com.sun.proxy.$Proxy52.listPartitions(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAllPartitionsOf(Hive.java:3793)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.getAllPartitions(PartitionPruner.java:485)
> {code}
> Large size partition metadata is causing this issue
> eg., impala stores huge stats chunk in partitionMetadata with {*}param_keys =
> (impala_intermediate_stats_chunk{*}{*}), these PARTITION_PARAM_KEYS are not
> required for Hive. These params should be skipped while preparing partition
> object from HMS to HS2.
> Similar to HIVE-25501, any user defined regex param_keys should be skipped in
> listPartitions HMS API call response.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27385) Backport of HIVE-22099: Several date related UDFs can't handle Julian dates properly since HIVE-20007
[ https://issues.apache.org/jira/browse/HIVE-27385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-27385. - Fix Version/s: 3.2.0 Assignee: Aman Raj (was: Diksha) Resolution: Fixed > Backport of HIVE-22099: Several date related UDFs can't handle Julian dates > properly since HIVE-20007 > - > > Key: HIVE-27385 > URL: https://issues.apache.org/jira/browse/HIVE-27385 > Project: Hive > Issue Type: Sub-task >Reporter: Diksha >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > > Backport of HIVE-22099: Several date related UDFs can't handle Julian dates > properly since HIVE-20007 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-22368) Hive JDBC Storage Handler: some mysql data type can not be cast to hive data type
[
https://issues.apache.org/jira/browse/HIVE-22368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763819#comment-17763819
]
James Boylan commented on HIVE-22368:
-
Are there any plans to include a fix for this in 3.1.X version codebase? With
4.0.0 in alpha state, many providers will not be ready to pull in that code,
and this is a breaking bug to leave unresolved in 3.1.3+
> Hive JDBC Storage Handler: some mysql data type can not be cast to hive data
> type
> -
>
> Key: HIVE-22368
> URL: https://issues.apache.org/jira/browse/HIVE-22368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Attachments: HIVE-22368.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Mysql data type（date、timestamp、decimal）can not be cast to hive data
> type（date、timestamp、decimal）。
> step to repo（take date type for example）:
> {code:java}
> //MySQL table:
> create table testdate(id date);
> //Hive table:
> CREATE EXTERNAL TABLE `hive_date`(
> id date )
> ROW FORMAT SERDE
> 'org.apache.hive.storage.jdbc.JdbcSerDe'
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='hive',
> 'hive.sql.dbcp.username'='hive',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
> 'hive.sql.table'='testdate');
> //Hive query:
> select * from hive_date;
> Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.sql.Date cannot be cast to
> org.apache.hadoop.hive.common.type.Date (state=,code=0)
> //Error stack trace
> Caused by: java.lang.ClassCastException: java.sql.Date cannot be cast to
> org.apache.hadoop.hive.common.type.Date
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaDateObjectInspector.getPrimitiveJavaObject(JavaDateObjectInspector.java:41)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaDateObjectInspector.getPrimitiveJavaObject(JavaDateObjectInspector.java:27)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:422)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.toThriftPayload(SerDeUtils.java:173)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.thrift.ThriftFormatter.convert(ThriftFormatter.java:49)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:94)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:519)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:511)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2706)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
>
[jira] [Updated] (HIVE-27322) Iceberg: metadata location overrides can cause data breach
[
https://issues.apache.org/jira/browse/HIVE-27322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-27322:
--
Labels: check pull-request-available (was: check)
> Iceberg: metadata location overrides can cause data breach
> --
>
> Key: HIVE-27322
> URL: https://issues.apache.org/jira/browse/HIVE-27322
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Affects Versions: 4.0.0-alpha-2
>Reporter: Janos Kovacs
>Priority: Blocker
> Labels: check, pull-request-available
>
> Set to bug/blocker instead of enhancement due to its security related nature,
> Hive4 should not be released w/o fix for this. Please reset if needed.
>
> Context:
> * There are some core tables with sensitive data that users can only query
> with data masking enforced (e.g. via Ranger). Let's assume this is the
> `default.icebergsecured` table.
> * An end-user can only access the masked form of the sensitive data as
> expected...
> * The users also have privilege to create new tables in their own sandbox
> databases - let's assume this is the `default.trojanhorse` table for now.
> * The user can create a malicious table that exposes the sensitive data
> non-masked leading to a possible data breach.
> * Hive runs with doAs=false to be able to enforce FGAC and prevent end-user
> direct file-system access needs
> Repro:
> * First make sure the data is secured by the masking policy:
> {noformat}
>
> beeline -e "
> DROP TABLE IF EXISTS default.icebergsecured PURGE;
> CREATE EXTERNAL TABLE default.icebergsecured (txt string, secret string)
> STORED BY ICEBERG;
> INSERT INTO default.icebergsecured VALUES ('You might be allowed to see
> this.','You are NOT allowed to see this!');
> "
>
> beeline -e "
> SELECT * FROM default.icebergsecured;
> "
> +++
> | icebergsecured.txt | icebergsecured.secret |
> +++
> | You might be allowed to see this. | MASKED BY RANGER FOR SECURITY |
> +++
> {noformat}
> * Now let the user to create the malicious table exposing the sensitive data:
> {noformat}
>
> SECURED_META_LOCATION=$(HADOOP_CLIENT_OPTS="-Djline.terminal=jline.UnsupportedTerminal"
> beeline -e "DESCRIBE FORMATTED default.icebergsecured;" 2>/dev/null |grep
> metadata_location |grep -v previous_metadata_location | awk '{print $5}')
> beeline -e "
> DROP TABLE IF EXISTS default.trojanhorse;
> CREATE EXTERNAL TABLE default.trojanhorse (txt string, secret string) STORED
> BY ICEBERG
> TBLPROPERTIES (
> 'metadata_location'='${SECURED_META_LOCATION}');
> SELECT * FROM default.trojanhorse;
> "
> ++---+
> | trojanhorse.txt |trojanhorse.secret |
> ++---+
> | You might be allowed to see this. | You are not allowed to see this! |
> ++---+
> {noformat}
>
> Currently - after HIVE-26707 - the rwstorage authorization only has either
> the dummy path or the explicit path set for uri:
> {noformat}
> Permission denied: user [oozie] does not have [RWSTORAGE] privilege on
> [iceberg://default/trojanhorse?snapshot=%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Ftrojanhorse%2Fmetadata%2Fdummy.metadata.json]
> Permission denied: user [oozie] does not have [RWSTORAGE] privilege on
> [iceberg://default/trojanhorse?snapshot=%2Fwarehouse%2Ftablespace%2Fexternal%2Fhive%2Ficebergsecured%2Fmetadata%2F1-f4c2a428-30ce-4afd-82ff-d46ecbf02244.metadata.json]
>
> {noformat}
> This is can be used only to decide whether a user is allowed to create
> iceberg tables in certain databases with certain names but controlling it's
> metadata location is hard in that form:
> * it does not provide a variable of "default table location" so a rule needs
> to know the per-database table location or per-catalog warehouse location to
> be able to construct it
> * it does not provide a rich regex to filter out `/../` style directory
> references
> * but basically there should be also a flag whether explicit metadata
> location is provided or not instead of the dummy reference, which then again
> needs explicit matching in the policy to handle
>
> Proposed enhancement:
> * The URL for the iceberg table's rwstorage authorization should be changed
> the following way
> ** the /? is good but
> *** the location should not be url encoded, or at least the authorizer
> should check the policy against the decoded url
> *** the
[jira] [Updated] (HIVE-23586) load data overwrite into bucket table failed
[ https://issues.apache.org/jira/browse/HIVE-23586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-23586: Labels: pull-request-available (was: hive-4.0.0-must pull-request-available) > load data overwrite into bucket table failed > > > Key: HIVE-23586 > URL: https://issues.apache.org/jira/browse/HIVE-23586 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 3.1.2, 4.0.0 >Reporter: zhaolong >Assignee: zhaolong >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23586.01.patch, image-2020-06-01-21-40-21-726.png, > image-2020-06-01-21-41-28-732.png > > Time Spent: 40m > Remaining Estimate: 0h > > load data overwrite into bucket table is failed if filename is not like > 00_0, but insert new data in the table. > > for example: > CREATE EXTERNAL TABLE IF NOT EXISTS test_hive2 (name string,account string) > PARTITIONED BY (logdate string) CLUSTERED BY (account) INTO 4 BUCKETS row > format delimited fields terminated by '|' STORED AS textfile; > load data inpath 'hdfs://hacluster/tmp/zltest' overwrite into table > default.test_hive2 partition (logdate='20200508'); > !image-2020-06-01-21-40-21-726.png! > load data inpath 'hdfs://hacluster/tmp/zltest' overwrite into table > default.test_hive2 partition (logdate='20200508');// should overwrite but > insert new data > !image-2020-06-01-21-41-28-732.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26220) Shade & relocate dependencies in hive-exec to avoid conflicting with downstream projects
[
https://issues.apache.org/jira/browse/HIVE-26220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HIVE-26220:
Labels: (was: hive-4.0.0-must)
> Shade & relocate dependencies in hive-exec to avoid conflicting with
> downstream projects
>
>
> Key: HIVE-26220
> URL: https://issues.apache.org/jira/browse/HIVE-26220
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 4.0.0-alpha-1
>Reporter: Chao Sun
>Assignee: Ayush Saxena
>Priority: Blocker
>
> Currently projects like Spark, Trino/Presto, Iceberg, etc, are depending on
> {{hive-exec:core}} which was removed in HIVE-25531. The reason these projects
> use {{hive-exec:core}} is because they have the flexibility to exclude, shade
> & relocate dependencies in {{hive-exec}} that conflict with the ones they
> brought in by themselves. However, with {{hive-exec}} this is no longer
> possible, since it is a fat jar that shade those dependencies but do not
> relocate many of them.
> In order for the downstream projects to consume {{hive-exec}}, we will need
> to make sure all the dependencies in {{hive-exec}} are properly shaded and
> relocated, so they won't cause conflicts with those from the downstream.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26537) Deprecate older APIs in the HMS
[ https://issues.apache.org/jira/browse/HIVE-26537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-26537: Labels: pull-request-available (was: hive-4.0.0-must pull-request-available) > Deprecate older APIs in the HMS > --- > > Key: HIVE-26537 > URL: https://issues.apache.org/jira/browse/HIVE-26537 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Critical > Labels: pull-request-available > Time Spent: 7h 10m > Remaining Estimate: 0h > > This Jira is to track the clean-up(deprecate older APIs and point the HMS > client to the newer APIs) work in the hive metastore server. > More details will be added here soon. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27681) select queries having Array UDFs throw errors when element type is varchar
[
https://issues.apache.org/jira/browse/HIVE-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Taraka Rama Rao Lethavadla updated HIVE-27681:
--
Summary: select queries having Array UDFs throw errors when element type is
varchar (was: Either insert operation or select queries having Array UDFs
throw errors when elment type is varchar)
> select queries having Array UDFs throw errors when element type is varchar
> --
>
> Key: HIVE-27681
> URL: https://issues.apache.org/jira/browse/HIVE-27681
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
>
> *Test Queries*
> {noformat}
> CREATE TABLE temp (top ARRAY);
> insert into temp values(array(cast('true' as VARCHAR(10;
> select array_contains(top,cast(true as VARCHAR(10))) from temp;
> {noformat}
>
> *Exception*
>
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Argument type
> mismatch ''TRUE'': "varchar(10)" expected at function ARRAY_CONTAINS, but
> "string" is found
> at
> org.apache.hadoop.hive.ql.parse.type.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1473){noformat}
> As per [Allowed Implicit
> Conversions|https://cwiki.apache.org/confluence/display/hive/languagemanual+types#LanguageManualTypes-AllowedImplicitConversions]
> String and varchar are convertible. So we can modify this check and allow
> varchar <-> String conversion
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27276) Enable debug options
[
https://issues.apache.org/jira/browse/HIVE-27276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Rátkai reassigned HIVE-27276:
Assignee: Zoltán Rátkai (was: Zhihua Deng)
> Enable debug options
>
>
> Key: HIVE-27276
> URL: https://issues.apache.org/jira/browse/HIVE-27276
> Project: Hive
> Issue Type: Sub-task
>Reporter: Denys Kuzmenko
>Assignee: Zoltán Rátkai
>Priority: Major
>
> {code}
> -p9866:9866 -p1:1 -p10001:10001 -p9000:9000 -p8000:8000 -p3306:3306
> -p50070:50070 -p50030:50030
> {code}
> For debug purpose, you can launch the container with:
> docker run -d -p 9083:9083 -p 8009:8009 --env
> SERVICE_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8009"
> --env SERVICE_NAME=metastore --name metastore-standalone
> apache/hive:4.0.0-SNAPSHOT
> SERVICE_OPTS will finally propagate to the JVM args of the service.
> PR for testing it out, #4240
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27681) Either insert operation or select queries having Array UDFs throw errors when elment type is varchar
Taraka Rama Rao Lethavadla created HIVE-27681:
-
Summary: Either insert operation or select queries having Array
UDFs throw errors when elment type is varchar
Key: HIVE-27681
URL: https://issues.apache.org/jira/browse/HIVE-27681
Project: Hive
Issue Type: Sub-task
Components: HiveServer2
Reporter: Taraka Rama Rao Lethavadla
Assignee: Taraka Rama Rao Lethavadla
*Test Queries*
{noformat}
CREATE TABLE temp (top ARRAY);
insert into temp values(array(cast('true' as VARCHAR(10;
select array_contains(top,cast(true as VARCHAR(10))) from temp;
{noformat}
*Exception*
{noformat}
org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Argument type
mismatch ''TRUE'': "varchar(10)" expected at function ARRAY_CONTAINS, but
"string" is found
at
org.apache.hadoop.hive.ql.parse.type.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1473){noformat}
As per [Allowed Implicit
Conversions|https://cwiki.apache.org/confluence/display/hive/languagemanual+types#LanguageManualTypes-AllowedImplicitConversions]
String and varchar are convertible. So we can modify this check and allow
varchar <-> String conversion
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27567) Support building multi-platform images
[ https://issues.apache.org/jira/browse/HIVE-27567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Rátkai reassigned HIVE-27567: Assignee: Simhadri Govindappa > Support building multi-platform images > -- > > Key: HIVE-27567 > URL: https://issues.apache.org/jira/browse/HIVE-27567 > Project: Hive > Issue Type: Sub-task >Reporter: Zhihua Deng >Assignee: Simhadri Govindappa >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27276) Enable debug options
[
https://issues.apache.org/jira/browse/HIVE-27276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763682#comment-17763682
]
Zoltán Rátkai commented on HIVE-27276:
--
[~dengzh] are you working on it, or can I assign it to myself?
> Enable debug options
>
>
> Key: HIVE-27276
> URL: https://issues.apache.org/jira/browse/HIVE-27276
> Project: Hive
> Issue Type: Sub-task
>Reporter: Denys Kuzmenko
>Assignee: Zhihua Deng
>Priority: Major
>
> {code}
> -p9866:9866 -p1:1 -p10001:10001 -p9000:9000 -p8000:8000 -p3306:3306
> -p50070:50070 -p50030:50030
> {code}
> For debug purpose, you can launch the container with:
> docker run -d -p 9083:9083 -p 8009:8009 --env
> SERVICE_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8009"
> --env SERVICE_NAME=metastore --name metastore-standalone
> apache/hive:4.0.0-SNAPSHOT
> SERVICE_OPTS will finally propagate to the JVM args of the service.
> PR for testing it out, #4240
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27669) Hive Acid CTAS fails incremental if no of rows inserted is > INT_MAX
[
https://issues.apache.org/jira/browse/HIVE-27669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harshal Patel updated HIVE-27669:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Hive Acid CTAS fails incremental if no of rows inserted is > INT_MAX
>
>
> Key: HIVE-27669
> URL: https://issues.apache.org/jira/browse/HIVE-27669
> Project: Hive
> Issue Type: Bug
>Reporter: Harshal Patel
>Assignee: Harshal Patel
>Priority: Major
> Labels: pull-request-available
>
> * If a Table is created using CTAS with rows > INT_MAX then beeline eats up
> the thrown error
> * As replication also uses the same infra it should also do the same
> instead of failing with NumberFormatException
> *Note:* This is happening in the customer's environment consistently but we
> are not able to reproduce it. So, we have gone through the whole code flow
> and handled the error accordingly.
>
> Error message while incremental replication:
> {code:java}
> 4:12:03.230 PMINFODriver [Scheduled Query
> Executor(schedule:repl_sample_acid_1, execution_id:49625)]: Starting task
> [Stage-10066:REPL_STATE_LOG] in serial mode4:12:03.231 PMINFO
> ReplState [Scheduled Query Executor(schedule:repl_sample_acid_1,
> execution_id:49625)]: REPL::EVENT_LOAD:
> {"dbName":"sample","eventId":"50442182","eventType":"EVENT_ALLOC_WRITE_ID","eventsLoadProgress":"2443/20424","loadTime":1687187523,"eventDuration":"159
> ms"}4:12:03.231 PM INFODriver [Scheduled Query
> Executor(schedule:repl_sample_acid_1, execution_id:49625)]: Starting task
> [Stage-10067:COLUMNSTATS] in serial mode4:12:03.488 PM INFODriver
> [Scheduled Query Executor(schedule:repl_sample_acid_1, execution_id:49625)]:
> Starting task [Stage-10068:DEPENDENCY_COLLECTION] in serial mode4:12:03.488
> PM INFODriver [Scheduled Query Executor(schedule:repl_sample_acid_1,
> execution_id:49625)]: Starting task [Stage-10069:DDL] in serial
> mode4:12:03.504 PM INFODriver [Scheduled Query
> Executor(schedule:repl_sample_acid_1, execution_id:49625)]: Starting task
> [Stage-10070:REPL_STATE_LOG] in serial mode4:12:03.504 PMINFO
> ReplState [Scheduled Query Executor(schedule:repl_sample_acid_1,
> execution_id:49625)]: REPL::EVENT_LOAD:
> {"dbName":"sample","eventId":"50442183","eventType":"EVENT_UPDATE_TABLE_COL_STAT","eventsLoadProgress":"2444/20424","loadTime":1687187523,"eventDuration":"273
> ms"}4:12:03.504 PMINFODriver [Scheduled Query
> Executor(schedule:repl_sample_acid_1, execution_id:49625)]: Starting task
> [Stage-10071:DDL] in serial mode4:12:03.596 PM ERROR Task
> [Scheduled Query Executor(schedule:repl_sample_acid_1, execution_id:49625)]:
> Failedorg.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter
> table. java.lang.NumberFormatException: For input string: "5744479373" at
> org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:854)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5] at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableReplaceMode(CreateTableOperation.java:127)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5] at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:90)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5] at
> org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:82)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5] at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5] at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5]at
> org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5]at
> org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5] at
> org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5] at
> org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5] at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:772)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5] at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:511)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5] at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:505)
> ~[hive-exec-3.1.3000.7.1.8.15-5.jar:3.1.3000.7.1.8.15-5] at
>
[jira] [Created] (HIVE-27680) Statistics observation and new features
Adriano created HIVE-27680: -- Summary: Statistics observation and new features Key: HIVE-27680 URL: https://issues.apache.org/jira/browse/HIVE-27680 Project: Hive Issue Type: New Feature Reporter: Adriano Dear Team, Currently, Hive statistics are consistently computed for all tables, leaving us with only the option to prevent these stats from being calculated by setting "hive.stats.autogather=false" either in the client session or globally. We lack the capability to selectively delete statistics table/columns from specific tables. While these statistics are beneficial for the planner in many cases, there are situations where enabling them for all tables can be suboptimal. I've observed that, for some tables, the size of these statistics can grow significantly, potentially leading to Out Of Memory (OOM) issues. The Apache Hive documentation does not provide a straightforward method for manually adding or updating statistics and column statistics. It would greatly benefit us if the following features could be considered: 1- An option that can be configured in the TBLPROPERTIES to disable statistics computation for specific tables, holding it active for the rest. 2- A command to drop statistics for tables/columns for specific tables. 3- The documentation lacks a clear procedure for manually adding table/column statistics. Although it might be possible to determine this through trial and error, it would be much more efficient to have a documented process and a command available, similar to many other settings. We kindly request that you evaluate this proposal and consider the possibility of further development to incorporate these features into the roadmap. Doing so would greatly improve and enhance both the product and its accompanying documentation. Thank you for your attention to this matter. Sincerely, Adriano -- This message was sent by Atlassian Jira (v8.20.10#820010)
