[
https://issues.apache.org/jira/browse/HIVE-13515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13515:
--------------------------------
Description:
I had this test case in my new Schema Evolution patch (not checked in yet) that
verifies STRING_GROUP to DECIMAL conversion:
{code}
-- SUBSECTION: ALTER TABLE CHANGE COLUMNS for STRING_GROUP -> DECIMAL: (STRING,
CHAR, VARCHAR)
--
CREATE TABLE table_change_string_group_decimal(insert_num int, c1 STRING, c2
CHAR(50), c3 VARCHAR(50), b STRING);
insert into table table_change_string_group_decimal
values (1, '753.7028', '753.7028', '753.7028', 'original'),
(2, '-3651.672121', '-3651.672121', '-3651.672121', 'original'),
(3, '-29.0764', '-29.0764', '-29.0764', 'original'),
(4, '-10.3', '-10.3', '-10.3', 'original');
-- Table-Non-Cascade CHANGE COLUMNS ...
alter table table_change_string_group_decimal replace columns (insert_num int,
c1 decimal(38,18), c2 decimal(38,18), c3 decimal(38,18), b STRING);
insert into table table_change_string_group_decimal
values (5, 30.774, 30.774, 30.774, 'new'),
(6, 20.31, 20.31, 20.31, 'new'),
(7, 46114.284799488, 46114.284799488, 46114.284799488, 'new'),
(8, -66475.561431, -66475.561431, -66475.561431, 'new');
insert into table table_change_string_group_decimal
values (9, 17808.963785, 17808.963785, 17808.963785, 'new'),
(10, 9250340.75 , 9250340.75 , 9250340.75 , 'new');
explain
select insert_num,c1,c2,c3,b from table_change_string_group_decimal order by
insert_num;
select insert_num,c1,c2,c3,b from table_change_string_group_decimal order by
insert_num;
drop table table_change_string_group_decimal;
{code}
that now fails on master (I suspect because of HIVE-13380 Decimal should have
lower precedence than double in type hierachy) when:
{code}
set hive.metastore.disallow.incompatible.col.type.changes=true;
{code}
with this call stack:
{code}
Caused by: InvalidOperationException(message:The following columns have types
incompatible with the existing columns in their respective positions :
c1,c2,c3)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.throwExceptionIfIncompatibleColTypeChange(MetaStoreUtils.java:599)
at
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:132)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3502)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3473)
{code}
Key point is MetaStoreUtils.throwExceptionIfIncompatibleColTypeChange is
calling TypeInfoUtils.areColTypesCompatible which has these lines removed:
{code}
- // Allow implicit String to Decimal conversion
- if (fromPg == PrimitiveGrouping.STRING_GROUP && to ==
PrimitiveCategory.DECIMAL) {
- return true;
- }
{code}
Will not current customers that have scripts with
hive.metastore.disallow.incompatible.col.type.changes=true that do Schema
Evolution get bitten?
was:
I had this test case in my new Schema Evolution patch (not checked in yet) that
verifies STRING_GROUP to DECIMAL conversion:
{code}
-- SUBSECTION: ALTER TABLE CHANGE COLUMNS for STRING_GROUP -> DECIMAL: (STRING,
CHAR, VARCHAR)
--
CREATE TABLE table_change_string_group_decimal(insert_num int, c1 STRING, c2
CHAR(50), c3 VARCHAR(50), b STRING);
insert into table table_change_string_group_decimal
values (1, '753.7028', '753.7028', '753.7028', 'original'),
(2, '-3651.672121', '-3651.672121', '-3651.672121', 'original'),
(3, '-29.0764', '-29.0764', '-29.0764', 'original'),
(4, '-10.3', '-10.3', '-10.3', 'original');
-- Table-Non-Cascade CHANGE COLUMNS ...
alter table table_change_string_group_decimal replace columns (insert_num int,
c1 decimal(38,18), c2 decimal(38,18), c3 decimal(38,18), b STRING);
insert into table table_change_string_group_decimal
values (5, 30.774, 30.774, 30.774, 'new'),
(6, 20.31, 20.31, 20.31, 'new'),
(7, 46114.284799488, 46114.284799488, 46114.284799488, 'new'),
(8, -66475.561431, -66475.561431, -66475.561431, 'new');
insert into table table_change_string_group_decimal
values (9, 17808.963785, 17808.963785, 17808.963785, 'new'),
(10, 9250340.75 , 9250340.75 , 9250340.75 , 'new');
explain
select insert_num,c1,c2,c3,b from table_change_string_group_decimal order by
insert_num;
select insert_num,c1,c2,c3,b from table_change_string_group_decimal order by
insert_num;
drop table table_change_string_group_decimal;
{code}
that now fails on master (I suspect because of HIVE-13380 Decimal should have
lower precedence than double in type hierachy) when:
{code}
set hive.metastore.disallow.incompatible.col.type.changes=true;
{code}
with this call stack:
{code}
Caused by: InvalidOperationException(message:The following columns have types
incompatible with the existing columns in their respective positions :
c1,c2,c3)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.throwExceptionIfIncompatibleColTypeChange(MetaStoreUtils.java:599)
at
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:132)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3502)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3473)
{code}
Key point is MetaStoreUtils.throwExceptionIfIncompatibleColTypeChange is
calling TypeInfoUtils.areColTypesCompatible which changed
- // Allow implicit String to Decimal conversion
- if (fromPg == PrimitiveGrouping.STRING_GROUP && to ==
PrimitiveCategory.DECIMAL) {
- return true;
- }
Will not current customers that have scripts with
hive.metastore.disallow.incompatible.col.type.changes=true that do Schema
Evolution get bitten?
> Semantics of TypeInfoUtils.areColTypesCompatible changed which also changed
> the semantics of hive.metastore.disallow.incompatible.col.type.changes
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-13515
> URL: https://issues.apache.org/jira/browse/HIVE-13515
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 2.1.0
> Reporter: Matt McCline
> Assignee: Ashutosh Chauhan
> Priority: Critical
>
> I had this test case in my new Schema Evolution patch (not checked in yet)
> that verifies STRING_GROUP to DECIMAL conversion:
> {code}
> -- SUBSECTION: ALTER TABLE CHANGE COLUMNS for STRING_GROUP -> DECIMAL:
> (STRING, CHAR, VARCHAR)
> --
> CREATE TABLE table_change_string_group_decimal(insert_num int, c1 STRING, c2
> CHAR(50), c3 VARCHAR(50), b STRING);
> insert into table table_change_string_group_decimal
> values (1, '753.7028', '753.7028', '753.7028', 'original'),
> (2, '-3651.672121', '-3651.672121', '-3651.672121', 'original'),
> (3, '-29.0764', '-29.0764', '-29.0764', 'original'),
> (4, '-10.3', '-10.3', '-10.3', 'original');
> -- Table-Non-Cascade CHANGE COLUMNS ...
> alter table table_change_string_group_decimal replace columns (insert_num
> int, c1 decimal(38,18), c2 decimal(38,18), c3 decimal(38,18), b STRING);
> insert into table table_change_string_group_decimal
> values (5, 30.774, 30.774, 30.774, 'new'),
> (6, 20.31, 20.31, 20.31, 'new'),
> (7, 46114.284799488, 46114.284799488, 46114.284799488, 'new'),
> (8, -66475.561431, -66475.561431, -66475.561431, 'new');
> insert into table table_change_string_group_decimal
> values (9, 17808.963785, 17808.963785, 17808.963785, 'new'),
> (10, 9250340.75 , 9250340.75 , 9250340.75 , 'new');
> explain
> select insert_num,c1,c2,c3,b from table_change_string_group_decimal order by
> insert_num;
> select insert_num,c1,c2,c3,b from table_change_string_group_decimal order by
> insert_num;
> drop table table_change_string_group_decimal;
> {code}
> that now fails on master (I suspect because of HIVE-13380 Decimal should have
> lower precedence than double in type hierachy) when:
> {code}
> set hive.metastore.disallow.incompatible.col.type.changes=true;
> {code}
> with this call stack:
> {code}
> Caused by: InvalidOperationException(message:The following columns have types
> incompatible with the existing columns in their respective positions :
> c1,c2,c3)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.throwExceptionIfIncompatibleColTypeChange(MetaStoreUtils.java:599)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:132)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3502)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3473)
> {code}
> Key point is MetaStoreUtils.throwExceptionIfIncompatibleColTypeChange is
> calling TypeInfoUtils.areColTypesCompatible which has these lines removed:
> {code}
> - // Allow implicit String to Decimal conversion
> - if (fromPg == PrimitiveGrouping.STRING_GROUP && to ==
> PrimitiveCategory.DECIMAL) {
> - return true;
> - }
> {code}
> Will not current customers that have scripts with
> hive.metastore.disallow.incompatible.col.type.changes=true that do Schema
> Evolution get bitten?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
