[ https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dilip Joseph updated HIVE-4409: ------------------------------- Description: If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter. Example: CREATE TABLE test_table123 (a INT, b MAP<STRING, STRING>) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE; INSERT OVERWRITE TABLE test_table123 PARTITION(ds="foo1") SELECT 1, MAP("a1", "b1") FROM src LIMIT 1; SELECT * from test_table123 WHERE ds="foo1"; ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); SELECT * from test_table123 WHERE ds="foo1"; The last SELECT fails with the following exception: Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) The ALTER TABLE statement is blocked if you set the following parameter, introduced int the fix to this JIRA: SET hive.metastore.disallow.incompatible.col.type.changes=true; was: If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter. Example: CREATE TABLE test_table123 (a INT, b MAP<STRING, STRING>) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE; INSERT OVERWRITE TABLE test_table123 PARTITION(ds="foo1") SELECT 1, MAP("a1", "b1") FROM src LIMIT 1; SELECT * from test_table123 WHERE ds="foo1"; SET hive.metastore.disallow.invalid.col.type.changes=true; ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); SELECT * from test_table123 WHERE ds="foo1"; The last SELECT fails with the following exception: Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) > Prevent incompatible column type changes > ---------------------------------------- > > Key: HIVE-4409 > URL: https://issues.apache.org/jira/browse/HIVE-4409 > Project: Hive > Issue Type: Improvement > Components: CLI, Metastore > Affects Versions: 0.10.0 > Reporter: Dilip Joseph > Assignee: Dilip Joseph > Priority: Minor > Fix For: 0.12.0 > > Attachments: hive.4409.1.patch, HIVE-4409.D10539.1.patch, > HIVE-4409.D10539.2.patch > > > If a user changes the type of an existing column of a partitioned table to an > incompatible type, subsequent accesses of old partitions will result in a > ClassCastException (see example below). We should prevent the user from > making incompatible type changes. This feature will be controlled by a new > config parameter. > Example: > CREATE TABLE test_table123 (a INT, b MAP<STRING, STRING>) PARTITIONED BY (ds > STRING) STORED AS SEQUENCEFILE; > INSERT OVERWRITE TABLE test_table123 PARTITION(ds="foo1") SELECT 1, MAP("a1", > "b1") FROM src LIMIT 1; > SELECT * from test_table123 WHERE ds="foo1"; > ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); > SELECT * from test_table123 WHERE ds="foo1"; > The last SELECT fails with the following exception: > Failed with exception java.io.IOException:java.lang.ClassCastException: > org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector > cannot be cast to > org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector > java.io.IOException: java.lang.ClassCastException: > org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector > cannot be cast to > org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) > at > org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) > at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) > at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) > at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) > at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) > The ALTER TABLE statement is blocked if you set the following parameter, > introduced int the fix to this JIRA: > SET hive.metastore.disallow.incompatible.col.type.changes=true; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira