[
https://issues.apache.org/jira/browse/HIVE-24987?focusedWorklogId=604694&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-604694
]
ASF GitHub Bot logged work on HIVE-24987:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Jun/21 18:36
Start Date: 01/Jun/21 18:36
Worklog Time Spent: 10m
Work Description: vihangk1 opened a new pull request #2336:
URL: https://github.com/apache/hive/pull/2336
### What changes were proposed in this pull request?
hive.metastore.disallow.incompatible.col.type.check config currently checks
if a alter table operation is making a incompatible schema change to the table.
By default it is set to true which would error out the alter table call when
such a change is detected. However, this change is too restrictive for certain
file-formats like Kudu. In case of Kudu, it is allowed to drop a column which
could result in the schema to be incompatible according to current
implementation of this check. This causes a bad user-experience for Kudu users
and there is no real work-around than to disable this check all together.
Disabling the check is not an option since for file-formats like Parquet this
is should be true to avoid data corruption/incorrect results.
This change introduces a new config which can be used by users to provide a
exception list based on table serde library name. If a table belongs to such a
serde, the check is skipped. By default currently, only Kudu tables are added
to this config.
### Why are the changes needed?
See above.
### Does this PR introduce _any_ user-facing change?
This introduces a new configuration option for metastore.
### How was this patch tested?
A new unit-test was added to exercise the specific use-case. Existing tests
make sure that previous behavior is not changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 604694)
Remaining Estimate: 0h
Time Spent: 10m
> hive.metastore.disallow.incompatible.col.type.changes is too restrictive for
> some storage formats
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-24987
> URL: https://issues.apache.org/jira/browse/HIVE-24987
> Project: Hive
> Issue Type: Improvement
> Reporter: Vihang Karajgaonkar
> Assignee: Vihang Karajgaonkar
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently when {{hive.metastore.disallow.incompatible.col.type.changes}} is
> set to true it disallows any schema changes which are deemed as backwards
> incompatible e.g dropping a column of a table. While this may be a correct
> thing to do for Parquet or Orc tables, it is too restrictive for storage
> formats like Kudu.
> Currently, for Kudu tables, Impala supports dropping a column. But if we set
> this config to true metastore disallows changing the schema of the metastore
> table. I am assuming this would be problematic for Iceberg tables too which
> supports such schema changes.
> The proposal is to have a new configuration which provided a exclusion list
> of the table fileformat where this check will be skipped. Currently, we will
> only include Kudu tables to skip this check.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
