[
https://issues.apache.org/jira/browse/IMPALA-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-7309:
-----------------------------------
Epic Link: IMPALA-12887
> Prevent the addition of Avro partitions to non-Avro tables with incompatible
> schema
> -----------------------------------------------------------------------------------
>
> Key: IMPALA-7309
> URL: https://issues.apache.org/jira/browse/IMPALA-7309
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Frontend
> Reporter: Todd Lipcon
> Priority: Major
> Labels: catalog-v2
>
> Per a recent [mailing list
> thread|https://lists.apache.org/thread.html/fb68c54bd66a40982ee17f9f16f87a4112220a5df035a311bda310f1@<dev.impala.apache.org>]
> the behavior of Avro partitions within non-Avro tables is inconsistent with
> Hive, and somewhat suprising. For example, the addition of a partition can
> cause the results of "describe" on the table to change, but only after a
> refresh or invalidate. In the mailing list thread, we decided to change the
> behavior to:
> 1. Schema handling:
> - if a table's properties indicate it's an avro table, parse and adopt the
> external avro schema as the table schema, or infer an avro-compatible schema
> from the existing columns
> - if a table's properties indicate it's _not_ an avro table, but there is
> an external avro schema defined in the table properties, then parse the
> avro schema and include it in the TableDescriptor (for use by avro
> partitions) but *do not* adopt it as the table schema.
> 2. Handling incompatible schemas:
> - If the table-level format is non-Avro,
> - AND the table contains column types incompatible with Avro (eg tinyint),
> - AND the table has an existing avro partition,
> - THEN the query will yield an error about incompatible types
> 3. Try to prevent shooting in the foot
> - If the table-level format is non-Avro,
> - AND the table contains column types incompatible with Avro (eg tinyint),
> - THEN disallow changing the file format of an existing partition to Avro
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
