[
https://issues.apache.org/jira/browse/IMPALA-12209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18040450#comment-18040450
]
ASF subversion and git services commented on IMPALA-12209:
----------------------------------------------------------
Commit 3d22c7fe052010c6b4012a5cc664a5ca1d8a0b03 in impala's branch
refs/heads/master from Daniel Vanko
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3d22c7fe0 ]
IMPALA-12209: Always include format-version in DESCRIBE FORMATTED and SHOW
CREATE TABLE for Iceberg tables
HiveCatalog does not include format-version for Iceberg tables in the
table's parameters, therefore the output of SHOW CREATE TABLE may not
replicate the original table.
This patch makes sure to add it to both the SHOW CREATE TABLE and
DESCRIBE FORMATTED/EXTENDED output.
Additionally, adds ICEBERG_DEFAULT_FORMAT_VERSION variable to E2E
tests, deducting from IMPALA_ICEBERG_VERSION environment variable.
If Iceberg version is at least 1.4, default format-version is 2, before
1.4 it's 1. This way tests can work with multiple Iceberg versions.
Testing:
* updated show-create-table.test and show-create-table-with-stats.test
for Iceberg tables
* added format-version checks to multiple DESCRIBE FORMATTED tests
Change-Id: I991edf408b24fa73e8a8abe64ac24929aeb8e2f8
Reviewed-on: http://gerrit.cloudera.org:8080/23514
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> format-version is not present in DESCRIBE FORMATTED and SHOW CREATE TABLE
> outputs
> ---------------------------------------------------------------------------------
>
> Key: IMPALA-12209
> URL: https://issues.apache.org/jira/browse/IMPALA-12209
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Gabor Kaszab
> Assignee: Dániel Gábor Vankó
> Priority: Major
> Labels: impala-iceberg, ramp-up
>
> Repro:
>
> {code:java}
> create table tmp (i int, s string) stored as iceberg tblproperties
> ('format-version'='2');
> describe extended/formatted tmp;
> show create table tmp;
> {code}
> Current behaviour:
> None of the following 2 commands contain 'format-version' in the output.
> Additionally, if you run what is returned from SHOW CREATE TABLE then you end
> up creating a V1 table instead of V2.
> The reason might be that format-version in the metadata.json is not stored
> within the tableproperties but it's on level above:
> {code:java}
> hdfs dfs -cat
> hdfs://localhost:20500/test-warehouse/tmp/metadata/00000-55bcfe84-1819-4fb7-ade8-9c132b117880.metadata.json
> {
> "format-version" : 2,
> "table-uuid" : "9f11c0c4-02c7-4688-823c-fe95dbe3ff72",
> "location" : "hdfs://localhost:20500/test-warehouse/tmp",
> "last-sequence-number" : 0,
> "last-updated-ms" : 1686640775184,
> "last-column-id" : 2,
> "current-schema-id" : 0,
> "schemas" : [ {
> "type" : "struct",
> "schema-id" : 0,
> "fields" : [ {
> "id" : 1,
> "name" : "i",
> "required" : false,
> "type" : "int"
> }, {
> "id" : 2,
> "name" : "s",
> "required" : false,
> "type" : "string"
> } ]
> } ],
> "default-spec-id" : 0,
> "partition-specs" : [ {
> "spec-id" : 0,
> "fields" : [ ]
> } ],
> "last-partition-id" : 999,
> "default-sort-order-id" : 0,
> "sort-orders" : [ {
> "order-id" : 0,
> "fields" : [ ]
> } ],
> "properties" : {
> "engine.hive.enabled" : "true",
> "external.table.purge" : "TRUE",
> "write.merge.mode" : "merge-on-read",
> "write.format.default" : "parquet",
> "write.delete.mode" : "merge-on-read",
> "OBJCAPABILITIES" : "EXTREAD,EXTWRITE",
> "write.update.mode" : "merge-on-read",
> "storage_handler" : "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler"
> },
> "current-snapshot-id" : -1,
> "refs" : { },
> "snapshots" : [ ],
> "statistics" : [ ],
> "snapshot-log" : [ ],
> "metadata-log" : [ ]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
