boroknagyz commented on a change in pull request #2948:
URL: https://github.com/apache/hive/pull/2948#discussion_r789669255
##########
File path:
iceberg/iceberg-handler/src/test/queries/positive/describe_iceberg_table.q
##########
@@ -8,7 +8,7 @@ DROP TABLE IF EXISTS ice_t_transform;
CREATE EXTERNAL TABLE ice_t_transform (year_field date, month_field date,
day_field date, hour_field timestamp, truncate_field string, bucket_field int,
identity_field int) PARTITIONED BY SPEC (year(year_field), month(month_field),
day(day_field), hour(hour_field), truncate(2, truncate_field), bucket(2,
bucket_field), identity_field) STORED BY ICEBERG;
DROP TABLE IF EXISTS ice_t_transform_prop;
-CREATE EXTERNAL TABLE ice_t_transform_prop (id int, year_field date,
month_field date, day_field date, hour_field timestamp, truncate_field string,
bucket_field int, identity_field int) STORED BY ICEBERG TBLPROPERTIES
('iceberg.mr.table.partition.spec'='{"spec-id":0,"fields":[{"name":"year_field_year","transform":"year","source-id":1,"field-id":1000},{"name":"month_field_month","transform":"month","source-id":2,"field-id":1001},{"name":"day_field_day","transform":"day","source-id":3,"field-id":1002},{"name":"hour_field_hour","transform":"hour","source-id":4,"field-id":1003},{"name":"truncate_field_trunc","transform":"truncate[2]","source-id":5,"field-id":1004},{"name":"bucket_field_bucket","transform":"bucket[2]","source-id":6,"field-id":1005},{"name":"identity_field","transform":"identity","source-id":7,"field-id":1006}]}');
+CREATE EXTERNAL TABLE ice_t_transform_prop (id int, year_field date,
month_field date, day_field date, hour_field timestamp, truncate_field string,
bucket_field int, identity_field int) STORED BY ICEBERG TBLPROPERTIES
('iceberg.mr.table.partition.spec'='{"spec-id":0,"fields":[{"name":"year_field_year","transform":"year","source-id":2,"field-id":1000},{"name":"month_field_month","transform":"month","source-id":3,"field-id":1001},{"name":"day_field_day","transform":"day","source-id":4,"field-id":1002},{"name":"hour_field_hour","transform":"hour","source-id":5,"field-id":1003},{"name":"truncate_field_trunc","transform":"truncate[2]","source-id":6,"field-id":1004},{"name":"bucket_field_bucket","transform":"bucket[2]","source-id":7,"field-id":1005},{"name":"identity_field","transform":"identity","source-id":8,"field-id":1006}]}');
Review comment:
Prior to this patch `HiveSchemaConverter` used 0-based indexing when it
assigned the field ids. E.g. in the above statement it would assign field id 0
to `id`, field id 1 to `year_field`, and so on. Hence in
'iceberg.mr.table.partition.spec' the source-id 1 referred to the `year_field`.
Everything was fine, but when Iceberg creates a table it reassigns the field
ids using 1-based indexing (field id 1 is `id`, field id 2 is `year_field`).
And Iceberg is smart enough to use the correct ids in the partition spec, i.e.
it replaces source id 1 to source id 2 and so on.
So everything worked OK, but you had to specify different field/source ids
in Hive than the actual field/source ids assigned by Iceberg.
With this change, you need to use the same 1-based indexing in the partition
spec that Iceberg will use later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
