architjainjain opened a new pull request, #6619:
URL: https://github.com/apache/hive/pull/6619
# HIVE-29749: Fix special characters in Iceberg partition column names with
quoted identifiers
### What changes were proposed in this pull request?
This PR fixes support for special characters (e.g. `!@#$%^&*()`) in Iceberg
partition column names
when using backtick-quoted identifiers
(`hive.support.quoted.identifiers=column`).
Two changes were made:
1. **`TransformSpec.java` — `toNamedStruct()`**: When generating the
`named_struct(...)` SQL
expression for Iceberg partition transforms during column stats
auto-gathering, the partition
field name is now wrapped with `unparseIdentifier()` so it is
backtick-quoted. This ensures the
resulting struct type string contains backtick-delimited field names
(e.g. `` `gpa_!@#$%^&*()` ``) that can be correctly parsed downstream.
2. **`TypeInfoUtils.java` — `tokenize()`**: The tokenizer now handles
backtick-quoted identifiers
as a single token. When a backtick is encountered, all characters up to
the closing backtick are
consumed as one token with `isType=true`, stripping the backticks from
the token text. This
allows `TypeInfoParser` to correctly parse struct type strings containing
field names with
special characters.
3. **`QuotedIdentifier_1.q`**: Expanded the Q test to cover multiple
scenarios with special
character column and partition names.
---
### Why are the changes needed?
When an Iceberg table is created with a backtick-quoted partition column
containing special
characters (e.g. `` `gpa_!@#$%^&*()` ``), the partition field name is stored
in the Iceberg
metadata as a raw unquoted string (`gpa_!@#$%^&*()`).
During an `INSERT` into such a table, Hive's `ColumnStatsAutoGatherContext`
triggers automatic
column statistics gathering. This calls `TransformSpec.toNamedStruct()`
which builds a
`named_struct(...)` SQL expression. The resulting struct type
(e.g. `struct<gpa_!@#$%^&*():double>`) is then serialized and passed to
`TypeInfoUtils.getTypeInfosFromTypeString()`, where `tokenize()` attempts to
parse it.
The tokenizer splits `!`, `@`, `#`, etc. into individual tokens since they
are not `isTypeChar()`
characters. When `parseType()` then calls `expect(":")` to find the
name/type separator, it
encounters `!` instead and throws:
```
IllegalArgumentException: Error: ':' expected at position 137 of
'...struct<gpa_!@#$%^&*():double>' but '!' is found.
```
The fix ensures the field name is backtick-quoted when the struct type
string is generated, and
that the tokenizer can correctly handle backtick-quoted tokens.
---
### Does this PR introduce _any_ user-facing change?
Yes. Previously, executing an `INSERT` into an Iceberg table with special
characters in partition
column names would fail with a `SemanticException` when
`hive.stats.autogather=true` (the
default). After this fix, such operations succeed correctly.
**Before:**
```sql
set hive.support.quoted.identifiers=column;
create table t(`col` string) partitioned by(`gpa_!@#$%^&*()` double) stored
by iceberg;
insert into t select name, gpa from src;
-- ERROR: SemanticException: Error: ':' expected at position 137 of
-- '...struct<gpa_!@#$%^&*():double>' but '!' is found.
```
**After:**
```sql
-- Same DDL and INSERT succeeds without error.
```
---
### How was this patch tested?
The existing Q test `QuotedIdentifier_1.q` was expanded with the following
additional scenarios,
all using `!@#$%^&*()` special characters in column and partition names:
1. **Basic INSERT** with special chars in both regular columns and partition
column
2. **View creation and count** on a table with all-special-char columns and
partition
3. **Multiple partition columns** both containing special chars
4. **`ANALYZE TABLE COMPUTE STATISTICS FOR COLUMNS`** — directly exercises
the
`ColumnStatsAutoGatherContext` → `TransformSpec.toNamedStruct()` →
`TypeInfoUtils.tokenize()`
path that was the root cause of the bug
Test run command:
```bash
mvn test -pl itests/qtest-iceberg -Pitests \
-Dtest=TestIcebergLlapLocalCliDriver \
-Dqfile=QuotedIdentifier_1.q \
-Dtest.output.overwrite
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
