fmorillo7694 commented on PR #206:
URL:
https://github.com/apache/flink-connector-aws/pull/206#issuecomment-5239656985
## Real AWS Glue integration test results (design rationale for
`originalName`)
The latest head (b1a05d3) includes fixes driven by an integration test run
against a **real Glue Data Catalog** (us-west-2), not Moto/Localstack. Posting
the evidence here since it settles a design question a reviewer might
reasonably raise: *why store lowercase names plus an `originalName` parameter
instead of writing mixed-case names directly?*
### Empirical facts about real Glue (reproducible via the AWS CLI)
1. **Glue lowercases column names on `CreateTable`.** Creating a table with
columns `userId` / `EventTime` stores and returns `userid` / `eventtime`. Glue
does *not* preserve mixed-case column names, so writing them as-is silently
loses case.
2. **Glue rejects column-level `Parameters` on partition columns** with
`400: Parameters not supported for partition columns`. The `originalName`
column-parameter mechanism therefore cannot be used for partition keys — their
declared case is preserved in an order-preserving *table-level* parameter
(`flink.original-partition-keys`) instead.
### What case loss breaks (observed before the fix, same suite)
- `DESCRIBE` returned `[userid, eventtime, username]` for a table declared
`[userId, EventTime, userName]`
- `SELECT userId FROM t` → *Column 'userId' not found in any table; did you
mean 'userid'?*
- Partition ops failed to match: spec `{eventDate=...}` vs stored keys
`[eventdate]`
- Streaming `INSERT INTO` / `SELECT` against catalog-registered Kinesis
tables failed to plan, and JSON field names on the wire would be lowercased
(silent data corruption for consumers)
### Validation on the current head
25-step integration suite against real AWS, all passing:
- **DDL**: mixed-case databases, tables, and columns round-trip exactly
(create / show / describe / drop); `ALTER TABLE ... SET` (now implemented via
`UpdateTable`) with a case-preservation regression check; partitioned tables
with `ADD/SHOW/DROP PARTITION` on mixed-case partition keys
- **Streaming**: unbounded job writing to and reading from a real Kinesis
stream through source/sink tables registered in this catalog; JSON field names
verified case-intact on the wire via `GetRecords`; `PARTITIONED BY` on a
Kinesis table verified to set the record partition key from the (case-restored)
column
- Unit tests: 126/126
One known behavior worth documenting: partition columns are appended after
data columns on read-back (Glue models partition keys outside the storage
descriptor, like Hive), so declaring partition columns last keeps the schema
order stable across the round-trip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
