Doris-Breakwater commented on issue #66811: URL: https://github.com/apache/doris/issues/66811#issuecomment-5311003056
Breakwater-GitHub-Analysis-Slot: slot_65f32fe08164 ## Initial triage This is a credible, high-impact Iceberg read regression, but the reported normal-query call path is not yet fully reconciled with the cited revision. The case-sensitive FE/BE contract violation is real: at `b9ee837a3e`, `IcebergSchemaUtils.buildCurrentSchema` can put lowercased top-level `TField.name` values into the `-1` schema dictionary, while BE builds its table-side lookup map from those names and `ParquetReader::_do_init_reader` checks the case-preserved scan-slot name verbatim. A dictionary key `channelid` therefore cannot satisfy a projected slot named `channelId`. There are currently no labels on the issue. Suggested triage labels are bug/regression plus the external-table/Iceberg component, with elevated priority because affected tables can become entirely unreadable. ## Verified from code - The BE failure is deterministic once the two names differ; it is not specific to complex types or to REST catalogs. - The proposed top-level rule in linked PR #66813 is directionally correct: use the request only for `Schema.caseInsensitiveFindField(...)`, then emit `field.name()` so the dictionary matches the Doris column/BE slot. The patch only changes the top-level override, so the intentional lowercasing of nested struct children remains intact. - The patch also corrects the empty-request/full-schema branch, which currently lowercases every top-level field. That branch is used by full-schema planning paths such as pinned-schema/time-travel and Top-N lazy materialization. ## Important unresolved discrepancy On unmodified `b9ee837a3e`, the normal projected-column path appears case-preserving already: - `IcebergConnectorMetadata.parseSchema` creates `ConnectorColumn` with `field.name()`. - `IcebergConnectorMetadata.buildColumnHandles` keys the map with `field.name()` and constructs `IcebergColumnHandle` with that same case-preserved name. - `PluginDrivenScanNode.buildColumnHandles` looks up the case-preserved slot name and passes the resulting handle to `requestedLowerNames`, which merely returns `IcebergColumnHandle#getName()` and does not lowercase it. Therefore, a plain `SELECT channelId ...` on a clean build of the cited revision should normally pass `channelId`, not `channelid`, into `buildCurrentSchema`. The helper's stale name/Javadoc says the values are lowercased, but the production code does not establish that. The direct unit test in #66813 proves the helper fix for a synthetic lowercase input; it does not yet prove how that lowercase input is reached in the reported query. One additional gap in #66813 should be checked before merge: `encodeEqualitySchemaEvolutionProp` still uses the `buildCurrentSchema(List<NestedField>, ...)` overload, which lowercases top-level names. Although file matching uses field IDs, BE also uses each current-schema `TField.name` as the table-side `StructNode` key before the same projected-slot guard. A mixed-case table with applicable equality deletes may therefore retain the failure. This path needs an explicit test or a code-level explanation showing why it cannot reach the name lookup. ## Information needed to close the evidence gap 1. Exact FE and BE commit IDs from the failing cluster, plus confirmation/checksum that the deployed loadable Iceberg connector plugin was built from the same revision. Artifact skew is particularly relevant after the connector-plugin refactor. 2. Whether the failing scan uses time travel, Top-N lazy materialization, or has live equality-delete files; also provide the Iceberg format version and data-file format. 3. `DESC cat.db.t`, `EXPLAIN VERBOSE SELECT channelId FROM cat.db.t LIMIT 1`, and the full query error. 4. From one failing request, the three actual values: the Doris slot name, the selected `IcebergColumnHandle#getName()`, and the decoded `history_schema_info[-1]` top-level `TField.name`. A focused test/log at that boundary is more useful than a broad profile for this initialization failure. ## Recommended next steps 1. Reproduce on a clean build of `b9ee837a3e` with a minimal Parquet Iceberg table containing `channelId STRING` and a lowercase sibling. If the normal projection does not reproduce, compare the deployed connector artifact with the FE revision before changing the root-cause statement. 2. Extend #66813 with an end-to-end connector/scan-node test that starts from `getColumnHandles`, not only a direct `buildCurrentSchema(..., "channelid")` call. Add coverage for the empty-request branch and for a table with applicable equality deletes. 3. Rename `requestedLowerNames` (for example, to `requestedNames`) and update the stale Javadocs/comments that currently claim handles are lowercased and names are echoed verbatim. 4. Once the production-path discrepancy and equality-delete branch are covered, the `field.name()` top-level fix is consistent with the BE lookup contract and should be suitable to merge. No Doris Profile is required for this failure because it occurs during reader initialization, before execution metrics can identify a performance bottleneck. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
