github-actions[bot] commented on code in PR #70476:
URL: https://github.com/apache/doris/pull/70476#discussion_r4552370540
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsLoadScanProvider.java:
##########
@@ -374,13 +380,19 @@ private void
fillContextExprMap(List<NereidsImportColumnDesc> columnDescList, Ne
} else {
Column slotColumn;
TFileFormatType fileFormatType =
fileGroup.getFileFormatProperties().getFileFormatType();
- // Use real column type for arrow format, other formats read
as varchar first
+ // Arrow uses target types; native Parquet VARIANT also keeps
its typed source slot.
if (fileFormatType == TFileFormatType.FORMAT_ARROW) {
if (tblColumn == null) {
throw new AnalysisException("Unknown column " +
realColName + " in table " + tbl.getName()
+ " for " + fileFormatType + " load");
}
slotColumn = new Column(realColName, tblColumn.getType(),
true);
+ } else if (fileFormatType == TFileFormatType.FORMAT_PARQUET
+ && tblColumn != null &&
tblColumn.getType().isVariantType()
Review Comment:
[P1] Preserve native VARIANT through mapped source names
Export `SELECT id, v AS raw_v ... INTO OUTFILE ... FORMAT AS PARQUET`, then
load into `(id, v VARIANT)` with `(id, raw_v) SET(v=raw_v)`. Since `raw_v` has
no matching target column, this condition leaves its source slot VARCHAR. The
resulting reduced plan is:
```text
Sink(v VARIANT)
Project(try_parse_to_variant(raw_v) AS v)
Scan(raw_v VARCHAR; physical VARIANT)
```
`TableColumnMapper` inserts `EvolutionCast(VARIANT -> VARCHAR)`, which uses
ordinary SQL string conversion: root string `"true"` reloads as BOOLEAN,
BOOLEAN `true` becomes integer `1`, and string `"hello"` becomes NULL or a
rejected row. The same mapping was lossless with the previous JSON UTF-8
export. Explicit `parse_to_variant(raw_v)` also receives the already lossy
text. Preserve native values through mapped source names, or provide
JSON-preserving text for them, and add an alias/renamed-destination roundtrip
test. This is distinct from the fixed direct `(id,v)`/source-filter case: it
needs only one consumer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
