Doris-Breakwater commented on issue #67371: URL: https://github.com/apache/doris/issues/67371#issuecomment-5490641944
## Initial assessment Confirmed as a valid server-side Arrow schema/payload mismatch in the reported `4.1.3-rc02` commit (`31263df4dc1d4d3a27517d264802cd4d6b92c874`). This is not only a Python presentation issue: Arrow `string` carries UTF-8 semantics, while this path can place arbitrary Iceberg binary bytes in its value buffer. A validating client can therefore reject the entire batch before exposing rows. The issue is actionable as written. The supplied fixture inserts non-UTF-8 sequences including byte `0x84`, so no BE log or profile is needed to establish the type-contract problem. ## Code evidence - `IcebergUtils.icebergPrimitiveTypeToDorisType()` maps Iceberg `UUID` and `BINARY` to Doris `STRING` when `enable.mapping.varbinary=false` (`fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:691-714`). - The file readers preserve that fallback: ORC `BINARY` becomes `TYPE_STRING` when mapping is disabled (`be/src/format/orc/vorc_reader.cpp:1705-1738`), and raw Parquet `BYTE_ARRAY`/UUID similarly become `TYPE_STRING` (`be/src/format/parquet/schema_desc.cpp:251-258,323-329`; the v2 reader has the same mapping). - Arrow Flight constructs its result schema only from the final BE output expression types (`be/src/exec/operator/result_sink_operator.cpp:58-67` and `be/src/format/arrow/arrow_row_batch.cpp:220-238`). At that point the original Iceberg binary identity is no longer present; Doris `STRING` is unconditionally mapped to Arrow `utf8`, while Doris `VARBINARY` maps to Arrow `binary` (`arrow_row_batch.cpp:88-96,176-178`). - Batch conversion then uses `DataTypeStringSerDe`, which appends the stored bytes unchanged to `StringBuilder` (`be/src/core/data_type_serde/data_type_string_serde.cpp:399-445`). The Flight conversion path does not run Arrow full validation before returning the batch. This directly explains why schema inspection reports `string` and PyArrow later raises `UnicodeDecodeError`. - With `enable.mapping.varbinary=true`, the FE/reader type remains `VARBINARY`, so the existing `VARBINARY -> arrow::binary()` path is internally consistent. The documented workaround is therefore sound. The MySQL/JDBC control path succeeding does not contradict this diagnosis: that protocol can transport length-delimited byte sequences under Doris string metadata and does not impose Arrow's UTF-8 invariant. ## Fix boundary and open design decision Changing only the string byte-copy loop is insufficient. The Arrow type must be selected before the stream schema is published, so the result schema builder needs binary semantic information that is currently lost between Iceberg schema mapping and the result sink. Two viable directions should be evaluated: 1. Carry an Arrow result-type hint/source binary provenance through the FE result sink/output expression path and emit `binary` (or fixed-size binary for UUID/FIXED where appropriate) for direct binary-origin results. The existing string SerDe already supports writing a Doris string column into binary/fixed-size-binary Arrow builders, so payload conversion support largely exists. 2. Deliberately expose these values as Doris `VARBINARY` for Flight SQL even when the catalog uses the legacy no-mapping mode. The second option is simpler but may make SQL/result metadata protocol-dependent. The first option must define propagation through aliases, projections, expressions, nested fields, and casts so a genuine Doris `STRING` is never silently reclassified as binary. Silently hex/base64-encoding only on Flight would avoid invalid UTF-8 but would change values relative to the SQL result and should not be done without an explicit compatibility decision. ## Missing information / triage notes - No labels, assignee, or milestone were attached when this issue was observed. Please add the repository's bug plus Arrow Flight/external-catalog component labels and route it to both owners. - Exact `adbc_driver_flightsql` and PyArrow versions would help document the client compatibility matrix, but they are not blocking for accepting the bug because the server-side schema invariant is independently visible in code. - Maintainers should explicitly confirm the intended cross-protocol contract for `enable.mapping.varbinary=false`: byte-preserving binary output or a textual representation. That decision determines which fix direction is compatible. ## Recommended next steps 1. Add a failing end-to-end Flight SQL regression using the existing `test_iceberg_varbinary` ORC and Parquet fixtures with mapping disabled. It should assert the Arrow field type, call a validating consumer such as Python ADBC/PyArrow `to_pylist()` (or `validate(full=True)`), and compare the exact returned bytes. Keep mapping-enabled cases as controls. 2. Add focused BE coverage proving ordinary valid Doris `STRING` still produces Arrow UTF-8 and binary-origin fallback data produces Arrow binary, including nullable values and both local/remote Flight readers. 3. Implement the chosen type-provenance/Flight typing approach at schema construction, then verify aliases/casts/nested columns and ensure MySQL/JDBC behavior is unchanged. No code was modified during this analysis, and the reported failure was not independently rerun against a live cluster; the root-cause judgment above is based on the exact reported commit and the checked-in fixture bytes. Breakwater-GitHub-Analysis-Slot: slot_c8ad7b1e212c -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
