morningman opened a new issue, #67371: URL: https://github.com/apache/doris/issues/67371
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Apache Doris 4.1.3-rc02, commit `31263df4dc1d4d3a27517d264802cd4d6b92c874` Client: Python + ADBC Flight SQL driver (`adbc_driver_flightsql`), FE `arrow_flight_sql_port` = 41070. The MySQL/JDBC protocol is used as the control path for comparison. ### What's Wrong? When an Iceberg catalog is configured with `"enable.mapping.varbinary" = "false"`, Doris Flight SQL declares an arbitrary `BINARY` column as Arrow `string`, but the payload actually contains non-UTF-8 bytes. PyArrow fails before returning any row: ``` UnicodeDecodeError (byte 0x84) ``` The Arrow schema for the query is `id: int32, col1: string, col2: string`. JDBC reads the same Iceberg rows without a problem, and with VARBINARY mapping enabled the ADBC binary read works. ### What You Expected? The Arrow type declared by the server and the encoding of the payload must agree. Arbitrary binary content should be sent as `binary`/`large_binary`, or, if it is sent as `string`, it must be a valid UTF-8 representation. ### How to Reproduce? 1. Bring up the Iceberg regression fixture. 2. Create or use an Iceberg catalog with `"enable.mapping.varbinary" = "false"`. 3. `SWITCH` to that catalog and `USE test_varbinary`. 4. Run the query over Python ADBC, inspect the schema and call `to_pylist()`. ```sql SWITCH test_iceberg_no_mapping; USE test_varbinary; SELECT * FROM test_ice_uuid_orc ORDER BY id; ``` Client side: ```python import adbc_driver_flightsql.dbapi as flight_sql conn = flight_sql.connect(uri="grpc://127.0.0.1:41070", db_kwargs={"username": "root", "password": ""}) cur = conn.cursor() cur.execute("SELECT * FROM test_iceberg_no_mapping.test_varbinary.test_ice_uuid_orc ORDER BY id") table = cur.fetch_arrow_table() print(table.schema) # id: int32, col1: string, col2: string print(table.to_pylist()) # UnicodeDecodeError on byte 0x84 ``` ### Anything Else? Without VARBINARY mapping, the Iceberg `binary`/`uuid` column falls back to the Doris `STRING` type, and the Arrow conversion follows that declared type rather than the real content. Since Arrow `string` is defined as UTF-8, any consumer that validates the encoding (PyArrow does) fails on the whole batch. **Workaround:** set `"enable.mapping.varbinary" = "true"` on the Iceberg catalog so the column is returned through the Arrow `binary` type. Found with the `external_table_p0/iceberg/test_iceberg_varbinary` fixture; four no-mapping cases are affected, while the mapping-enabled binary reads keep passing over ADBC. Tracking issue: #65615 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
