Owen-CH-Leung opened a new pull request, #70656: URL: https://github.com/apache/doris/pull/70656
### Background Doris reads an Iceberg BINARY column as VARBINARY and a FIXED(n) column as VARBINARY(n) (#68786). CREATE TABLE builds the Iceberg schema in IcebergSchemaBuilder.buildSchema, which had no VARBINARY case and threw "Unsupported type for Iceberg: VARBINARY". ### What problem does this PR solve? Issue Number: None Related PR: #68786 (read mapping), #69285 (MODIFY COLUMN restating BINARY and FIXED) PR Type: Feature Problem Summary: Doris cannot create an Iceberg BINARY column, so tables created through Doris store binary payloads as STRING. Iceberg requires STRING to be UTF-8, and since 4.1.4 the FE decodes STRING column bounds as UTF-8 at commit, so a non-UTF-8 value fails the insert with "Failed to decode value as UTF-8". This commit maps VARBINARY and VARBINARY(n) to BINARY when CREATE TABLE builds the schema, at the top level and inside ARRAY, MAP and STRUCT. Users see no change yet, because fe-core still rejects VARBINARY before the connector is called; the next commit lifts that. ### Details - **Scope:** only IcebergSchemaBuilder.buildSchema. ADD COLUMN and MODIFY COLUMN go through buildColumnType and still reject VARBINARY, so #69285 keeps resolving a restated VARBINARY against the current column. - **Design:** a flag on the private IdAllocator, set only by buildSchema. toIcebergPrimitive is untouched because ALTER also calls it. VARBINARY(n) becomes BINARY rather than FIXED(n). Doris treats n as a maximum and does not enforce it (see the comment in data_type_varbinary_serde.cpp), while Iceberg FIXED(n) holds exactly n bytes. With FIXED(n), every shorter value would fail on write, and so would a CTAS from a MySQL BLOB, which arrives as VARBINARY(65535), or a UNION, which takes the largest length. Iceberg also rejects the truncate transform on FIXED and cannot promote FIXED to BINARY or to a longer FIXED. #69285 gives the same reason for rejecting this mapping, and Paimon already maps every VARBINARY to VarBinaryType(MAX_LENGTH). - **Usage:** none until the next commit. - **Default behavior:** a column declared VARBINARY(n) reads back as varbinary(2147483647), as any Iceberg BINARY column does. A later MODIFY COLUMN can restate it as VARBINARY but not as VARBINARY(n). - **Compatibility:** no change to existing tables or to any statement that works today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
