zghong opened a new issue, #69912: URL: https://github.com/apache/doris/issues/69912
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description Doris should support converting raw `VARBINARY` values directly into `BITMAP` and `HLL` values. The existing Hive Bitmap/HLL integration documented in the [Hive UDF documentation](https://doris.apache.org/zh-CN/docs/dev/connection-integration/data-integration/hive-udf) relies on Doris Bitmap/HLL serialized data being consumed by the existing Base64-oriented functions, such as `bitmap_from_base64` and `hll_from_base64`. However, external binary type mapping has changed: - PR [#57821](https://github.com/apache/doris/pull/57821) introduced `enable.mapping.varbinary` for external binary columns; - PR [#68786](https://github.com/apache/doris/pull/68786) made external binary mapping unconditional and migrated legacy catalog settings; - Hive/HMS `BINARY`, as well as supported Parquet/ORC binary columns, are now exposed as Doris `VARBINARY` while preserving their raw bytes. The Doris Hive UDF implementations for `to_bitmap`/`bitmap_union` and `to_hll`/`hll_union` produce raw serialized byte arrays. With the current mapping behavior, those values are read as `VARBINARY`, while the existing functions expect Base64 text (`STRING`). Therefore, the documented Hive UDF workflow can no longer be used directly: ```sql bitmap_from_base64(binary_column) hll_from_base64(binary_column) ``` The type and encoding no longer match. Users should not have to convert the raw bytes to Base64 text first merely to convert them back into a Doris Bitmap or HLL. Please add the following functions: ```sql bitmap_from_binary(VARBINARY) -> BITMAP bitmap_to_binary(BITMAP) -> VARBINARY hll_from_binary(VARBINARY) -> HLL hll_to_binary(HLL) -> VARBINARY ``` The `*_from_binary` functions should validate the complete input buffer and return row-level `NULL` for empty, truncated, or malformed values instead of reading beyond the input buffer or failing the entire query. The Bitmap path should also validate Roaring container contents because the input may come from an external user-controlled binary column. ### Use case This feature is needed for the following scenarios: 1. **Read existing Hive UDF output** Hive `to_bitmap`/`bitmap_union` and `to_hll`/`hll_union` output Doris-compatible serialized binary values. Doris should be able to read those values directly from Hive `BINARY` columns through an external catalog. 2. **Read Parquet/ORC binary columns** Parquet and ORC files can store Bitmap/HLL values as binary data. Since those columns are exposed as `VARBINARY`, users need a direct conversion function to query them with `bitmap_count`, `bitmap_union`, `hll_cardinality`, and related functions. 3. **Write Doris values to external binary columns** Users need to write Doris Bitmap/HLL values to Hive, Parquet, or ORC binary columns without an unnecessary Base64 text conversion. 4. **Avoid a Base64 conversion workaround** A workaround such as the following adds CPU, memory, intermediate strings, and data expansion: ```sql bitmap_from_base64(to_base64_binary(binary_column)) ``` Direct binary functions would preserve the original bytes and express the intended conversion clearly in SQL. 5. **Safe handling of malformed external data** External binary columns may contain empty, truncated, or corrupted values. Invalid rows should become `NULL` and must not cause out-of-bounds reads, heap corruption, or backend crashes during subsequent Bitmap/HLL operations. ### Related issues _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
