Doris-Breakwater commented on issue #68309: URL: https://github.com/apache/doris/issues/68309#issuecomment-5755297399
## Initial analysis **Verdict: confirmed Java-reader compatibility gap at the reported commit, with corrections needed to the reproduction and stated blast radius.** No Doris profile or runtime logs are needed to establish the class-level defect. ### Verified facts - At `6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907`, BE defaults `enable_set_in_bitmap_value` to `true` ([config](https://github.com/apache/doris/blob/6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907/be/src/common/config.cpp#L1451-L1452)). Its binary contract defines `SET = 5`, and the SET writer emits `type:u8`, `count:u8`, then `count` little-endian `uint64` values ([type codes](https://github.com/apache/doris/blob/6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907/be/src/core/value/bitmap_value.h#L53-L90), [writer](https://github.com/apache/doris/blob/6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907/be/src/core/value/bitmap_value.h#L2425-L2463)). The inline SET capacity is 32 ([definition](https://github.com/apache/doris/blob/6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907/be/src/core/value/bitmap_value.h#L2975-L2987)). BE's own reader already handles flag 5 and rejects counts above 32 ([reader](https://github.com/apache/doris/blob/6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907/be/src/core/value/bitmap_value.h#L2471-L25 28)). - FE `BitmapValue` defines only codes 0 through 4 and its `deserialize()` switch handles only those codes; flag 5 deterministically reaches the reported `unknown bitmap type 5` default branch ([FE constants and reader](https://github.com/apache/doris/blob/6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907/fe/fe-common/src/main/java/org/apache/doris/common/io/BitmapValue.java#L37-L42)). The reader switch is at [lines 131-153](https://github.com/apache/doris/blob/6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907/fe/fe-common/src/main/java/org/apache/doris/common/io/BitmapValue.java#L131-L153). - Existing FE coverage is a Java-writer-to-Java-reader round trip. Since the Java writer cannot emit SET, it does not test BE/FE wire compatibility ([test](https://github.com/apache/doris/blob/6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907/fe/fe-common/src/test/java/org/apache/doris/common/io/BitmapValueTest.java#L292-L353)). ### Reproduction correction The Base64 value in the issue, `BQMFAAAAAAAAAEAAAAAAAAAAwAAAAAAAAA=`, is malformed (35 characters). `Base64.getDecoder().decode(...)` throws `IllegalArgumentException: Input byte array has wrong 4-byte ending unit`, so the posted Java snippet does not reach `BitmapValue.deserialize()` as written. A valid flag-5 fixture containing `{1, 3, 5}` is: ```text BQMBAAAAAAAAAAMAAAAAAAAABQAAAAAAAAA= ``` It decodes to 26 bytes: type `5`, count `3`, followed by little-endian `1`, `3`, and `5`. Passing those bytes to the cited FE reader reaches `unknown bitmap type 5`. SET payload order is not part of the semantic contract, so another ordering of the same three values is also valid if correctly encoded. ### Impact qualification / missing evidence - The Java Hive UDF module definitely uses this reader for binary bitmap arguments ([utility](https://github.com/apache/doris/blob/6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907/fe/hive-udf/src/main/java/org/apache/doris/common/BitmapValueUtil.java#L37-L42)), so a Hive UDF consuming BE-produced SET bytes is a verified affected boundary. - Doris SQL `bitmap_from_base64` itself is implemented and deserialized in BE, whose reader supports SET ([implementation](https://github.com/apache/doris/blob/6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907/be/src/exprs/function/function_bitmap.cpp#L263-L313)). Therefore that builtin alone is not evidence of an FE Java failure. Please provide the full stack trace and the exact Hive Catalog/import call path if a catalog workflow routes the bytes through `fe-common` first. - The Spark Load impact is plausible from the purpose of this legacy Java bitmap class, but the issue does not include a failing job or stack trace. Please add the exact load mode/artifact version and full exception stack if this broader impact needs to be tracked as verified rather than inferred. These missing call-path details do **not** block fixing the deterministic flag-5 reader defect. ### Recommended next steps 1. Add a `SET = 5` read case in FE: read the count as an unsigned byte, enforce the 32-element format limit, read each value as little-endian 64-bit, and populate the existing Java single/bitmap representation. A distinct in-memory SET representation is unnecessary for semantic compatibility. 2. Add hard-coded cross-language fixtures rather than another Java self-round-trip. Cover a normal SET, the 32-element boundary, a 64-bit value, truncated input, count greater than 32, and the chosen duplicate-value policy (BE rejects duplicate values in a flag-5 payload). 3. Clarify the PR scope relative to the expectation of “all valid bitmap binary formats.” The Java reader also lacks BE's `SET_V2 = 10`, `BITMAP32_V2 = 12`, and `BITMAP64_V2 = 13` codes ([BE type table](https://github.com/apache/doris/blob/6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907/be/src/core/value/bitmap_value.h#L53-L90)). BE defaults `bitmap_serialize_version` to 1, so flag 5 is the immediate default-path bug, but adding only flag 5 should not be described as complete support for every BE format. Breakwater-GitHub-Analysis-Slot: slot_595e6f9856b0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
