jiangxt2 commented on PR #58066: URL: https://github.com/apache/spark/pull/58066#issuecomment-5339562501
Additional ecosystem references: For reference, the same four bitmap set operations are also available in several analytical SQL engines: - [Flink 2.3](https://nightlies.apache.org/flink/flink-docs-release-2.3/docs/sql/functions/built-in-functions/#bitmap-functions) exposes `BITMAP_AND`, `BITMAP_OR`, `BITMAP_ANDNOT`, and `BITMAP_XOR` over its native `BITMAP` type. - [ClickHouse](https://clickhouse.com/docs/reference/functions/regular-functions/bitmap-functions) has provided `bitmapAnd`, `bitmapOr`, `bitmapAndnot`, and `bitmapXor` since v20.1. - Apache Doris provides `BITMAP_AND`, `BITMAP_OR`, `BITMAP_AND_NOT`, and `BITMAP_XOR`, and documents bitmap use cases such as [exact distinct counting and audience segmentation](https://doris.apache.org/docs/4.x/query-acceleration/distinct-counts/bitmap-precise-deduplication/). - [StarRocks](https://docs.starrocks.io/docs/category/bitmap/) provides the corresponding `bitmap_and`, `bitmap_or`, `bitmap_andnot`, and `bitmap_xor` functions and documents bitmap use cases such as exact distinct counting and retention analysis. These engines use different physical bitmap representations, so this comparison does not imply binary-format compatibility. Spark continues to use its existing 4096-byte `BINARY` flat bitmap, and both operands must use the same bit-position mapping and, when bucketed, represent the same bucket. These ecosystem references support the completeness and naming of the proposed API. The four scalar functions allow users to compute row-wise intersection, union, difference, and symmetric difference between precomputed bitmaps without implementing equivalent UDFs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
