HwangDongJun opened a new issue, #13619:
URL: https://github.com/apache/gluten/issues/13619
### Backend
VL (Velox)
### Bug description
`bit_count` is offloaded to Velox and returns a wrong result, without any
error or fallback message, when the argument is a **negative** `TINYINT`,
`SMALLINT` or `INT`. `BIGINT`, `BOOLEAN`, non-negative values and `NULL` are
correct. The function support table lists `bit_count` as supported (`S`) with
no limitation.
Spark counts the set bits of the value **sign-extended to a 64-bit long**
(the function is documented as returning "the number of bits that are set in
the argument expr as an unsigned 64-bit integer"). Velox's `BitCountFunction`
counts only as many bits as the input type is wide.
**Reproduce** (plain SQL, default optimizer settings, Parquet table):
```sql
CREATE TABLE t_bitcount (c_tinyint TINYINT, c_smallint SMALLINT, c_int INT,
c_bigint BIGINT) USING parquet;
INSERT INTO t_bitcount VALUES (-1, -1, -1, -1), (-7, -7, -7, -7), (7, 7, 7,
7), (0, 0, 0, 0);
SELECT c_int,
bit_count(c_tinyint) AS bc_tinyint,
bit_count(c_smallint) AS bc_smallint,
bit_count(c_int) AS bc_int,
bit_count(c_bigint) AS bc_bigint
FROM t_bitcount ORDER BY c_int;
```
Spark (`spark.gluten.enabled=false`):
| c_int | bc_tinyint | bc_smallint | bc_int | bc_bigint |
|---|---|---|---|---|
| -7 | 62 | 62 | 62 | 62 |
| -1 | 64 | 64 | 64 | 64 |
| 0 | 0 | 0 | 0 | 0 |
| 7 | 3 | 3 | 3 | 3 |
Gluten (the plan contains `ProjectExecTransformer`, so the expression is
executed natively):
| c_int | bc_tinyint | bc_smallint | bc_int | bc_bigint |
|---|---|---|---|---|
| -7 | **6** | **14** | **30** | 62 |
| -1 | **8** | **16** | **32** | 64 |
| 0 | 0 | 0 | 0 | 0 |
| 7 | 3 | 3 | 3 | 3 |
Boundary values, same behavior (Spark vs Gluten): `TINYINT -128` 57 vs 1,
`SMALLINT -32768` 49 vs 1, `INT -2147483648` 33 vs 1.
I compared about 30 other functions (bitwise ops, shifts, arithmetic,
rounding, hashing, casts, aggregates) over TINYINT/SMALLINT/INT/BIGINT boundary
values and found no other difference of this kind, so this looks specific to
`bit_count`.
**Analysis**
- Spark (`BitwiseCount.nullSafeEval`) calls
`java.lang.Long.bitCount(input.asInstanceOf[Byte|Short|Int])`, i.e. the narrow
value is widened to a signed long first, so every negative value has the upper
bits set.
- Velox (`velox/functions/sparksql/Bitwise.h`):
```cpp
struct BitCountFunction {
template <typename TInput>
FOLLY_ALWAYS_INLINE void call(int32_t& result, TInput num) {
constexpr int kMaxBits = sizeof(TInput) * CHAR_BIT;
auto value = static_cast<uint64_t>(num);
result = bits::countBits(&value, 0, kMaxBits);
}
};
```
The value is sign-extended into `uint64_t`, but only `sizeof(TInput) * 8`
bits are countedtwiseTest.bitCount`) pins this behavior, e.g.
`bitCount<int32_t>(kMin32) == 1`. I read thisfrom `facebookincubator/velox`
main; I could not check the exact branch Gluten builds from.
- Gluten maps `BitwiseCount` to `bit_count` unchanged
(`ExpressionMappings`), without casting the argument.
- The existing SQL tests (`bitwise.sql` for the Velox backend, Spark 3.5 to
4.2) and `ScalarFunctionsValidateSuite` only use positive values for
TINYINT/SMALLINT/INT (the only negative inputs are BIGINT), which is why this
is not caught.
**Possible fixes** (maintainers' choice):
1. Count 64 bits in the Spark flavor of the function in Velox (and update
its unit test), or
2. In Gluten, cast the argument of `BitwiseCount` to `BIGINT` for
byte/short/int inputs before it is passed to Velox (BIGINT already matches
Spark).
I am happy to prepare a PR for the Gluten-side change plus a SQL test with
negative narrow inputs, if that direction is acceptable.
Reproduced on: the Gluten 1.7.0 release bundle and the 1.9.0-SNAPSHOT
nightly built from `main` (commit `ca9b6dadf`, 2026-11-11), with the same
result in both. Spark 4.1.1.
### Gluten version
main branch
### Spark version
Spark-4.1.x
### Spark configurations
spark.plugins=org.apache.gluten.GlutenPlugin
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=1g
spark.sql.ansi.enabled=false
### System information
Linux x86_64 (CentOS 8 based build), JDK 17
### Relevant logs
```bash
No error or warning is logged; the query succeeds with the wrong values.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]