HwangDongJun opened a new pull request, #13621:
URL: https://github.com/apache/gluten/pull/13621

   ## What changes are proposed in this pull request?
   
   `bit_count` returns a different result from Spark for a negative `TINYINT`, 
`SMALLINT` or `INT` argument, with no error and no fallback. Spark counts the 
set bits of the value sign-extended to a long, so `bit_count(cast(-1 as 
tinyint))` is 64. Velox's `BitCountFunction` counts only as many bits as the 
input type is wide (8, 16 or 32). `BIGINT`, `BOOLEAN`, non-negative values and 
`NULL` already match Spark. See #13619.
   
   This PR widens byte, short and int arguments of `BitwiseCount` to `BIGINT` 
in `VeloxSparkPlanExecApi.extraExpressionConverter`, the same hook that already 
rewrites `RoundCeil` / `RoundFloor` on decimals. The cast is a widening one and 
cannot fail. The rewritten expression is passed as `original`, so the function 
signature is built from `BIGINT`. Only the Velox backend is touched. The 
alternative mentioned in the issue, counting 64 bits in the Spark flavor of the 
function in Velox, would make this workaround unnecessary.
   
   | `bit_count(x)` as | Spark | Gluten before |
   |---|---|---|
   | `x = -1`: tinyint / smallint / int | 64 / 64 / 64 | 8 / 16 / 32 |
   | `x = -7`: tinyint / smallint / int | 62 / 62 / 62 | 6 / 14 / 30 |
   
   Fixes #13619.
   
   ## How was this patch tested?
   
   Added `bit_count on negative tinyint, smallint and int` to 
`ScalarFunctionsValidateSuite`: the boundary values of every integral type 
(`-1`, `-7`, MIN, MAX, `0`, `7`) and `NULL` are written to Parquet and 
`bit_count` is run on all four columns with `runQueryAndCompare` (compared with 
vanilla Spark) and `checkGlutenPlan[ProjectExecTransformer]` (the projection 
stays offloaded). The `BIGINT` column was already correct and guards against 
the cast changing it.
   
   CI on a PR in my fork (same base as this one) passed (one `tpc-test-ubuntu` 
job failed once while downloading a Maven plugin and passed on re-run), and the 
new test is listed as passed in the `spark-test-spark35`, `spark-test-spark40` 
and `spark-test-spark41` logs. CI only runs the code after the fix, so for the 
before/after I built the Spark 4.1 bundle from this branch with the native 
libraries of the 2026-11-11 nightly (`ca9b6dadf`, two commits behind this base, 
the only native change in between is `SparkExprToSubfieldFilterParser.cc`) and 
ran the same data in local mode: before the fix `-1` gives 8 / 16 / 32 and 
`Int.MinValue` gives 1 for every type; after it all 7 rows equal vanilla Spark 
(`-1`: 64, `Int.MinValue`: 57 / 49 / 33 / 1) and the plan still contains 
`ProjectExecTransformer`.
   
   Not verified: the Velox branch Gluten builds from (I read `BitCountFunction` 
and its test in `facebookincubator/velox` main), ANSI mode, `bit_count` nested 
in larger expressions or aggregates, and the Spark 3.4 and 4.2 test logs (those 
jobs compiled and passed). I did not extend the SQL golden test `bitwise.sql`, 
which has four copies and only positive tinyint/smallint/int inputs.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: claude-sonnet-5.5


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to