schenksj opened a new issue, #7479:
URL: https://github.com/apache/datafusion-comet/issues/7479
### Describe the bug
A `Project` or `Filter` that contains `rand(<int>)` or `randn(<int>)`, for
example `rand(42)`, is
not converted, and Comet records no fallback reason on the operator or on
any of its expressions.
In SQL, an integer literal seed such as `rand(42)` has type `IntegerType`.
Only a `LongType`
seed, such as `rand(42L)`, is converted.
With `spark.comet.explain.fallback.strict.enabled=true`, which
`CometTestBase` turns on for
Comet's own suites, the query fails with:
```
java.lang.IllegalStateException: Comet did not convert Filter but recorded
no fallback reason on
the operator or any of its expressions. Add a withFallbackReason call
stating why conversion
failed.
```
(or `... did not convert Project ...` for a projection). Without the strict
check, the only
reason shown is the generic "`<operator>` is not supported" from
`CometExecRule.reportUnexplainedFallback`, which does not say what to change.
### Steps to reproduce
Run on `main` (`0f7f8e453`) with the default profile (Spark 4.1.3), in a
`CometTestBase` suite:
```scala
spark.range(0, 200, 1, numPartitions = 1)
.selectExpr("CAST(id AS INT) AS id", "IF(id % 3 = 0, NULL, CAST(id AS
INT)) AS y")
.write.parquet(path)
spark.read.parquet(path).createOrReplaceTempView("t")
```
These four queries each fail with the error above:
```sql
SELECT id FROM t WHERE y IS NOT NULL AND rand(42) < 0.5;
SELECT id FROM t WHERE abs(rand(42)) < 0.5;
SELECT id, rand(1) AS r FROM t;
SELECT id, randn(1) AS r FROM t;
```
The same queries with a `Long` seed are converted:
| Query | Executed plan |
| --- | --- |
| `SELECT id FROM t WHERE y IS NOT NULL AND rand(42L) < 0.5` |
`CometProject` > `CometFilter` > `CometNativeScan` |
| `SELECT id FROM t WHERE abs(rand(42L)) < 0.5` | contains `CometFilter` |
| `SELECT id, rand(1L) AS r FROM t` | `CometProject` |
| `SELECT id, randn(1L) AS r FROM t` | `CometProject` |
In the analyzed plans, the seed of `rand(1)` and `rand(42)` is
`IntegerType`, and the seed of
`rand(1L)` is `LongType`.
### Cause
In `spark/src/main/scala/org/apache/comet/serde/nondetermenistic.scala`, the
seeded serde's
`getSupportLevel` returns `Compatible()` for any `Literal` seed. But
`CometRand.convert` and
`CometRandn.convert` build the proto from `extractSeedFromExpr`:
```scala
protected def extractSeedFromExpr(expr: Expression): Option[Long] = {
expr match {
case Literal(seed: Long, _) => Some(seed)
case Literal(null, _) => Some(0L)
case _ => None
}
}
```
An `IntegerType` literal matches neither case, so `convert` returns `None`
without recording a
reason.
### Expected behavior
`rand` and `randn` with an integer literal seed are converted, as they are
with a `Long` seed.
Spark accepts an `IntegerType` or `LongType` seed and widens an integer seed
to a long
(`RDG.seed` in Spark's `randomExpressions.scala`). If an integer seed is not
supported,
the conversion should record a fallback reason that says so.
### Additional context
A possible fix: accept an `Int` literal in `extractSeedFromExpr` and widen
it to a `Long`, or have
`getSupportLevel` reject what `convert` cannot take, with a reason.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
