HyukjinKwon opened a new pull request, #196: URL: https://github.com/apache/spark-connect-rust/pull/196
### What changes were proposed in this pull request? [SPARK-60994](https://issues.apache.org/jira/browse/SPARK-60994): a literal of a Python `Decimal` now has the precision 10 and scale 0 of `DecimalType()`, as PySpark's `LiteralExpression` of a `Decimal` does. That covers `lit()` and any other Column built from one, such as `F.col("a") == Decimal("1.5")`. The server widens them to the precision and scale of the value. Before, the literal had the precision and scale of the value itself. ### Why are the changes needed? The literals, and the expressions over them, had other types than in PySpark. Against a Spark 4.3.0 server: | Column | PySpark 4.3.0 and now | Before | | --- | --- | --- | | `lit(Decimal("1.5"))` | `decimal(10,1)` | `decimal(2,1)` | | `lit(Decimal(0))` | `decimal(10,0)` | `decimal(1,0)` | | `lit(Decimal("-0.001"))` | `decimal(10,3)` | `decimal(3,3)` | | `lit(Decimal("1.5")) * lit(Decimal("2.25"))` | `decimal(21,3)` | `decimal(6,3)` | So an expression could overflow where PySpark's does not. A literal of a value with more than 10 digits, such as `Decimal("123456789012.5")`, keeps its own precision in both. ### Does this PR introduce _any_ user-facing change? Yes, decimal literals have the types of PySpark 4.3.0. ### How was this patch tested? - Against a local Spark 4.3.0 server, the types and values of the literals above give the same results as official PySpark 4.3.0. - A new offline test checks, through a mock Connect server, the decimal literal that `lit()` and a comparison send. A new live test checks the types and values above. It also passes with official PySpark 4.3.0. Both fail without this change. - The lib tests and every integration test, e2e included, against the local server (648 passed, and 123 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (721 passed, 9 skipped), and the live drop-in and Arrow suites (77 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
