[
https://issues.apache.org/jira/browse/SPARK-59098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59098:
-----------------------------------
Labels: pull-request-available (was: )
> format_string/printf throws IllegalFormatConversionException for decimal
> arguments with %f
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-59098
> URL: https://issues.apache.org/jira/browse/SPARK-59098
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 5.0.0
> Reporter: Joel Robin
> Priority: Major
> Labels: pull-request-available
>
> {{printf}} and {{format_string}} throw if you pass a decimal to {{%f}} (or
> {{%e}}, {{%g}}, {{%a}}):
> {code:sql}
> spark-sql> SELECT printf('%.2f', 1.5);
> java.lang.IllegalArgumentException: f != org.apache.spark.sql.types.Decimal
> {code}
> No cast, nothing unusual. {{1.5}} is {{decimal(2,1)}}, so that query is just
> "format a decimal with %f". Decimal columns fail the same way.
> The error is hard to act on. {{IllegalFormatConversionException}} extends
> {{IllegalArgumentException}}, so PySpark surfaces it as a plain
> {{IllegalArgumentException}} and nothing in the message or the traceback
> mentions {{format_string}}. And {{org.apache.spark.sql.types.Decimal}} is an
> internal class - it isn't a type the user wrote.
> {{%f}} on a NULL decimal returns {{"null"}} and doesn't throw. So this hides
> in
> pipelines that format a sparse decimal column, and breaks the day real values
> show up.
> h3. Repro
> {code:sql}
> SELECT printf('%.2f', 1.5);
> {code}
> And via a column, which goes through whole-stage codegen rather than constant
> folding:
> {code:python}
> spark.range(3).selectExpr("CAST(id + 0.5 AS DECIMAL(10,2)) AS
> d").createOrReplaceTempView("t")
> spark.sql("SELECT format_string('%.2f', d) FROM t").show()
> {code}
> Both fail on 4.0.0 (pyspark 4.0.0, local[1], OpenJDK 17.0.15). The codegen
> one throws from
> generated code:
> {code:none}
> java.util.IllegalFormatConversionException: f !=
> org.apache.spark.sql.types.Decimal
> at
> java.base/java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4442)
> at
> java.base/java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2976)
> at java.base/java.util.Formatter.format(Formatter.java:2689)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
> Source)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
