[
https://issues.apache.org/jira/browse/SPARK-47646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun closed SPARK-47646.
---------------------------------
> try_to_number fails with NPE for malformed input
> ------------------------------------------------
>
> Key: SPARK-47646
> URL: https://issues.apache.org/jira/browse/SPARK-47646
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.1, 4.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.5.2, 3.4.3, 4.0.0
>
>
> {code}
> scala> import org.apache.spark.sql.functions._
> | val df =
> spark.createDataset(spark.sparkContext.parallelize(Seq("11")))
> | df.select(try_to_number($"value", lit("$99.99"))).show()
> 24/03/29 16:19:16 ERROR Executor: Exception in task 10.0 in stage 2.0 (TID 15)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.spark.sql.types.Decimal.toPlainString()" because "<local7>" is
> null
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.serializefromobject_doConsume_0$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:50)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
> at
> org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:894)
> at
> org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:894)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:368)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:332)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
