Serge Rielau created SPARK-50560:
------------------------------------
Summary: randstr(-1) causes Java stacktrace and argument types are
too restrictive
Key: SPARK-50560
URL: https://issues.apache.org/jira/browse/SPARK-50560
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Serge Rielau
randstr(-1) result in a runtime stacktrace:
SparkException: Job aborted due to stage failure: Task 0 in stage 18.0 failed 4
times, most recent failure: Lost task 0.3 in stage 18.0 (TID 111)
(10.68.143.147 executor 0): java.lang.NegativeArraySizeException: -1 at
org.apache.spark.sql.catalyst.expressions.ExpressionImplUtils.randStr(ExpressionImplUtils.java:327)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
Source) at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
Source) at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
Further, it is unusual not to allow static numeric types to down cast or upcast
to the supported type.
In this case both length and see must be SMALLINT or INT.
TINYINT shouls work for sure
And BIGINT should work as long as teh value itself is not too large.
In fact compare to substr('hello', 1.0, 1.0) which happily works!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
