zhengruifeng opened a new pull request, #39016:
URL: https://github.com/apache/spark/pull/39016
### What changes were proposed in this pull request?
check the bounds of integer and choose the correct datatypes
### Why are the changes needed?
to match pyspark:
```
In [15]: int_max = 2147483647
In [16]: long_max = 9223372036854775807
In [17]: sdf = spark.range(0, 1)
In [18]: sdf.select(lit(0), lit(int_max), lit(int_max + 1), lit(long_max))
Out[18]: DataFrame[0: int, 2147483647: int, 2147483648: bigint,
9223372036854775807: bigint]
In [19]: sdf.select(lit(0), lit(int_max), lit(int_max + 1), lit(long_max),
lit(long_max + 1))
---------------------------------------------------------------------------
Py4JError Traceback (most recent call last)
Cell In[19], line 1
----> 1 sdf.select(lit(0), lit(int_max), lit(int_max + 1), lit(long_max),
lit(long_max + 1))
File ~/Dev/spark/python/pyspark/sql/functions.py:179, in lit(col)
177 if dt is not None:
178 return _invoke_function("lit", col).astype(dt)
--> 179 return _invoke_function("lit", col)
File ~/Dev/spark/python/pyspark/sql/functions.py:88, in
_invoke_function(name, *args)
86 assert SparkContext._active_spark_context is not None
87 jf = _get_jvm_function(name, SparkContext._active_spark_context)
---> 88 return Column(jf(*args))
...
File ~/Dev/spark/python/pyspark/sql/utils.py:199, in
capture_sql_exception.<locals>.deco(*a, **kw)
197 def deco(*a: Any, **kw: Any) -> Any:
198 try:
--> 199 return f(*a, **kw)
200 except Py4JJavaError as e:
201 converted = convert_exception(e.java_exception)
File ~/Dev/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py:330, in
get_return_value(answer, gateway_client, target_id, name)
326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)
329 else:
--> 330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}.
Trace:\n{3}\n".
332 format(target_id, ".", name, value))
333 else:
334 raise Py4JError(
335 "An error occurred while calling {0}{1}{2}".
336 format(target_id, ".", name))
Py4JError: An error occurred while calling
z:org.apache.spark.sql.functions.lit. Trace:
java.lang.NumberFormatException: For input string: "9223372036854775808"
at
java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
```
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
added ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
