xinrong-databricks commented on a change in pull request #32955:
URL: https://github.com/apache/spark/pull/32955#discussion_r658315864
##########
File path: python/pyspark/pandas/spark/functions.py
##########
@@ -32,6 +41,22 @@ def repeat(col: Column, n: Union[int, Column]) -> Column:
return _call_udf(sc, "repeat", _to_java_column(col), n)
+def lit(literal: Any) -> Column:
+ """
+ Creates a Column of literal value.
+ """
+ if isinstance(literal, np.int64):
Review comment:
Adding an input converter might be tricky in this case.
Py4j takes the instance of `java.lang.Integer` as `int` rather than a
`JavaObject`, but the return value of `def convert` is expected to be a
`JavaObject`.
Unfortunately, there is also no wrapper in `java.sql` package for numeric
values (as `java.sql.Timestamp`). And instantiating a `JavaObject` out of a
value seems impossible according to its constructor.
Do you have insights on this? @HyukjinKwon @ueshin
Attached please find my pseudo-code
```py
class SomeConverter(object):
def can_convert(self, obj):
import numpy as np
return isinstance(obj, np.generic)
def convert(self, obj, gateway_client):
Integer = JavaClass("java.lang.Integer", gateway_client)
return Integer.valueOf(obj.item()) # This is an `int`
```
And the exception stack trace looks like
```
Traceback (most recent call last):
File
"/Users/xinrong.meng/spark/python/pyspark/pandas/tests/data_type_ops/test_udt_ops.py",
line 43, in test
print(F.lit(np.int64(1)))
File "/Users/xinrong.meng/spark/python/pyspark/sql/functions.py", line
100, in lit
return col if isinstance(col, Column) else _invoke_function("lit", col)
File "/Users/xinrong.meng/spark/python/pyspark/sql/functions.py", line 58,
in _invoke_function
return Column(jf(*args))
File
"/miniconda2/envs/pyspark-dev-pd-1.1.5/lib/python3.9/site-packages/py4j/java_gateway.py",
line 1313, in __call__
temp_arg._detach()
AttributeError: 'int' object has no attribute '_detach'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
