Dejan Krakovic created SPARK-61298:
--------------------------------------
Summary: Settle one rule for FILE in the output of row-returning
Python functions
Key: SPARK-61298
URL: https://issues.apache.org/jira/browse/SPARK-61298
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Dejan Krakovic
Follow-up of SPARK-60386 (https://github.com/apache/spark/pull/59640).
The query-time check of user-defined routine return types skips non-scalar
Python UDFs (mapInPandas, mapInArrow, applyInPandas and other grouped-map
functions), because their data type is the schema of the rows they return. So
these functions may return a FILE EXTERNAL column, while a Python UDTF
registered with a static return type may not. Settle one rule for FILE in the
output of row-returning Python functions, with tests, together with the PySpark
support of the FILE type.
Review thread: https://github.com/apache/spark/pull/59640#discussion_r4353281773
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
