[
https://issues.apache.org/jira/browse/SPARK-50126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17924706#comment-17924706
]
Sakthi commented on SPARK-50126:
--------------------------------
Hello all, can I get some reviews here in the above PR please?
[~panbingkun]
> PySpark expr() (expression) SQL Function returns None in Spark Connect
> ----------------------------------------------------------------------
>
> Key: SPARK-50126
> URL: https://issues.apache.org/jira/browse/SPARK-50126
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 3.5.0, 3.5.1, 3.5.2, 3.5.3
> Environment: Spark 3.5.3
> Python 3.9.20
> Reporter: Nick Herzig
> Priority: Major
> Labels: connect, expression, pull-request-available, pyspark,
> pyspark.sql.functions, remote
>
> When PySpark is launched with remote Spark Connect configuration, calls to
> pyspark.sql.functions.expr always return `Column<None>`.
>
> PySpark without Spark Connect
> {code:java}
> $ pyspark
> ...
> >>> from pyspark.sql.functions import expr
> >>> e = expr("*")
> >>> print(e)
> Column<'unresolvedstar()'>
> {code}
> PySpark with Spark Connect
> {code:java}
> $ pyspark --remote "sc://localhost:15002"
> ...
> Client connected to the Spark Connect server at localhost:15002
> SparkSession available as 'spark'.
> >>> from pyspark.sql.functions import expr
> >>> e = expr("*")
> >>> print(e)
> Column<'None'>
> {code}
> Additional example
> {code:java}
> $ pyspark
> ...
> Spark context available as 'sc' (master = local[*], app id =
> local-1729876117745).
> SparkSession available as 'spark'.
> >>> from pyspark.sql.functions import expr
> >>> expr("*")
> Column<'unresolvedstar()'>
> >>> SparkSession.builder.master("local[*]").getOrCreate().stop()
> >>> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> >>> expr("*")
> Column<'None'> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
