Paul Deng created SPARK-36757:
---------------------------------
Summary: countDistinct converts to count when converted with to
SQL string
Key: SPARK-36757
URL: https://issues.apache.org/jira/browse/SPARK-36757
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.1.1
Reporter: Paul Deng
In pyspark=3.1.1
{code:python}
> F.countDistinct(F.col("bar"))
Column<'count(bar)'>{code}
In pyspark=2.4
{code:python}
> F.countDistinct(F.col("bar"))
Column<b'count(DISTINCT foo)'>{code}
This creates problems when trying to convert the pyspark function into SQL code
to compose SQL queries via:
{code:python}
> F.countDistinct(F.col("bar"))._jc.toString()
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
