[
https://issues.apache.org/jira/browse/SPARK-39560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17557672#comment-17557672
]
Kboh edited comment on SPARK-39560 at 6/22/22 8:10 PM:
-------------------------------------------------------
Closing, appears to be related to Photon. Disabled Photon and it worked.
Workaround:
display(df.withColumn('test', F.when(
F.col('office_status') != 'Active',
F.when(
F.col('_dedupe_count') > 1,
F.lit('good')
).otherwise(F.lit('bad'))
).otherwise(F.lit('bad'))).drop('_dedupe_count'))
was (Author: kboh):
Closing, appears to be related to Photon. Disabled Photon and it worked.
> Certain conditions raise error with when..otherwise but not with where
> ----------------------------------------------------------------------
>
> Key: SPARK-39560
> URL: https://issues.apache.org/jira/browse/SPARK-39560
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 3.3.0
> Environment: DBR 11.0 Photon
> Reporter: Kboh
> Priority: Major
> Attachments: spark_query_bug.png
>
>
> Please view the attachment. Certain conditions raise error with
> when..otherwise but not with where. Hard to debug with ambiguous exception
> message.
> !spark_query_bug.png!
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
