Michael H created SPARK-24210:
---------------------------------
Summary: incorrect handling of boolean expressions when using
column in expressions in pyspark.sql.DataFrame filter function
Key: SPARK-24210
URL: https://issues.apache.org/jira/browse/SPARK-24210
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 2.1.2
Reporter: Michael H
{code:python}
ex = spark.createDataFrame([
('r0', 0, 0),
('r1', 0, 1),
('r2', 1, 0),
('r3', 1, 1)]\
, "row: string, c1: int, c2: int")
#KO: returns r1 and r3
ex.filter(('c1 = 1') and ('c2 = 1')).show()
#OK, raises an exception
ex.filter(('c1 == 1') & ('c2 == 1')).show()
#KO: returns r0 and r3
ex.filter('c1 = 1 & c2 = 1').show()
#KO: returns r0 and r3
ex.filter('c1 == 1 & c2 == 1').show()
#OK: returns r3 only
ex.filter('c1 = 1 and c2 = 1').show()
#OK: returns r3 only
ex.filter('c1 == 1 and c2 == 1').show()
{code}
building the expressions using {code}ex.c1{code} or {code}ex['c1']{code} we
don't have this.
Issue seems related with
https://github.com/apache/spark/pull/6961
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
