Lokesh Kumar created SPARK-11257:
------------------------------------
Summary: Spark dataframe negate filter conditions
Key: SPARK-11257
URL: https://issues.apache.org/jira/browse/SPARK-11257
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.5.0
Environment: Fedora 21 core i5
Reporter: Lokesh Kumar
Fix For: 1.5.0
I am trying to apply a negation of filter condition on the DataFrame as shown
below.
!(`Ship Mode` LIKE '%Truck%')
Which is throwing an exception below
Exception in thread "main" java.lang.RuntimeException: [1.3] failure:
identifier expected
(!(`Ship Mode` LIKE '%Truck%'))
^
at scala.sys.package$.error(package.scala:27)
at
org.apache.spark.sql.catalyst.SqlParser.parseExpression(SqlParser.scala:47)
at org.apache.spark.sql.DataFrame.filter(DataFrame.scala:748)
at Main.main(Main.java:73)
Where as the same kind of negative filter conditions are working fine in MySQL.
Please find below
mysql> select count(*) from audit_log where !(operation like '%Log%' or
operation like '%Proj%');
+----------+
| count(*) |
+----------+
| 129 |
+----------+
1 row in set (0.05 sec)
Can anyone please let me know if this is planned to be fixed in Spark
DataFrames in future releases
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
