[
https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust updated SPARK-9182:
------------------------------------
Target Version/s: 1.5.0
> filter and groupBy on DataFrames are not passed through to jdbc source
> ----------------------------------------------------------------------
>
> Key: SPARK-9182
> URL: https://issues.apache.org/jira/browse/SPARK-9182
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.1
> Reporter: Greg Rahn
> Priority: Critical
>
> When running all of these API calls, the only one that passes the filter
> through to the backend jdbc source is equality. All filters in these
> commands should be able to be passed through to the jdbc database source.
> {code}
> val url="jdbc:postgresql:grahn"
> val prop = new java.util.Properties
> val emp = sqlContext.read.jdbc(url, "emp", prop)
> emp.filter(emp("sal") === 5000).show()
> emp.filter(emp("sal") < 5000).show()
> emp.filter("sal = 3000").show()
> emp.filter("sal > 2500").show()
> emp.filter("sal >= 2500").show()
> emp.filter("sal < 2500").show()
> emp.filter("sal <= 2500").show()
> emp.filter("sal != 3000").show()
> emp.filter("sal between 3000 and 5000").show()
> emp.filter("ename in ('SCOTT','BLAKE')").show()
> {code}
> We see from the PostgreSQL query log the following is run, and see that only
> equality predicates are passed through.
> {code}
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp WHERE
> sal = 5000
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp WHERE
> sal = 3000
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT
> "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
