Greg Rahn created SPARK-9182:
--------------------------------
Summary: filters (where clause) on DataFrames are not pass through
to jdbc source
Key: SPARK-9182
URL: https://issues.apache.org/jira/browse/SPARK-9182
Project: Spark
Issue Type: Bug
Affects Versions: 1.4.1
Reporter: Greg Rahn
When running all of these API calls, the only one that passes the filter
through to the backend jdbc source is equality. All filters in these commands
should be able to be passed through to the jdbc database source.
{code}
val url="jdbc:postgresql:grahn"
val prop = new java.util.Properties
val emp = sqlContext.read.jdbc(url, "emp", prop)
emp.filter(emp("sal") === 5000).show()
emp.filter(emp("sal") < 5000).show()
emp.filter("sal = 3000").show()
emp.filter("sal > 2500").show()
emp.filter("sal >= 2500").show()
emp.filter("sal < 2500").show()
emp.filter("sal <= 2500").show()
emp.filter("sal != 3000").show()
emp.filter("sal between 3000 and 5000").show()
emp.filter("ename in ('SCOTT','BLAKE')").show()
{code}
We see from the PostgreSQL query log the following is run, and see that only
equality predicates are passed through.
{code}
LOG: execute <unnamed>: SET extra_float_digits = 3
LOG: execute <unnamed>: SELECT
"empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp WHERE sal
= 5000
LOG: execute <unnamed>: SET extra_float_digits = 3
LOG: execute <unnamed>: SELECT
"empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
LOG: execute <unnamed>: SET extra_float_digits = 3
LOG: execute <unnamed>: SELECT
"empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp WHERE sal
= 3000
LOG: execute <unnamed>: SET extra_float_digits = 3
LOG: execute <unnamed>: SELECT
"empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
LOG: execute <unnamed>: SET extra_float_digits = 3
LOG: execute <unnamed>: SELECT
"empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
LOG: execute <unnamed>: SET extra_float_digits = 3
LOG: execute <unnamed>: SELECT
"empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
LOG: execute <unnamed>: SET extra_float_digits = 3
LOG: execute <unnamed>: SELECT
"empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
LOG: execute <unnamed>: SET extra_float_digits = 3
LOG: execute <unnamed>: SELECT
"empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
LOG: execute <unnamed>: SET extra_float_digits = 3
LOG: execute <unnamed>: SELECT
"empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
LOG: execute <unnamed>: SET extra_float_digits = 3
LOG: execute <unnamed>: SELECT
"empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
