Luis Lozano Coira created SPARK-38288:
-----------------------------------------
Summary: Aggregate push down doesnt work using Spark SQL jdbc
datasource with postgresql
Key: SPARK-38288
URL: https://issues.apache.org/jira/browse/SPARK-38288
Project: Spark
Issue Type: Question
Components: SQL
Affects Versions: 3.2.1
Reporter: Luis Lozano Coira
I am establishing a connection with postgresql using the Spark SQL jdbc
datasource. I have started the spark shell including the postgres driver and I
can connect and execute queries without problems. I am using this statement:
{code:java}
val df = spark.read.format("jdbc").option("url",
"jdbc:postgresql://host:port/").option("driver",
"org.postgresql.Driver").option("dbtable", "test").option("user",
"postgres").option("password",
"*******").option("pushDownAggregate",true).load()
{code}
I am adding the pushDownAggregate option because I would like the aggregations
are delegated to the source. But for some reason this is not happening.
Reviewing this pull request, it seems that this feature should be merged into
3.2. [https://github.com/apache/spark/pull/29695]
I am making the aggregations considering the mentioned limitations. An example
case where I don't see pushdown being done would be this one:
{code:java}
df.groupBy("name").max("age").show()
{code}
What could be the problem? Should pushDownAggregate work in this case?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
