Marina Krasilnikova created SPARK-46077:
-------------------------------------------
Summary: Error in postgresql when pushing down filter by
timestamp_ntz field
Key: SPARK-46077
URL: https://issues.apache.org/jira/browse/SPARK-46077
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.5.0
Reporter: Marina Krasilnikova
code to reproduce:
SparkSession sparkSession = SparkSession
.builder()
.appName("test-app")
.master("local[*]")
.config("spark.sql.timestampType", "TIMESTAMP_NTZ")
.getOrCreate();
String url = "...";
String catalogPropPrefix = "spark.sql.catalog.myc";
sparkSession.conf().set(catalogPropPrefix, JDBCTableCatalog.class.getName());
sparkSession.conf().set(catalogPropPrefix + ".url", url);
Map<String, String> options = new HashMap<>();
options.put("driver", "org.postgresql.Driver");
// options.put("pushDownPredicate", "false"); it works fine if this line is
uncommented
Dataset<Row> dataset = sparkSession.read()
.options(options)
.table("myc.demo.`My table`");
dataset.createOrReplaceTempView("view1");
String sql = "select * from view1 where `my date` = '2021-04-01 00:00:00'";
Dataset<Row> result = sparkSession.sql(sql);
result.show();
result.printSchema();
Field `my date` is of type timestamp. This code results in
org.postgresql.util.PSQLException syntax error , because resulting sql lacks
straight quotes in filter condition. (Something like this "my date" =
2021-04-01T00:00)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
