[
https://issues.apache.org/jira/browse/SPARK-46077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marina Krasilnikova updated SPARK-46077:
----------------------------------------
Description:
code to reproduce:
SparkSession sparkSession = SparkSession
.builder()
.appName("test-app")
.master("local[*]")
.config("spark.sql.timestampType", "TIMESTAMP_NTZ")
.getOrCreate();
String url = "...";
String catalogPropPrefix = "spark.sql.catalog.myc";
sparkSession.conf().set(catalogPropPrefix, JDBCTableCatalog.class.getName());
sparkSession.conf().set(catalogPropPrefix + ".url", url);
Map<String, String> options = new HashMap<>();
options.put("driver", "org.postgresql.Driver");
// options.put("pushDownPredicate", "false"); it works fine if this line is
uncommented
Dataset<Row> dataset = sparkSession.read()
.options(options)
.table("myc.demo.`My table`");
dataset.createOrReplaceTempView("view1");
String sql = "select * from view1 where `my date` = '2021-04-01 00:00:00'";
Dataset<Row> result = sparkSession.sql(sql);
result.show();
result.printSchema();
Field `my date` is of type timestamp. This code results in
org.postgresql.util.PSQLException syntax error
String sql = "select * from view1 where `my date` = to_timestamp('2021-04-01
00:00:00', 'yyyy-MM-dd HH:mm:ss')"; // this query also doesn't work
String sql = "select * from view1 where `my date` = date_trunc('DAY',
to_timestamp('2021-04-01 00:00:00', 'yyyy-MM-dd HH:mm:ss'))"; // but this is OK
Is it a bug or I got something wrong?
was:
code to reproduce:
SparkSession sparkSession = SparkSession
.builder()
.appName("test-app")
.master("local[*]")
.config("spark.sql.timestampType", "TIMESTAMP_NTZ")
.getOrCreate();
String url = "...";
String catalogPropPrefix = "spark.sql.catalog.myc";
sparkSession.conf().set(catalogPropPrefix, JDBCTableCatalog.class.getName());
sparkSession.conf().set(catalogPropPrefix + ".url", url);
Map<String, String> options = new HashMap<>();
options.put("driver", "org.postgresql.Driver");
// options.put("pushDownPredicate", "false"); it works fine if this line is
uncommented
Dataset<Row> dataset = sparkSession.read()
.options(options)
.table("myc.demo.`My table`");
dataset.createOrReplaceTempView("view1");
String sql = "select * from view1 where `my date` = '2021-04-01 00:00:00'";
Dataset<Row> result = sparkSession.sql(sql);
result.show();
result.printSchema();
Field `my date` is of type timestamp. This code results in
org.postgresql.util.PSQLException syntax error , because resulting sql lacks
straight quotes in filter condition. (Something like this "my date" =
2021-04-01T00:00)
> Error in postgresql when pushing down filter by timestamp_ntz field
> -------------------------------------------------------------------
>
> Key: SPARK-46077
> URL: https://issues.apache.org/jira/browse/SPARK-46077
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.0
> Reporter: Marina Krasilnikova
> Priority: Minor
>
> code to reproduce:
> SparkSession sparkSession = SparkSession
> .builder()
> .appName("test-app")
> .master("local[*]")
> .config("spark.sql.timestampType", "TIMESTAMP_NTZ")
> .getOrCreate();
> String url = "...";
> String catalogPropPrefix = "spark.sql.catalog.myc";
> sparkSession.conf().set(catalogPropPrefix, JDBCTableCatalog.class.getName());
> sparkSession.conf().set(catalogPropPrefix + ".url", url);
> Map<String, String> options = new HashMap<>();
> options.put("driver", "org.postgresql.Driver");
> // options.put("pushDownPredicate", "false"); it works fine if this line is
> uncommented
> Dataset<Row> dataset = sparkSession.read()
> .options(options)
> .table("myc.demo.`My table`");
> dataset.createOrReplaceTempView("view1");
> String sql = "select * from view1 where `my date` = '2021-04-01 00:00:00'";
> Dataset<Row> result = sparkSession.sql(sql);
> result.show();
> result.printSchema();
> Field `my date` is of type timestamp. This code results in
> org.postgresql.util.PSQLException syntax error
>
>
> String sql = "select * from view1 where `my date` = to_timestamp('2021-04-01
> 00:00:00', 'yyyy-MM-dd HH:mm:ss')"; // this query also doesn't work
> String sql = "select * from view1 where `my date` = date_trunc('DAY',
> to_timestamp('2021-04-01 00:00:00', 'yyyy-MM-dd HH:mm:ss'))"; // but this is
> OK
>
> Is it a bug or I got something wrong?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
