[
https://issues.apache.org/jira/browse/SPARK-42450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vadim updated SPARK-42450:
--------------------------
Description:
dataset.where()/filter() omit string quotes if where IN clause has more than 10
operands. With datasourceV1 works as expected.
Attached files: java-code.txt, stacktrace.txt
- Spark verison 3.3.0
- Scala version 2.12
- DatasourceV2
- Postgres
- Postrgres JDBC Driver: 42+
- Java8
*Expected query:*
SELECT
"flight_id",
"flight_no"
FROM
"bookings"."flights"
WHERE
(
"flight_no"IN (
'55',
'826',
'845',
'799',
'561',
'39',
'385',
'549',
'576',
'15',
'857',
'248',
'324',
'569',
'267'
)
)
*actual query:*
SELECT
"flight_id",
"flight_no",
FROM
"bookings"."flights"
WHERE
(
"flight_no"IN (
55,
826,
845,
799,
561,
39,
385,
549,
576,
15,
857,
248,
324,
569,
267
)
)
was:
dataset.where()/filter() omit string quotes if where IN clause has more than 10
operands. With datasourceV1 works as expected.
Attached files: java-code.txt, stacktrace.txt
- Spark verison 3.3.0
- Scala version 2.12
- DatasourceV2
- Postgres
- Postrgres JDBC Driver: 42+
- Java8
*Expected query:*
SELECT
"flight_id",
"flight_no"
FROM
"bookings"."flights"
WHERE
(
"flight_no"IN (
'55',
'826',
'845',
'799',
'561',
'39',
'385',
'549',
'576',
'15',
'857',
'248',
'324',
'569',
'267'
)
)
*actual query:*
SELECT
"flight_id",
"flight_no",
FROM
"bookings"."flights"
WHERE
(
"flight_no"IN (
55,
826,
845,
799,
561,
39,
385,
549,
576,
15,
857,
248,
324,
569,
267
)
)
> dataset.where() omit quotes if where IN clause has more than 10 operands
> ------------------------------------------------------------------------
>
> Key: SPARK-42450
> URL: https://issues.apache.org/jira/browse/SPARK-42450
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Vadim
> Priority: Major
> Fix For: 3.3.2, 3.4.0
>
> Attachments: java-code.txt, stacktrace.txt
>
>
> dataset.where()/filter() omit string quotes if where IN clause has more than
> 10 operands. With datasourceV1 works as expected.
> Attached files: java-code.txt, stacktrace.txt
> - Spark verison 3.3.0
> - Scala version 2.12
> - DatasourceV2
> - Postgres
> - Postrgres JDBC Driver: 42+
> - Java8
> *Expected query:*
> SELECT
> "flight_id",
> "flight_no"
> FROM
> "bookings"."flights"
> WHERE
> (
> "flight_no"IN (
> '55',
> '826',
> '845',
> '799',
> '561',
> '39',
> '385',
> '549',
> '576',
> '15',
> '857',
> '248',
> '324',
> '569',
> '267'
> )
> )
> *actual query:*
> SELECT
> "flight_id",
> "flight_no",
> FROM
> "bookings"."flights"
> WHERE
> (
> "flight_no"IN (
> 55,
> 826,
> 845,
> 799,
> 561,
> 39,
> 385,
> 549,
> 576,
> 15,
> 857,
> 248,
> 324,
> 569,
> 267
> )
> )
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
