Serge Rielau created SPARK-60479:
------------------------------------
Summary: Push down CHAR filters when read-side CHAR padding is
enabled
Key: SPARK-60479
URL: https://issues.apache.org/jira/browse/SPARK-60479
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Serge Rielau
spark.sql.readSideCharPadding defaults to true. For CHAR columns,
ApplyCharTypePadding wraps the scan in
StaticInvoke(CharVarcharCodegenUtils.readSidePadding, col, n) and strips CHAR
metadata from the scan output.
JDBC / DSv2 filter translation (V2ExpressionBuilder / PushablePredicate) does
not understand that StaticInvoke. A predicate such as CHAR_COL = 'Y' stays on
the Spark side. Only pushable pieces such as IS NOT NULL go to the remote
system, so JDBC tables (including Oracle CHAR(n)) are fully scanned.
With the flag off, the same predicate is rewritten as RPAD(col, n, ' ') = 'Y'.
That form is translatable after SPARK-50692 / SPARK-51321, and dialects that
implement RPAD (H2, Postgres, SQL Server) push it. Oracle's OSS dialect still
does not list RPAD in isSupportedFunction, so Oracle JDBC would not push even
that form today.
The padding rewrite is required for CHAR semantics. Pushdown should still
happen, using an equivalent remote predicate.
Expected
With spark.sql.readSideCharPadding=true (and with
spark.sql.charVarchar.standardSemantics.enabled=true, which also pads on read),
CHAR comparisons and similar predicates against JDBC/DSv2 CHAR columns should
push down when the dialect can evaluate the equivalent padding.
Example: WHERE prod_version_current = 'Y' on Oracle CHAR(1) should appear in
the remote SQL as something like RPAD("PROD_VERSION_CURRENT", 1, ' ') = 'Y' (or
an equivalent CHAR comparison), not only IS NOT NULL.
Spark-side read padding of the returned values can stay.
Proposed approach
Translate StaticInvoke of readSidePadding (and, if safe, charTypeReadSideCheck)
in V2ExpressionBuilder to the same V2 RPAD expression already used for
StringRPad, so existing RPAD-capable dialects keep working with the default
config.
Add RPAD to OracleDialect.isSupportedFunction (Oracle already has RPAD).
Tests: JDBC V2 CHAR column + equality / IN / IS NOT NULL with the flag on and
off; assert the pushed SQL contains the comparison, not only null checks. Cover
at least H2; Oracle if the docker suite is in reach.
Do not push a bare col = 'Y' and rely on remote CHAR padding. Spark with
read-side padding uses string equality on the padded value (RPAD(col, n, ' ') =
lit). For CHAR(n) vs a shorter literal those are not the same as Oracle CHAR
comparison.
Related
SPARK-40697: read-side CHAR padding (default true)
SPARK-50692: RPAD pushdown for predicate-side CHAR padding
SPARK-51321: SQL Server LPAD/RPAD
SPARK-58794: full CHAR/VARCHAR (this bug is independent of standard-semantics;
it hits the default 3.4+ path)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
