stevomitric commented on code in PR #60059:
URL: https://github.com/apache/spark/pull/60059#discussion_r4483920481
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala:
##########
@@ -463,6 +463,13 @@ case class OuterReference(e: NamedExpression)
extends LeafExpression with NamedExpression with Unevaluable {
override def dataType: DataType = e.dataType
override def nullable: Boolean = e.nullable
+ // Forward only the raw CHAR/VARCHAR type when aliased. Other keys are not
forwarded; some,
+ // such as the SQL UDF parameter and auto-generated alias markers, must not
leak.
+ override def metadata: Metadata =
CharVarcharUtils.getRawTypeString(e.metadata) match {
Review Comment:
done
##########
docs/sql-migration-guide.md:
##########
@@ -52,6 +52,7 @@ license: |
- Since Spark 4.4, the Oracle JDBC datasource maps bare `NUMBER` columns
(declared without explicit precision or scale) to `DecimalType(38, 18)` instead
of `DecimalType(38, 10)`. This uses `DecimalType.DEFAULT_SCALE` and preserves
up to 18 fractional digits instead of 10, reducing silent precision loss.
However, the integer range shrinks from 28 to 20 digits; bare `NUMBER` values
with more than 20 integer digits that previously read correctly will raise
`NUMERIC_VALUE_OUT_OF_RANGE`. The same change applies to Oracle `FLOAT`
columns, which JDBC reports as `NUMERIC` with `scale = -127`. Columns with
explicit `NUMBER(p,s)` are not affected. To restore the previous
`DecimalType(38, 10)` mapping, set
`spark.sql.legacy.oracle.numberMapping.enabled` to `true`.
- Since Spark 4.4, when `spark.sql.charVarchar.standardSemantics.enabled` is
false and `spark.sql.readSideCharPadding` is true (the default), JDBC Data
Source V2 catalog tables on H2, PostgreSQL, SQL Server, and Oracle push CHAR
equality, inequality, null-safe equality, and `IN`/`NOT IN`. In 4.3 the default
read-side padding wrap kept those filters in Spark. Pushed predicates use the
remote collation, so a case-insensitive SQL Server collation can match a stored
`'ABC'` with `flag = 'abc'` while a Spark-local comparison would not. The V1
path (`spark.read.jdbc` and `USING jdbc`) still evaluates these CHAR filters in
Spark. PostgreSQL's internal `"char"` type (JDBC type name `char`) is mapped to
`STRING` instead of `CHAR(1)`, so Spark no longer blank-pads its empty value.
- Since Spark 4.4, when `spark.sql.charVarchar.standardSemantics.enabled` is
false and `spark.sql.readSideCharPadding` is true (the default), `IS [NOT]
NULL` over a CHAR scan column is unwrapped to the raw column for every Data
Source V2 source, including every JDBC dialect. Previously the padding wrap
kept that null check in Spark.
+- Since Spark 4.4, a CHAR/VARCHAR column of the outer query that a LATERAL or
correlated subquery returns keeps its CHAR/VARCHAR type, as an inner column
does. Previously, comparisons against it were not padded and could silently
return no rows, and CTAS and views over it stored `STRING`. This affected `t.c
AS k` in all earlier versions, `t.*` since Spark 4.2, and `t.c` since Spark
4.3. To keep storing `STRING`, set `spark.sql.legacy.charVarcharAsString` to
`true`, which treats all CHAR/VARCHAR columns as `STRING`.
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
