srielau commented on code in PR #59467:
URL: https://github.com/apache/spark/pull/59467#discussion_r4275231071
##########
sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala:
##########
@@ -3999,4 +4087,36 @@ class DSV2CharVarcharTestSuite extends
CharVarcharTestSuite
}
}
}
+
+ test("SPARK-59016: V2 column prune does not re-stamp CHAR padding
breadcrumb") {
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
Review Comment:
Dropped the `charVarcharStandardSemantics` disjunct so `toOutputAttrs` keys
only on the relation's own padding state. Added default-mode cases next to this
test: padded (`readSideCharPadding=true`) asserts the direct `toOutputAttrs`
CHAR-readSchema call drops the marker, and unpadded
(`readSideCharPadding=false`) asserts a still-stamped relation keeps it on both
that call and the pruned scan. The end-to-end InMemory scan is no longer the
regression guard for the padded path.
##########
sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala:
##########
@@ -1540,6 +1544,41 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
"%s(c, c, c)", "%s(array(array(c)))", "%s(named_struct('x', c))",
"%s(map(c, 1))", "%s(map(1, c))")
+ // Signatures the scalar/nested templates do not typecheck: 3-string
(translate),
+ // (string, string, int) overlay/split_part/substring_index, format_string's
format
+ // arg, HOF lambdas, and window OVER. '%%s' survives String.format as '%s'.
+ private val inventoryGatedShapes = Seq(
+ "%s(c, 'x', 1)", "%s(c, '-', 1)", "%s(c, c, 1)", "%s(c, 'ab', 'xy')",
+ "%s('%%s', c)", "%s(array(c), x -> x)", "%s(array(c), x -> reverse(x))",
+ "%s(c) OVER (ORDER BY 1)", "%s(c, 1) OVER (ORDER BY 1)")
+
+ // These must typecheck (not Try.toOption). split_part / substring_index /
+ // translate reject case-insensitive collations (NonCSAI).
Review Comment:
You are right: `StringTypeNonCSAICollation` rejects only case-sensitive
accent-insensitive collations (e.g. `UNICODE_AI`), not `UTF8_LCASE`. The
required-call test now runs the full list on the UTF8_LCASE column as well, and
the comment matches that.
##########
sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala:
##########
@@ -1540,6 +1544,41 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
"%s(c, c, c)", "%s(array(array(c)))", "%s(named_struct('x', c))",
"%s(map(c, 1))", "%s(map(1, c))")
+ // Signatures the scalar/nested templates do not typecheck: 3-string
(translate),
Review Comment:
Reworded. The comment now says only HOF lambdas and window OVER are out of
reach of the earlier templates, and that the rest of the gated shapes add
literal argument positions and a format string that actually consumes `c`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
