srielau commented on code in PR #58224:
URL: https://github.com/apache/spark/pull/58224#discussion_r4330577229
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:
##########
@@ -58,6 +58,13 @@ import org.apache.spark.util.Utils
is undefined: it may fail or return arbitrary result.
To deserialize the data with a compatible and evolved schema, the expected
Avro schema can be
set via the corresponding option.
+ CHAR/VARCHAR assignment is accepted when CHAR/VARCHAR are first-class
(this flag or
+ spark.sql.preserveCharVarcharTypeInfo). Then STRING values are assigned
(CHAR padded).
+ Overflow is a malformed record: FAILFAST fails with EXCEED_LIMIT_LENGTH as
the cause;
+ PERMISSIVE returns a null record. spark.sql.legacy.charVarcharAsString
reads the
+ schema as STRING. Otherwise the schema raises
UNSUPPORTED_CHAR_OR_VARCHAR_AS_STRING.
Review Comment:
Thanks. from_avro is now two-state: CHAR/VARCHAR come from
spark.sql.catalyst.type on Avro string fields; assignment when first-class;
otherwise they are read as STRING with no error, no padding, and no length
check. Removed the legacy-flag / UNSUPPORTED_CHAR_OR_VARCHAR_AS_STRING wording.
See dee7bd4d8a1.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:
##########
@@ -1926,6 +1926,14 @@ object JsonExistsExpressionBuilder extends
JsonPathExpressionBuilder {
> SELECT _FUNC_('{"teacher": "Alice", "student": [{"name": "Bob",
"rank": 1}, {"name": "Charlie", "rank": 2}]}', 'STRUCT<teacher: STRING,
student: ARRAY<STRUCT<name: STRING, rank: INT>>>');
{"teacher":"Alice","student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}
""",
+ note = """
+ A CHAR `jsonStr` is not rtrimmed; padding is parsed as trailing whitespace
in the JSON
+ text. CHAR/VARCHAR in `schema` is accepted when CHAR/VARCHAR are
first-class (this flag
Review Comment:
Thanks. The from_json/from_csv/from_xml/from_avro notes (and the PySpark
from_avro docstring) now spell spark.sql.charVarchar.standardSemantics.enabled
instead of "this flag". See dee7bd4d8a1.
##########
python/pyspark/sql/avro/functions.py:
##########
@@ -60,6 +60,14 @@ def from_avro(
Avro is built-in but external data source module since Spark 2.4. Please
deploy the
application as per the deployment section of "Apache Avro Data Source
Guide".
+ CHAR/VARCHAR assignment is accepted when types are first-class (the preview
Review Comment:
Thanks. The PySpark from_avro docstring now names
spark.sql.charVarchar.standardSemantics.enabled, drops the legacy-flag
sentence, and matches the two-state Scala note. See dee7bd4d8a1.
##########
docs/sql-migration-guide.md:
##########
@@ -29,9 +29,12 @@ license: |
## Upgrading from Spark SQL 4.3 to 4.4
- Since Spark 4.4, `spark.sql.execution.arrow.pythonUDF.columnarInput.enabled`
is `false` by default: Arrow-based Python UDFs read rows from an Arrow-backed
upstream operator, such as a Data Source V2 scan or the Arrow cache, as before
Spark 4.2. With columnar input, they kept the columns that they pass through
until the results of their batch came back from Python, and failed or returned
wrong results when the operator reused or released the vectors of the batch in
the meantime. To enable columnar input again, set
`spark.sql.execution.arrow.pythonUDF.columnarInput.enabled` to `true`.
-- Since Spark 4.4, when `spark.sql.charVarchar.standardSemantics.enabled` is
true, `from_json` length-checks JSON object names used as `MAP<CHAR(n), _>` or
`MAP<VARCHAR(n), _>` keys without padding or trimming them. A `CHAR(n)` key
must already be exactly `n` characters, and a `VARCHAR(n)` key must already be
at most `n` characters; duplicate names are kept, exactly as for STRING keys,
and `spark.sql.mapKeyDedupPolicy` is not applied. A key that fails the check
makes the enclosing map a bad record: in `PERMISSIVE` mode the nearest
enclosing struct field becomes `null` (its sibling fields are preserved), and
if the map is inside an array or another map, that whole array or map becomes
`null`; in `FAILFAST` mode parsing fails with
`UNSUPPORTED_JSON_CHAR_VARCHAR_MAP_KEY` (SQLSTATE `0A000`) as the cause of
`MALFORMED_RECORD_IN_PARSING`. The same parser-level check runs for any JSON
scan whose reader schema carries a CHAR/VARCHAR map key, including catalog JSON
tables, since under standa
rd semantics those keys stay CHAR/VARCHAR all the way to the scan: an
off-width key becomes a bad record there too, rather than being padded or
raising `EXCEED_LIMIT_LENGTH`. Unlike `from_json`, keys that pass the check are
then rebuilt read-side, so on the scan path `spark.sql.mapKeyDedupPolicy` does
apply and a duplicate name can raise `DUPLICATED_MAP_KEY`. JSON file scans,
including scans inside persisted views, read the flag from the executing
session.
+- Since Spark 4.4, `spark.sql.charVarchar.standardSemantics.enabled` (default
`false`) is a preview feature that may change in future releases. When true, it
enables first-class `CHAR(n)` and `VARCHAR(n)` semantics throughout schemas and
query plans: `CHAR` -> `VARCHAR` -> `STRING` form a type-precedence family,
explicit casts can return constrained types, least-common-type resolution for
`COALESCE`/`CASE`/`UNION` can return `CHAR` or `VARCHAR`, and string
expressions that transform their input return `STRING`. Assignment and reads
enforce the declared length, padding `CHAR` to `n` characters and raising
`EXCEED_LIMIT_LENGTH` when non-space content does not fit. The default remains
the Spark 3.1 table-schema overlay on `STRING`. This opt-in mode supersedes the
experimental `spark.sql.preserveCharVarcharTypeInfo`, which instead keeps
constrained types through transforming expressions. See [Character String
Types](sql-ref-character-string-types.html) for the preview semantics; the ent
ries below note area-specific behavior that changed. Those later CHAR/VARCHAR
bullets apply only when this preview flag is true, except where they name
`preserveCharVarcharTypeInfo` alone.
+- Since Spark 4.4, when `spark.sql.charVarchar.standardSemantics.enabled` is
true, `from_json` length-checks JSON object names used as `MAP<CHAR(n), _>` or
`MAP<VARCHAR(n), _>` keys without padding or trimming them. A `CHAR(n)` key
must already be exactly `n` characters, and a `VARCHAR(n)` key must already be
at most `n` characters; duplicate names are kept, exactly as for STRING keys,
and `spark.sql.mapKeyDedupPolicy` is not applied. A key that fails the check
makes the enclosing map a bad record: in `PERMISSIVE` mode the nearest
enclosing struct field becomes `null` (its sibling fields are preserved), and
if the map is inside an array or another map, that whole array or map becomes
`null`; in `FAILFAST` mode parsing fails with
`UNSUPPORTED_JSON_CHAR_VARCHAR_MAP_KEY` (SQLSTATE `0A000`) as the cause of
`MALFORMED_RECORD_IN_PARSING`. The same parser-level check runs for any JSON
scan whose reader schema carries a CHAR/VARCHAR map key, including catalog JSON
tables, since under standa
rd semantics those keys stay CHAR/VARCHAR all the way to the scan: an
off-width key becomes a bad record there too, rather than being padded or
raising `EXCEED_LIMIT_LENGTH`. Unlike `from_json`, keys that pass the check are
then rebuilt read-side, so on the scan path `spark.sql.mapKeyDedupPolicy` does
apply and a duplicate name can raise `DUPLICATED_MAP_KEY`. JSON file scans,
including scans inside persisted views, read the flag from the executing
session. The optimizer no longer folds `from_json(to_json(m))` to `m` when that
schema has a CHAR/VARCHAR map key, so the length check still runs on the round
trip.
+- Since Spark 4.4, `from_json`, `from_csv`, `from_xml`, and the corresponding
`schema_of_*` functions do not rtrim a CHAR document argument. CHAR padding is
part of the STRING value after CAST, so it is parsed as trailing spaces in the
document. JSON and XML tolerate that trailing whitespace; CSV can tokenize it
into extra fields or attach it to the last unquoted field (use
`ignoreTrailingWhiteSpace` to restore a trimmed last-field parse).
Review Comment:
Thanks. Dropped the false 4.3 rtrim bullet and replaced it with the
schema_of_* first-class-docs change. See dee7bd4d8a1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
