srielau commented on PR #58549: URL: https://github.com/apache/spark/pull/58549#issuecomment-5590971641
Thanks for the questions. No new configuration is needed, and this PR does not add one. CHAR/VARCHAR previously failed at these Arrow boundaries because the mapper rejected them, not because the semantics were undefined. Existing configs already own that: `spark.sql.legacy.charVarcharAsString`, `spark.sql.preserveCharVarcharTypeInfo`, and `spark.sql.charVarchar.standardSemantics.enabled`. Assignment uses `CharVarcharUtils.shouldApplyWriteSideLengthCheck`, so first-class / standard mode still pads CHAR and rejects over-length VARCHAR, and pure legacy-as-string still skips those checks. We audited `to_arrow_type` / `to_arrow_schema` call sites. The mapper itself only normalizes CHAR/VARCHAR to physical Arrow STRING; it does not encode assignment policy. Each consumer then either stays unsupported or applies write-side checks on the JVM ingest path: - Admitted, with JVM assignment checks: scalar batched / Arrow / pandas / pandas-iter UDFs, plus `createDataFrame` / `toArrow` via `ArrowConverters`. - Explicitly rejected, including nested CHAR/VARCHAR and analyze-derived schemas: Arrow UDTFs (`SQL_ARROW_UDTF` / `SQL_ARROW_TABLE_UDF`) on both classic and Connect. `ArrowEvalPythonUDTFExec` still has no physical-STRING + write-side consumer. - Default-deny for other Python eval types (grouped / map / cogrouped / window / incremental / TransformWithState): CHAR/VARCHAR return types raise `PySparkNotImplementedError`. Python DataSource is the remaining gap from that audit. `plan_data_source_read.py` now succeeds at `to_arrow_schema` for CHAR/VARCHAR (physical STRING), but the JVM reader still goes through `MapInBatchEvaluatorFactory`, which projects identity and does not call `stringLengthCheck`. I will follow up by rejecting CHAR/VARCHAR on that path until it has the same assignment consumer as scalar Arrow UDFs, rather than letting the mapper change silently admit them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
