[
https://issues.apache.org/jira/browse/SPARK-61442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61442:
-----------------------------------
Labels: pull-request-available (was: )
> Parse the atomic and parameterized type strings of JSON and of DDL with one
> parser in Spark Connect Rust
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61442
> URL: https://issues.apache.org/jira/browse/SPARK-61442
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In Spark Connect Rust, DataType::from_json and DataType::from_ddl
> (crates/spark-connect/src/types.rs) each have their own table of atomic type
> names and their own chain of prefix checks for decimal(p,s), char(n),
> varchar(n), time(p), timestamp_ntz(p), intervals and collated strings, and
> each reads "string collate <name>" with separate code. The leaf functions
> that they do share take no account of which grammar is being read, so a
> change made for one grammar has changed the other without that being decided,
> and both differ from PySpark in places:
> * JSON reads decimal(5), which PySpark's _FIXED_DECIMAL does not (it came
> with the DDL form).
> * JSON does not read "interval second to day", the json() of
> DayTimeIntervalType(3, 0), which PySpark's _parse_datatype_json_value reads,
> so the JSON of such a type does not read back.
> * DDL reads a negative scale, decimal(10,-2), which is not an INTEGER_VALUE
> of Spark's grammar.
> * Both read a signed argument such as char(-3), time(+6) or decimal(+5,2);
> PySpark's patterns are \d+.
> * "string collate" without a name and "string collateX" are read by both, by
> DDL as a string of the empty collation; JSON does not read "string collate
> X", which PySpark's \s+ does; neither requires the name to be \w+.
> * DDL does not read "string collate system.builtin.<name>" as
> CollationFactory.resolveFullyQualifiedName does.
> Both parsers should go through one parser of the atomic names and one of the
> parameterized forms, which is told the grammar (JSON or DDL) for the places
> where the two grammars of PySpark really differ: letter case and whitespace,
> whether a decimal needs its scale and may have a negative one, the order of
> the fields of an interval range, and whether the name of a collation is kept
> as written (JSON, as StringType(name)) or given in the canonical form of the
> server (DDL). A table-driven test should feed each form through both parsers
> and assert the same type, the same error, or the listed difference.
> Neither parser checks the range of the precision or scale of a decimal, as
> PySpark's does not; that stays as SPARK-61423 left it (checked at the
> conversion to Arrow).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]