L. C. Hsieh created SPARK-61442:
-----------------------------------

             Summary: Parse the atomic and parameterized type strings of JSON 
and of DDL with one parser in Spark Connect Rust
                 Key: SPARK-61442
                 URL: https://issues.apache.org/jira/browse/SPARK-61442
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: L. C. Hsieh


In Spark Connect Rust, DataType::from_json and DataType::from_ddl 
(crates/spark-connect/src/types.rs) each have their own table of atomic type 
names and their own chain of prefix checks for decimal(p,s), char(n), 
varchar(n), time(p), timestamp_ntz(p), intervals and collated strings, and each 
reads "string collate <name>" with separate code. The leaf functions that they 
do share take no account of which grammar is being read, so a change made for 
one grammar has changed the other without that being decided, and both differ 
from PySpark in places:

* JSON reads decimal(5), which PySpark's _FIXED_DECIMAL does not (it came with 
the DDL form).
* JSON does not read "interval second to day", the json() of 
DayTimeIntervalType(3, 0), which PySpark's _parse_datatype_json_value reads, so 
the JSON of such a type does not read back.
* DDL reads a negative scale, decimal(10,-2), which is not an INTEGER_VALUE of 
Spark's grammar.
* Both read a signed argument such as char(-3), time(+6) or decimal(+5,2); 
PySpark's patterns are \d+.
* "string collate" without a name and "string collateX" are read by both, by 
DDL as a string of the empty collation; JSON does not read "string  collate  
X", which PySpark's \s+ does; neither requires the name to be \w+.
* DDL does not read "string collate system.builtin.<name>" as 
CollationFactory.resolveFullyQualifiedName does.

Both parsers should go through one parser of the atomic names and one of the 
parameterized forms, which is told the grammar (JSON or DDL) for the places 
where the two grammars of PySpark really differ: letter case and whitespace, 
whether a decimal needs its scale and may have a negative one, the order of the 
fields of an interval range, and whether the name of a collation is kept as 
written (JSON, as StringType(name)) or given in the canonical form of the 
server (DDL). A table-driven test should feed each form through both parsers 
and assert the same type, the same error, or the listed difference.

Neither parser checks the range of the precision or scale of a decimal, as 
PySpark's does not; that stays as SPARK-61423 left it (checked at the 
conversion to Arrow).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to