mbutrovich commented on code in PR #391:
URL: https://github.com/apache/datafusion-spark/pull/391#discussion_r4645351399


##########
crates/sqllogictest/scripts/pyspark_slt_runner.py:
##########
@@ -788,6 +865,123 @@ def translate(sql: str, time_zone: str = "UTC") -> str:
     return "".join(out)
 
 
+# Keywords after which a `-` is a sign, not a subtraction, and a `(` does
+# not start the arguments of a call.
+_KEYWORDS = {
+    "ALL", "AND", "ANY", "AS", "BETWEEN", "BY", "CASE", "DISTINCT", "ELSE",
+    "EXISTS", "FROM", "HAVING", "IN", "IS", "JOIN", "LIKE", "LIMIT", "NOT",
+    "ON", "OR", "RETURN", "SELECT", "THEN", "VALUES", "WHEN", "WHERE",
+}
+
+
+# The type names of typed literals, such as DATE '2024-01-01'.
+_TYPED_LITERALS = ("DATE", "TIME", "INTERVAL", "X", *_TIMESTAMP_TYPES)
+
+# Brackets that nest in a type name, such as `array<decimal(10, 2)>`.
+_TYPE_NESTING = {"(": 1, "<": 1, ")": -1, ">": -1}
+
+_UNTRANSLATABLE = "a `::` cast this runner cannot translate"
+
+
+def rewrite_double_colons(sql: str) -> str:
+    """Replace each `expr::type` with `CAST(expr AS type)`.
+
+    Spark 3.5 does not parse `::`. As in Spark 4, `::` applies to the primary
+    expression before it: a literal, a column, a call, a parenthesized
+    expression or a subscript. A sign belongs to a number literal, so
+    `-1::int` is `CAST(-1 AS int)`, but `-(1)::int` is `-CAST((1) AS int)`.
+    """
+    tokens = _tokenize(sql)
+    out: list[tuple[str, str]] = []
+    i = 0
+    while i < len(tokens):
+        if tokens[i][0] != "cast":
+            out.append(tokens[i])
+            i += 1
+            continue
+        start = _operand_start(out)
+        operand = "".join(t for _, t in out[start:]).strip()
+        del out[start:]
+        end = _type_end(tokens, i + 1)
+        type_sql = "".join(t for _, t in tokens[i + 1 : end]).strip()
+        if not operand or not type_sql:
+            raise Skip(_UNTRANSLATABLE)
+        out.append(("operand", f"CAST({operand} AS {type_sql})"))
+        i = end
+    return "".join(t for _, t in out)
+
+
+def _significant_before(tokens: list[tuple[str, str]], end: int) -> int:
+    """The index of the last token before `end` that is not whitespace, or 
-1."""
+    k = end - 1
+    while k >= 0 and tokens[k][0] in ("space", "comment"):
+        k -= 1
+    return k
+
+
+def _operand_start(tokens: list[tuple[str, str]]) -> int:
+    """The index where the primary expression that ends `tokens` starts."""
+    k = _significant_before(tokens, len(tokens))
+    if k < 0:
+        raise Skip(_UNTRANSLATABLE)
+    kind, text = tokens[k]
+    if kind == "other" and text in (")", "]"):
+        k = _matching_open(tokens, k)
+        before = _significant_before(tokens, k)
+        prev = tokens[before] if before >= 0 else ("", "")
+        if text == ")" and prev[0] == "word" and prev[1].upper() not in 
_KEYWORDS:
+            # A call, f(...)
+            return _name_start(tokens, before)
+        if text == "]" and (prev[0] in ("word", "quoted", "string") or prev[1] 
in (")", "]")):
+            # A subscript, a[...], of the operand that ends before it
+            return _operand_start(tokens[: before + 1])
+        return k
+    if kind == "string":
+        before = _significant_before(tokens, k)
+        if before >= 0 and tokens[before][0] == "word" and 
tokens[before][1].upper() in _TYPED_LITERALS:
+            # A typed literal, such as DATE '2024-01-01'.
+            return before
+        return k
+    if kind == "number":
+        before = _significant_before(tokens, k)
+        if before >= 0 and tokens[before][1] == "-":
+            sign_of = _significant_before(tokens, before)
+            prev = tokens[sign_of] if sign_of >= 0 else None
+            if (
+                prev is None
+                or (prev[0] == "other" and prev[1] not in (")", "]"))
+                or (prev[0] == "word" and prev[1].upper() in _KEYWORDS)
+            ):
+                return before
+        return k
+    if kind in ("word", "quoted"):
+        return _name_start(tokens, k)
+    if kind == "operand":
+        # A cast that this function already rewrote, as in `x::int::string`.
+        return k
+    raise Skip(_UNTRANSLATABLE)
+
+
+def _name_start(tokens: list[tuple[str, str]], k: int) -> int:
+    """The start of a possibly qualified name, such as `t.a`, that ends at 
tokens[k]."""
+    while k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][0] in ("word", 
"quoted"):
+        k -= 2
+    return k

Review Comment:
   A field of a call or a subscript loses its base. `SELECT struct(1 AS 
x).x::string` becomes `SELECT struct(1 AS x).CAST(x AS string)`, where Spark 
4.2 casts `struct(1 AS x).x`. As with the keyword case, Spark 3.5 skips the 
record as a parse error. Could we continue into the operand before the `.`?
   
   ```suggestion
       while k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][0] in 
("word", "quoted"):
           k -= 2
       if k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][1] in (")", "]"):
           # A field of a call or subscript, such as `struct(1 AS x).x`
           return _operand_start(tokens[: k - 1])
       return k
   ```
   
   With this, `map(1, struct(2 AS y))[1].y::string` also becomes `CAST(map(1, 
struct(2 AS y))[1].y AS string)`.



##########
crates/sqllogictest/scripts/pyspark_slt_runner.py:
##########
@@ -788,6 +865,123 @@ def translate(sql: str, time_zone: str = "UTC") -> str:
     return "".join(out)
 
 
+# Keywords after which a `-` is a sign, not a subtraction, and a `(` does
+# not start the arguments of a call.
+_KEYWORDS = {
+    "ALL", "AND", "ANY", "AS", "BETWEEN", "BY", "CASE", "DISTINCT", "ELSE",
+    "EXISTS", "FROM", "HAVING", "IN", "IS", "JOIN", "LIKE", "LIMIT", "NOT",

Review Comment:
   `exists` and `any` are Spark functions, but `_operand_start` treats them as 
keywords, so a `::` after a call to either is rewritten wrong. `SELECT 
exists(arr, x -> x > 1)::string FROM t` becomes `SELECT existsCAST((arr, x -> x 
> 1) AS string) FROM t`, and `any(a > 1)::string` becomes `anyCAST((a > 1) AS 
string)`. Spark 4.2 parses both originals as a cast of the call. Spark 3.5 
raises `PARSE_SYNTAX_ERROR` on the rewritten SQL, and the runner [turns a parse 
error into a 
skip](https://github.com/apache/datafusion-spark/blob/87939bb0c14d4661fbe5f525ff47083df2b3eb68/crates/sqllogictest/scripts/pyspark_slt_runner.py#L1731-L1733).
 So such a record would silently go unchecked on 3.5. No current record hits 
this.
   
   Neither word can come before a sign or a parenthesized expression that `::` 
applies to, so they can leave the set:
   
   ```suggestion
       "ALL", "AND", "AS", "BETWEEN", "BY", "CASE", "DISTINCT", "ELSE",
       "FROM", "HAVING", "IN", "IS", "JOIN", "LIKE", "LIMIT", "NOT",
   ```
   
   Could you add both cases to `test_double_colon_casts_for_spark_3`?



##########
crates/sqllogictest/scripts/pyspark_slt_runner.py:
##########
@@ -149,12 +216,12 @@ def parse_slt(path: Path) -> list[Record]:
         if words[0] not in ("statement", "query"):
             # Directives such as `include` and `halt`, which these files do 
not use.
             conditions = []
-            xfail_reason = None
+            xfails = []
             i += 1
             continue
 
-        record = Record(path=path, line=i + 1, kind=words[0], 
xfail_reason=xfail_reason)
-        xfail_reason = None
+        record = Record(path=path, line=i + 1, kind=words[0], xfails=xfails)

Review Comment:
   Version conditions make it possible to write a marker that never applies on 
any Spark version CI runs. When the record passes, nothing reports it. For 
example, `# xfail spark>=4.0, <4.2: x` on a record that passes on both versions 
gives `pass` on 3.5.9 and on 4.2.0, not `xpass`. A marker shadowed by an 
earlier one that matches the same versions behaves the same way. That breaks 
the guarantee in [`run_record`'s 
docstring](https://github.com/apache/datafusion-spark/blob/87939bb0c14d4661fbe5f525ff47083df2b3eb68/crates/sqllogictest/scripts/pyspark_slt_runner.py#L1675-L1677)
 and the 
[README](https://github.com/apache/datafusion-spark/blob/87939bb0c14d4661fbe5f525ff47083df2b3eb68/crates/sqllogictest/scripts/README.md?plain=1#L18-L21)
 that a stale comment gets reported.
   
   Could we reject such markers at parse time? Then the error shows up before 
Spark starts, like a malformed condition does:
   
   ```suggestion
           record = Record(path=path, line=i + 1, kind=words[0], xfails=xfails)
           for x in xfails:
               if all(record.xfail(v) is not x for v in SUPPORTED_VERSIONS):
                   raise SystemExit(
                       f"error: {_display_path(path)}:{i + 1}: `{x.text}` never 
applies, because no "
                       "supported Spark version matches it before an earlier 
marker does"
                   )
   ```
   
   Every current `.slt` file parses with this check. 
`test_xfail_spark_versions` and the README example both use `spark>=4.0, <4.2`, 
which matches neither 3.5 nor 4.2, so they would need a different condition.



##########
crates/sqllogictest/scripts/pyspark_slt_runner.py:
##########
@@ -788,6 +865,123 @@ def translate(sql: str, time_zone: str = "UTC") -> str:
     return "".join(out)
 
 
+# Keywords after which a `-` is a sign, not a subtraction, and a `(` does
+# not start the arguments of a call.
+_KEYWORDS = {
+    "ALL", "AND", "ANY", "AS", "BETWEEN", "BY", "CASE", "DISTINCT", "ELSE",
+    "EXISTS", "FROM", "HAVING", "IN", "IS", "JOIN", "LIKE", "LIMIT", "NOT",
+    "ON", "OR", "RETURN", "SELECT", "THEN", "VALUES", "WHEN", "WHERE",
+}
+
+
+# The type names of typed literals, such as DATE '2024-01-01'.
+_TYPED_LITERALS = ("DATE", "TIME", "INTERVAL", "X", *_TIMESTAMP_TYPES)
+
+# Brackets that nest in a type name, such as `array<decimal(10, 2)>`.
+_TYPE_NESTING = {"(": 1, "<": 1, ")": -1, ">": -1}
+
+_UNTRANSLATABLE = "a `::` cast this runner cannot translate"
+
+
+def rewrite_double_colons(sql: str) -> str:
+    """Replace each `expr::type` with `CAST(expr AS type)`.
+
+    Spark 3.5 does not parse `::`. As in Spark 4, `::` applies to the primary
+    expression before it: a literal, a column, a call, a parenthesized
+    expression or a subscript. A sign belongs to a number literal, so
+    `-1::int` is `CAST(-1 AS int)`, but `-(1)::int` is `-CAST((1) AS int)`.
+    """
+    tokens = _tokenize(sql)
+    out: list[tuple[str, str]] = []
+    i = 0
+    while i < len(tokens):
+        if tokens[i][0] != "cast":
+            out.append(tokens[i])
+            i += 1
+            continue
+        start = _operand_start(out)
+        operand = "".join(t for _, t in out[start:]).strip()
+        del out[start:]
+        end = _type_end(tokens, i + 1)
+        type_sql = "".join(t for _, t in tokens[i + 1 : end]).strip()
+        if not operand or not type_sql:
+            raise Skip(_UNTRANSLATABLE)
+        out.append(("operand", f"CAST({operand} AS {type_sql})"))
+        i = end
+    return "".join(t for _, t in out)
+
+
+def _significant_before(tokens: list[tuple[str, str]], end: int) -> int:
+    """The index of the last token before `end` that is not whitespace, or 
-1."""
+    k = end - 1
+    while k >= 0 and tokens[k][0] in ("space", "comment"):
+        k -= 1
+    return k
+
+
+def _operand_start(tokens: list[tuple[str, str]]) -> int:
+    """The index where the primary expression that ends `tokens` starts."""
+    k = _significant_before(tokens, len(tokens))
+    if k < 0:
+        raise Skip(_UNTRANSLATABLE)
+    kind, text = tokens[k]
+    if kind == "other" and text in (")", "]"):
+        k = _matching_open(tokens, k)
+        before = _significant_before(tokens, k)
+        prev = tokens[before] if before >= 0 else ("", "")
+        if text == ")" and prev[0] == "word" and prev[1].upper() not in 
_KEYWORDS:
+            # A call, f(...)
+            return _name_start(tokens, before)
+        if text == "]" and (prev[0] in ("word", "quoted", "string") or prev[1] 
in (")", "]")):
+            # A subscript, a[...], of the operand that ends before it
+            return _operand_start(tokens[: before + 1])
+        return k
+    if kind == "string":
+        before = _significant_before(tokens, k)
+        if before >= 0 and tokens[before][0] == "word" and 
tokens[before][1].upper() in _TYPED_LITERALS:
+            # A typed literal, such as DATE '2024-01-01'.
+            return before
+        return k
+    if kind == "number":
+        before = _significant_before(tokens, k)
+        if before >= 0 and tokens[before][1] == "-":
+            sign_of = _significant_before(tokens, before)
+            prev = tokens[sign_of] if sign_of >= 0 else None
+            if (
+                prev is None
+                or (prev[0] == "other" and prev[1] not in (")", "]"))
+                or (prev[0] == "word" and prev[1].upper() in _KEYWORDS)
+            ):
+                return before
+        return k
+    if kind in ("word", "quoted"):
+        return _name_start(tokens, k)
+    if kind == "operand":
+        # A cast that this function already rewrote, as in `x::int::string`.
+        return k
+    raise Skip(_UNTRANSLATABLE)
+
+
+def _name_start(tokens: list[tuple[str, str]], k: int) -> int:
+    """The start of a possibly qualified name, such as `t.a`, that ends at 
tokens[k]."""
+    while k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][0] in ("word", 
"quoted"):
+        k -= 2
+    return k
+
+
+def _type_end(tokens: list[tuple[str, str]], start: int) -> int:
+    """The index after the type name that starts at tokens[start], such as
+    `int`, `decimal(10, 2)` or `array<map<string, int>>`."""
+    k = _next_significant(tokens, start)
+    if k >= len(tokens) or tokens[k][0] != "word":
+        raise Skip(_UNTRANSLATABLE)
+    k += 1
+    j = _next_significant(tokens, k)
+    if j < len(tokens) and tokens[j][1] in ("(", "<"):

Review Comment:
   `_type_end` reads a `<` after any type name as the start of type parameters, 
so a comparison after a cast breaks. `SELECT a::int < 5 AND b > 3 FROM t` 
becomes `SELECT CAST(a AS int < 5 AND b >) 3 FROM t`, and `SELECT a::int < 5 
FROM t` raises `Skip("unbalanced parentheses")`. Spark 4.2 parses the first as 
`(CAST(a AS INT) < 5) AND (b > 3)`. On 3.5 either case ends as a skip. 
Comparisons after a cast are common in `WHERE` clauses, so I'd expect future 
tests to hit this. Only `ARRAY`, `MAP`, and `STRUCT` take `<...>`:
   
   ```suggestion
       name = tokens[k][1].upper()
       k += 1
       j = _next_significant(tokens, k)
       # After any other type, `<` is a comparison, as in `a::int < 5`.
       if j < len(tokens) and (tokens[j][1] == "(" or (tokens[j][1] == "<" and 
name in ("ARRAY", "MAP", "STRUCT"))):
   ```
   
   Could you add `a::int < 5` and `a::int<b::int` to 
`test_double_colon_casts_for_spark_3`?



##########
crates/sqllogictest/test_files/math/greatest.slt:
##########
@@ -93,11 +93,13 @@ SELECT arrow_typeof(greatest(1::decimal(38,10), 
1::decimal(38,0)));
 Decimal128(38, 0)
 
 # The fractional digits that do not fit are rounded away
+# xfail spark<4.0: intended: Spark 3.5 picks DECIMAL(38, 10) as the wider type 
and returns NULL for the value that does not fit, and datafusion-spark follows 
Spark 4

Review Comment:
   This reason doesn't describe what happens for this record. No value here 
fails to fit `DECIMAL(38, 10)`, and Spark 3.5.9 returns `1.5000000000` where 
the file expects `2`. The reason on line 102 is right for its record: the 
`DECIMAL(38, 0)` value becomes NULL, and `greatest` returns `1.0000000000`.
   
   ```suggestion
   # xfail spark<4.0: intended: Spark 3.5 picks DECIMAL(38, 10) as the wider 
type and keeps the fraction, and datafusion-spark follows Spark 4
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to