kosiew commented on code in PR #316:
URL: https://github.com/apache/datafusion-spark/pull/316#discussion_r4612050821


##########
crates/sqllogictest/test_files/README.md:
##########
@@ -69,6 +69,27 @@ SELECT expm1(0::INT);
 SELECT expm1(a) FROM (VALUES (0::INT), (1::INT)) AS t(a);
 ```
 
+## Simplify and evaluate modes
+
+`cargo test -p datafusion-spark-sqllogictest` runs every file twice:
+
+- `simplify` uses the default optimizer, so a call whose arguments are all 
literals,
+  such as `SELECT expm1(0::INT)`, is constant-folded while planning. This 
calls the
+  function's `simplify` (if it has one) and evaluates it on a one-row batch.
+- `evaluate` removes the `simplify_expressions` optimizer rule, so every call 
runs

Review Comment:
   Evaluate mode does not bypass earlier SQL planning, which can simplify 
table-function arguments. Please narrow the documentation claims and clarify 
that runtime projections are needed to verify direct evaluation.



##########
crates/sqllogictest/tests/sqllogictests.rs:
##########
@@ -83,21 +242,44 @@ async fn main() -> ExitCode {
         {
             continue;
         }
-        handles.push((relative.clone(), tokio::spawn(run_file(path, 
relative))));
+        for &mode in &modes {
+            let task = tokio::spawn(run_file(path.clone(), relative.clone(), 
mode));
+            handles.push((relative.clone(), mode, task));
+        }
     }
 
     let total = handles.len();
     let mut failed = 0;
-    for (relative, handle) in handles {
-        match handle.await.expect("test task panicked") {
-            Ok(()) => println!("PASS {}", relative.display()),
-            Err(e) => {
+    for (relative, mode, handle) in handles {
+        let name = format!("[{}] {}", mode.label(), relative.display());
+        let known_failure = EVALUATE_KNOWN_FAILURES
+            .iter()
+            .find(|(file, _)| mode == Mode::Evaluate && Path::new(file) == 
relative)
+            .map(|(_, issue)| *issue);
+        // A panic in one file (for example a function indexing past the end 
of an
+        // array) is reported as that file failing, so the other results still 
print.
+        match (
+            handle.await.map_err(|e| e.to_string()).and_then(|r| r),
+            known_failure,
+        ) {
+            (Ok(()), None) => println!("PASS {name}"),
+            (Err(_), Some(issue)) => {

Review Comment:
   Whole-file XFAIL accepts unrelated failures and prevents later records from 
running. Please use issue-linked record-level skips or expectations. Verify 
that subsequent `unbase64` checks execute and mismatches fail the run.



##########
crates/sqllogictest/test_files/math/bround.slt:
##########
@@ -145,12 +148,18 @@ SELECT bround(column1, 1) FROM VALUES 
(0.25::decimal(3,2)), (0.35::decimal(3,2))
 NULL
 
 # A constant scale that is not a bare literal
+# The scale is a constant expression, which is a literal only after constant 
folding,
+# so the decimal result type is computed from it only when folding runs.
+skipif evaluate

Review Comment:
   Please link the decimal `bround` skips to a tracking issue if evaluate mode 
should support constant-expression scales. Otherwise, document this 
planner-only limitation.



##########
crates/sqllogictest/tests/sqllogictests.rs:
##########
@@ -83,21 +242,44 @@ async fn main() -> ExitCode {
         {
             continue;
         }
-        handles.push((relative.clone(), tokio::spawn(run_file(path, 
relative))));
+        for &mode in &modes {
+            let task = tokio::spawn(run_file(path.clone(), relative.clone(), 
mode));
+            handles.push((relative.clone(), mode, task));
+        }
     }
 
     let total = handles.len();
     let mut failed = 0;
-    for (relative, handle) in handles {
-        match handle.await.expect("test task panicked") {
-            Ok(()) => println!("PASS {}", relative.display()),
-            Err(e) => {
+    for (relative, mode, handle) in handles {
+        let name = format!("[{}] {}", mode.label(), relative.display());
+        let known_failure = EVALUATE_KNOWN_FAILURES
+            .iter()
+            .find(|(file, _)| mode == Mode::Evaluate && Path::new(file) == 
relative)
+            .map(|(_, issue)| *issue);
+        // A panic in one file (for example a function indexing past the end 
of an
+        // array) is reported as that file failing, so the other results still 
print.
+        match (
+            handle.await.map_err(|e| e.to_string()).and_then(|r| r),
+            known_failure,
+        ) {
+            (Ok(()), None) => println!("PASS {name}"),
+            (Err(_), Some(issue)) => {
+                println!("XFAIL {name} (known failure, see {issue})")
+            }
+            (Ok(()), Some(issue)) => {
+                failed += 1;
+                println!(
+                    "FAIL {name}\nexpected to fail ({issue}) but passed, \
+                     remove it from EVALUATE_KNOWN_FAILURES\n"
+                );
+            }
+            (Err(e), None) => {
                 failed += 1;
-                println!("FAIL {}\n{e}\n", relative.display());
+                println!("FAIL {name}\n{e}\n");
             }
         }
     }
-    println!("{} of {total} test files passed", total - failed);
+    println!("{} of {total} test runs passed", total - failed);

Review Comment:
   If XFAIL handling remains, please report expected failures separately from 
passes. The current summary counts XFAIL runs as passed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to