andygrove commented on code in PR #316:
URL: https://github.com/apache/datafusion-spark/pull/316#discussion_r4615381061
##########
crates/sqllogictest/tests/sqllogictests.rs:
##########
@@ -83,21 +242,44 @@ async fn main() -> ExitCode {
{
continue;
}
- handles.push((relative.clone(), tokio::spawn(run_file(path,
relative))));
+ for &mode in &modes {
+ let task = tokio::spawn(run_file(path.clone(), relative.clone(),
mode));
+ handles.push((relative.clone(), mode, task));
+ }
}
let total = handles.len();
let mut failed = 0;
- for (relative, handle) in handles {
- match handle.await.expect("test task panicked") {
- Ok(()) => println!("PASS {}", relative.display()),
- Err(e) => {
+ for (relative, mode, handle) in handles {
+ let name = format!("[{}] {}", mode.label(), relative.display());
+ let known_failure = EVALUATE_KNOWN_FAILURES
+ .iter()
+ .find(|(file, _)| mode == Mode::Evaluate && Path::new(file) ==
relative)
+ .map(|(_, issue)| *issue);
+ // A panic in one file (for example a function indexing past the end
of an
+ // array) is reported as that file failing, so the other results still
print.
+ match (
+ handle.await.map_err(|e| e.to_string()).and_then(|r| r),
+ known_failure,
+ ) {
+ (Ok(()), None) => println!("PASS {name}"),
+ (Err(_), Some(issue)) => {
Review Comment:
Done in 59ed9f9. EVALUATE_KNOWN_FAILURES is gone. Each record that calls a
function with no invoke_with_args yet is marked `skipif evaluate`, and a note
at the top of the file links the issue. The rest of each file now runs in
evaluate mode. I checked base64.slt: the unbase64 records run, and changing one
expected value fails the run.
##########
crates/sqllogictest/test_files/README.md:
##########
@@ -69,6 +69,27 @@ SELECT expm1(0::INT);
SELECT expm1(a) FROM (VALUES (0::INT), (1::INT)) AS t(a);
```
+## Simplify and evaluate modes
+
+`cargo test -p datafusion-spark-sqllogictest` runs every file twice:
+
+- `simplify` uses the default optimizer, so a call whose arguments are all
literals,
+ such as `SELECT expm1(0::INT)`, is constant-folded while planning. This
calls the
+ function's `simplify` (if it has one) and evaluates it on a one-row batch.
+- `evaluate` removes the `simplify_expressions` optimizer rule, so every call
runs
Review Comment:
Narrowed in 59ed9f9. The README now says that evaluate mode only changes the
optimizer, that SQL planning can still evaluate some expressions first, such as
table function arguments, and that a call over rows is the reliable way to test
invoke_with_args.
##########
crates/sqllogictest/tests/sqllogictests.rs:
##########
@@ -83,21 +242,44 @@ async fn main() -> ExitCode {
{
continue;
}
- handles.push((relative.clone(), tokio::spawn(run_file(path,
relative))));
+ for &mode in &modes {
+ let task = tokio::spawn(run_file(path.clone(), relative.clone(),
mode));
+ handles.push((relative.clone(), mode, task));
+ }
}
let total = handles.len();
let mut failed = 0;
- for (relative, handle) in handles {
- match handle.await.expect("test task panicked") {
- Ok(()) => println!("PASS {}", relative.display()),
- Err(e) => {
+ for (relative, mode, handle) in handles {
+ let name = format!("[{}] {}", mode.label(), relative.display());
+ let known_failure = EVALUATE_KNOWN_FAILURES
+ .iter()
+ .find(|(file, _)| mode == Mode::Evaluate && Path::new(file) ==
relative)
+ .map(|(_, issue)| *issue);
+ // A panic in one file (for example a function indexing past the end
of an
+ // array) is reported as that file failing, so the other results still
print.
+ match (
+ handle.await.map_err(|e| e.to_string()).and_then(|r| r),
+ known_failure,
+ ) {
+ (Ok(()), None) => println!("PASS {name}"),
+ (Err(_), Some(issue)) => {
+ println!("XFAIL {name} (known failure, see {issue})")
+ }
+ (Ok(()), Some(issue)) => {
+ failed += 1;
+ println!(
+ "FAIL {name}\nexpected to fail ({issue}) but passed, \
+ remove it from EVALUATE_KNOWN_FAILURES\n"
+ );
+ }
+ (Err(e), None) => {
failed += 1;
- println!("FAIL {}\n{e}\n", relative.display());
+ println!("FAIL {name}\n{e}\n");
}
}
}
- println!("{} of {total} test files passed", total - failed);
+ println!("{} of {total} test runs passed", total - failed);
Review Comment:
With the XFAIL handling removed in 59ed9f9, every run is a plain pass or
fail.
##########
crates/sqllogictest/test_files/math/bround.slt:
##########
@@ -145,12 +148,18 @@ SELECT bround(column1, 1) FROM VALUES
(0.25::decimal(3,2)), (0.35::decimal(3,2))
NULL
# A constant scale that is not a bare literal
+# The scale is a constant expression, which is a literal only after constant
folding,
+# so the decimal result type is computed from it only when folding runs.
+skipif evaluate
Review Comment:
Documented in 59ed9f9 rather than tracked. The planner computes the decimal
result type from literal arguments only, so a scale like `1::bigint` is not
seen without constant folding. invoke_with_args is fine. A caller that builds
the physical expression, such as Comet, passes the result type itself. The
comment on each skipped record now says this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]