This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new d2278a90b4 Check sqllogictests for any dangling config settings
(#17914) (#20838)
d2278a90b4 is described below
commit d2278a90b4543939cefb0f3ffbea8b025fe922f0
Author: Sergey Zhukov <[email protected]>
AuthorDate: Fri Mar 13 15:50:06 2026 +0400
Check sqllogictests for any dangling config settings (#17914) (#20838)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #https://github.com/apache/datafusion/issues/17914.
## Rationale for this change
In a previous PR https://github.com/apache/datafusion/pull/20474, I
added a bash script that parsed the `SLT` files and checked whether any
DataFusion configuration options were modified without being reset.
While that approach worked, it relied on external scripting and
additional parsing logic. This PR introduces a simpler and more direct
solution implemented in Rust.
At the end of each `SLT` test file execution, the current configuration
is compared with the default configuration using a `Drop`
implementation. If any configuration values were modified and not
restored, a warning is printed.
This approach is easier to maintain and keeps the validation logic
within the Rust codebase rather than relying on an external bash script.
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
- Capture the default DataFusion configuration when the `SLT` runner is
initialized.
- Implement `Drop` for the DataFusion SLT engine.
- When an `SLT` file finishes executing, compare the current
configuration with the default configuration.
- If differences are detected, print a warning showing which
configuration options were modified.
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
This behavior is exercised by the existing `SLT` test suite. The
configuration check runs automatically when each `SLT` file completes
execution.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
No. This change only affects internal `SLT` test infrastructure and does
not modify any public APIs.
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---------
Co-authored-by: Martin Grigorov <[email protected]>
---
.../src/engines/datafusion_engine/runner.rs | 63 ++++++++++++++++++++--
1 file changed, 58 insertions(+), 5 deletions(-)
diff --git a/datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs
b/datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs
index df43a9a34c..c682d081f8 100644
--- a/datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs
+++ b/datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs
@@ -15,6 +15,7 @@
// specific language governing permissions and limitations
// under the License.
+use std::collections::HashMap;
use std::sync::Arc;
use std::{path::PathBuf, time::Duration};
@@ -38,15 +39,26 @@ pub struct DataFusion {
relative_path: PathBuf,
pb: ProgressBar,
currently_executing_sql_tracker: CurrentlyExecutingSqlTracker,
+ default_config: HashMap<String, Option<String>>,
}
impl DataFusion {
pub fn new(ctx: SessionContext, relative_path: PathBuf, pb: ProgressBar)
-> Self {
+ let default_config = ctx
+ .state()
+ .config()
+ .options()
+ .entries()
+ .iter()
+ .map(|e| (e.key.clone(), e.value.clone()))
+ .collect();
+
Self {
ctx,
relative_path,
pb,
currently_executing_sql_tracker:
CurrentlyExecutingSqlTracker::default(),
+ default_config,
}
}
@@ -54,13 +66,11 @@ impl DataFusion {
///
/// This is useful for logging and debugging purposes.
pub fn with_currently_executing_sql_tracker(
- self,
+ mut self,
currently_executing_sql_tracker: CurrentlyExecutingSqlTracker,
) -> Self {
- Self {
- currently_executing_sql_tracker,
- ..self
- }
+ self.currently_executing_sql_tracker = currently_executing_sql_tracker;
+ self
}
fn update_slow_count(&self) {
@@ -135,6 +145,49 @@ impl sqllogictest::AsyncDB for DataFusion {
async fn shutdown(&mut self) {}
}
+impl Drop for DataFusion {
+ fn drop(&mut self) {
+ let mut changed = false;
+
+ for e in self.ctx.state().config().options().entries() {
+ let default_entry = self.default_config.remove(&e.key);
+
+ if let Some(default_entry) = default_entry
+ && default_entry.as_ref() != e.value.as_ref()
+ {
+ if !changed {
+ changed = true;
+ self.pb.println(format!(
+ "SLT file {} left modified configuration",
+ self.relative_path.display()
+ ));
+ }
+
+ let default = default_entry.as_deref().unwrap_or("NULL");
+ let current = e.value.as_deref().unwrap_or("NULL");
+
+ self.pb
+ .println(format!(" {}: {} -> {}", e.key, default,
current));
+ }
+ }
+
+ // Any remaining entries were present initially but removed during
execution
+ for (key, value) in &self.default_config {
+ if !changed {
+ changed = true;
+ self.pb.println(format!(
+ "SLT file {} left modified configuration",
+ self.relative_path.display()
+ ));
+ }
+
+ let default = value.as_deref().unwrap_or("NULL");
+
+ self.pb.println(format!(" {key}: {default} -> NULL"));
+ }
+ }
+}
+
async fn run_query(
ctx: &SessionContext,
is_spark_path: bool,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
