alamb commented on code in PR #23066:
URL: https://github.com/apache/datafusion/pull/23066#discussion_r3501083246
##########
datafusion/core/src/execution/context/mod.rs:
##########
@@ -1224,6 +1237,14 @@ impl SessionContext {
let mut state = self.state.write();
+ if key == "max_spill_merge_fan_in" {
Review Comment:
ditto -- I think this should follow the same pattern as the other variabales
##########
datafusion/execution/src/runtime_env.rs:
##########
@@ -95,15 +95,28 @@ impl Debug for RuntimeEnv {
/// This helper function defines the structure and metadata for all runtime
configuration
/// entries to avoid duplication between `RuntimeEnv::config_entries()` and
/// `RuntimeEnvBuilder::entries()`.
-fn create_runtime_config_entries(
+struct RuntimeConfigValues {
memory_limit: Option<String>,
max_temp_directory_size: Option<String>,
+ max_spill_merge_fan_in: Option<String>,
temp_directory: Option<String>,
metadata_cache_limit: Option<String>,
list_files_cache_limit: Option<String>,
list_files_cache_ttl: Option<String>,
file_statistics_cache_limit: Option<String>,
-) -> Vec<ConfigEntry> {
+}
+
+fn create_runtime_config_entries(values: RuntimeConfigValues) ->
Vec<ConfigEntry> {
Review Comment:
How about making this a method on `RuntimeConfigValues` -- like
```rust
impl RuntimeConfigValues {
fn into_config_entries(values: RuntimeConfigValues) -> Vec<ConfigEntry> {
..
}
..
}
```
##########
datafusion/execution/src/disk_manager.rs:
##########
@@ -31,6 +31,7 @@ use tempfile::{Builder, NamedTempFile, TempDir};
use datafusion_common::human_readable_size;
pub const DEFAULT_MAX_TEMP_DIRECTORY_SIZE: u64 = 100 * 1024 * 1024 * 1024; //
100GB
+pub const DEFAULT_MAX_SPILL_MERGE_FAN_IN: usize = 0;
Review Comment:
I think limiting it to 100 files or something is probably reasonable. We can
consider doing this as a follow on PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
