mingmwang commented on a change in pull request #1987:
URL: https://github.com/apache/arrow-datafusion/pull/1987#discussion_r825543118
##########
File path: datafusion/src/execution/context.rs
##########
@@ -1220,6 +1233,74 @@ impl FunctionRegistry for ExecutionContextState {
}
}
+/// Task Context Properties
+pub enum TaskProperties {
Review comment:
I will cover this in the following PR. For batch_size, it will be moved
out from RuntimeEnv to SessionConfig.
And SessionConfig just includes the configuration entires, no RuntimeEnv
anymore.
And in the TaskContext, there will be a method to get the current
SessionConfig.
````
impl TaskProperties {
/// Return the SessionConfig associated with the Task
pub fn session_config(&self) -> SessionConfig {
let task_settings = &self.task_settings;
match task_settings {
TaskProperties::KVPairs(props) => {
let session_config = SessionConfig::new();
session_config
.with_batch_size(props.get(BATCH_SIZE).unwrap().parse().unwrap())
.with_target_partitions(
props.get(TARGET_PARTITIONS).unwrap().parse().unwrap(),
)
.with_repartition_joins(
props.get(REPARTITION_JOINS).unwrap().parse().unwrap(),
)
.with_repartition_aggregations(
props
.get(REPARTITION_AGGREGATIONS)
.unwrap()
.parse()
.unwrap(),
)
.with_repartition_windows(
props.get(REPARTITION_WINDOWS).unwrap().parse().unwrap(),
)
.with_parquet_pruning(
props.get(PARQUET_PRUNING).unwrap().parse().unwrap(),
)
}
TaskProperties::SessionConfig(session_config) =>
session_config.clone(),
}
}
}
````
In this PR, to retrieve the batch size, currently, we can use below code,
but this will be changed soon in the following PR.
````
let batch_size = context.runtime.batch_size();
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
