alamb commented on code in PR #7424:
URL: https://github.com/apache/arrow-datafusion/pull/7424#discussion_r1306404983
##########
datafusion-cli/src/main.rs:
##########
@@ -59,6 +60,14 @@ struct Args {
)]
command: Vec<String>,
+ #[clap(
+ short = 'm',
+ long,
+ help = "The memory pool limitation (e.g. '10g'), default to 0",
Review Comment:
```suggestion
help = "The memory pool limitation (e.g. '10g'), default to None (no
limit)",
```
##########
datafusion-cli/src/main.rs:
##########
@@ -109,7 +125,32 @@ pub async fn main() -> Result<()> {
session_config = session_config.with_batch_size(batch_size);
};
- let runtime_env = create_runtime_env()?;
+ let rn_config = RuntimeConfig::new();
+ let rn_config =
+ // set memory pool size
+ if let Some(memory_limit) = args.memory_limit {
+ let memory_limit = memory_limit[..memory_limit.len() - 1]
Review Comment:
I think this simply passes `10` (bytes) along to the pool size when `-m 10G`
is passed:
```shell
$ RUST_LOG=debug /Users/alamb/Software/target-df/debug/datafusion-cli -m 10g
DataFusion CLI v30.0.0
[2023-08-26T11:06:26Z DEBUG datafusion_execution::memory_pool::pool] Created
new GreedyMemoryPool(pool_size=10)
```
I think the pool size should work like this. What do you think?
* `-m 1000` -- set pool size to 1000
* `-m 500m` -- set pool size to `500 * 1024*1024` (500 MB)
* `-m 10G` -- set pool size to `10 * 1024*1024*1024` (10 GB)
##########
datafusion-cli/src/main.rs:
##########
@@ -87,6 +96,13 @@ struct Args {
help = "Reduce printing other than the results and work quietly"
)]
quiet: bool,
+
+ #[clap(
+ long,
+ help = "Specify the memory pool type 'greedy' or 'fair', default to
'greedy'",
+ validator(is_valid_memory_pool_type)
+ )]
+ mem_pool_type: Option<String>,
Review Comment:
Another, perhaps more ideomatic way to do this, is to define an enum like
```suggestion
mem_pool_type: Option<PoolType>,
```
```rust
enum PoolType {
Greedy,
Fair
}
impl FromStr for PoolType {
...
}
```
And then I think clap will do the right thing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
