alamb commented on issue #7573: URL: https://github.com/apache/arrow-datafusion/issues/7573#issuecomment-1724504600
Hi @hengfeiyang -- this sound like a great idea to me Rather than hard coding the concurrency, perhaps you can add a config parameter, perhaps like `ExecutionOptions::meta_fetch_concurrency`, following the model of https://docs.rs/datafusion/latest/datafusion/config/struct.ExecutionOptions.html#structfield.planning_concurrency The rationale for a configuration setting is that the optimal value will likely depend on the network configuration of the system, so there is no good constant that would work in all cases. Perhaps it can default to 10 or 32 ? cc @Ted-Jiang who I think was working on some other settings to cache statistics for multiple queries in the same session. See https://github.com/apache/arrow-datafusion/pull/7570 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
