andygrove opened a new pull request, #7420: URL: https://github.com/apache/datafusion-comet/pull/7420
## Which issue does this PR close? Part of #7288. ## Rationale for this change `comet-scheduler` and `comet-executor` only expose the settings that a local, single-machine cluster needs. Running them on a real cluster, as the TPC-H runs in #7289 do, needs a few more of Ballista's settings: an external host for the scheduler, larger gRPC message limits for tasks with many scan partitions, a fixed executor memory pool, and the scheduler's per-query event logs for looking at stage timings. This splits those settings out of #7289. Each one sets a field that the Ballista 55.0.0 release already has, so this does not depend on any unmerged Ballista change. ## What changes are included in this PR? New environment variables, each documented in the binary's module docs: - `COMET_SCHEDULER_EXTERNAL_HOST`: the host executors reach the scheduler at. - `COMET_GRPC_MAX_MESSAGE_SIZE`: the largest gRPC message the scheduler and executor send or accept (Ballista's default is 16 MiB). - `COMET_EXECUTOR_MEMORY_POOL_SIZE`: the bytes of memory an executor's tasks share. - `COMET_SCHEDULER_EVENT_LOG_DIR`: a directory for one Ballista event log per query. The event-log writer is behind `ballista-scheduler`'s `rest-api` feature, so that feature is enabled. In 55.0.0 that feature also needs the `tracing` feature to compile without the default features, so that is enabled too. ## How are these changes tested? Each variable is passed straight through to the matching Ballista config field. The existing `native-executors` cluster tests still pass, and clippy is clean with `--all-targets -D warnings`. The settings were used for the TPC-H runs in #7289. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
