comphead commented on code in PR #7420: URL: https://github.com/apache/datafusion-comet/pull/7420#discussion_r4494552213
########## native/native-executors/src/bin/comet-scheduler.rs: ########## @@ -20,12 +20,35 @@ //! //! Configured through environment variables, with Ballista's defaults otherwise: //! - `COMET_SCHEDULER_BIND_HOST` (default `127.0.0.1`) +//! - `COMET_SCHEDULER_EXTERNAL_HOST`: the host executors reach this scheduler at, which they +//! report task status to (default `localhost`) //! - `COMET_SCHEDULER_PORT`: Ballista's gRPC port, for executors (default `50050`) //! - `COMET_SCHEDULER_FLIGHT_PORT`: the Flight port, for drivers (default `50060`) //! - `COMET_SCHEDULER_RESULT_RETENTION_SECONDS`: how long a query's results stay readable after //! the query ends, whether or not a driver has read them (default `300`). `0` keeps them. +//! - `COMET_GRPC_MAX_MESSAGE_SIZE`: the largest message, in bytes, of Ballista's gRPC service +//! between the scheduler and its executors, such as a task with many partitions of a scan +//! (default: Ballista's 16 MiB). Set it on the executors as well, for their side of the same +//! messages. The Flight service for drivers has no limit. Shuffle reads between executors keep +//! the limit of the query's `ballista.client.grpc_max_message_size` (16 MiB by default), which +//! this does not change. +//! - `COMET_SCHEDULER_EVENT_LOG_DIR`: a directory to write one event log per query to, as +//! Ballista's scheduler does (default: no event logs). The scheduler creates it if needed and +//! doesn't start if it can't write there. A log has the query's plan and a record for each +//! stage and task, with the task's times and the rows it wrote. Ballista sees a stage's Comet Review Comment: do we need to mention ballista ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
