andygrove opened a new issue, #2766: URL: https://github.com/apache/datafusion-ballista/issues/2766
**Describe the bug** When the scheduler can't write to `event_log_dir`, it starts and runs queries normally, writes no event logs, and only logs warnings. `EventLogWriter::new` (`ballista/history/src/writer.rs`) creates the directory in `run`, on the task it spawns. When that fails, `run` logs a warning and returns, which closes the writer's channel, so every later event is dropped with another warning. A directory that exists but isn't writable, such as a volume mounted with another owner, gets past `create_dir_all`, and then each job's log fails to open with `event-log writer: cannot open ...`. **To Reproduce** Set `event_log_dir` (`--event-log-dir`) to a path inside a regular file and run a query. The query succeeds, and the scheduler logs (path shortened): ``` WARN ballista_history::writer] event-log writer: cannot create /var/folders/.../comet-scratch-80543/logs: Not a directory (os error 20) WARN ballista_history::writer] event-log writer: dropping event for 038D3AT922000 (channel full or closed) WARN ballista_history::writer] event-log writer: dropping event for 038D3AT922000 (channel full or closed) WARN ballista_history::writer] event-log writer: failed to enqueue terminal event for 038D3AT922000 (channel closed) WARN ballista_history::writer] event-log writer: failed to enqueue finish for 038D3AT922000 (channel closed) ``` **Expected behavior** The scheduler fails to start, with an error naming the directory: an operator who sets `event_log_dir` expects logs. For example, `SchedulerServer` could create the directory and check that it can create a file there before it starts the writer, or `EventLogWriter::new` could do that and return a `Result`. **Additional context** Seen with 55.0.0, through Apache DataFusion Comet's scheduler binary, which embeds Ballista's scheduler; `main` at 70a074322 has the same code. Comet's binary now checks the directory itself before it starts the scheduler (apache/datafusion-comet#7420), and could drop that check once Ballista does it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
