samuel7james commented on code in PR #2787:
URL:
https://github.com/apache/datafusion-ballista/pull/2787#discussion_r4493977633
##########
ballista/history/src/writer.rs:
##########
@@ -142,6 +142,18 @@ impl EventLogWriter {
}
}
+/// Check that event logs can be written under `log_dir`: create it if it is
+/// missing, then create and remove a file in it.
+///
+/// [`EventLogWriter`] only logs a warning when it cannot write, so a caller
+/// that needs the logs should check the directory before starting the writer.
+pub async fn check_log_dir(log_dir: &Path) -> std::io::Result<()> {
+ tokio::fs::create_dir_all(log_dir).await?;
+ let probe = log_dir.join(format!(".write-check-{}", std::process::id()));
Review Comment:
Good catch, thanks. The probe is now named by pid, time and a counter and
created with `create_new`, so a check only ever removes the file it made. Added
`check_log_dir_runs_concurrently_on_one_dir` (16 checks at once in one process,
so all with the same pid); it fails with the old name. In b81a160.
##########
ballista/scheduler/src/scheduler_server/mod.rs:
##########
@@ -310,6 +310,18 @@ impl<T: 'static + AsLogicalPlan, U: 'static +
AsExecutionPlan> SchedulerServer<T
/// Initializes the scheduler and starts background tasks.
pub async fn init(&mut self) -> Result<()> {
+ // The event-log writer only warns when it cannot write, so fail here
+ // instead of running without the logs that were asked for.
+ #[cfg(feature = "rest-api")]
+ if let Some(dir) = &self.config.event_log_dir {
+ ballista_history::writer::check_log_dir(std::path::Path::new(dir))
Review Comment:
Added a note under API changes in `docs/source/upgrading/55.0.0.md` in
b81a160.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
