andygrove opened a new issue, #2267: URL: https://github.com/apache/datafusion-ballista/issues/2267
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** The event log added in #2264 is hard-wired to the local filesystem. `EventLogWriter` (`ballista/history/src/writer.rs`) opens a `tokio::fs::File` per job and appends JSONL to `<event_log_dir>/<job_id>.eventlog`, the reader (`ballista/history/src/reader.rs`) takes a `&Path` and opens it with `std::fs::File::open`, and `HistoryStore::load` (`ballista/scheduler/src/history/mod.rs`) discovers jobs with `std::fs::read_dir`. That is fine for a single scheduler on a box with a persistent disk, but it is the wrong shape for how Ballista is usually deployed: - In Kubernetes the scheduler pod's filesystem is ephemeral, so job history dies with the pod unless someone attaches and manages a PVC. - The history server has to run somewhere that can see the same directory the scheduler wrote to, which in practice means a shared volume rather than a separately deployed read-only service. - With multiple schedulers there is no single directory holding the whole cluster's history. Writing logs to an object store (S3, GCS, Azure, or anything else `object_store` supports) removes all three constraints: the scheduler ships logs to a durable remote location, and the history server reads them from anywhere. **Describe the solution you'd like** Put the `object_store` crate behind the event log's read and write paths, so `--event-log-dir` accepts a URL (`s3://bucket/prefix`, `file:///var/log/ballista`, plain paths staying local for compatibility): - `EventLogWriter::new` takes an `Arc<dyn ObjectStore>` plus a base `Path` instead of a `PathBuf`. - The reader takes a store and a path rather than `&Path`, and `HistoryStore::load` uses `ObjectStore::list` in place of `read_dir`. - The history server binary resolves its own store from the same URL, so it can be deployed on its own with no shared volume. **Describe alternatives you've considered** - *Keep it local and let operators sync the directory out of band* (sidecar, `aws s3 sync`, PVC + backup). Works, but pushes the durability problem onto every deployment and leaves the history server needing volume access. - *Trait-abstract the storage layer ourselves* rather than depending on `object_store`. More code for less coverage; `object_store` is already a workspace dependency, already used by `ballista-scheduler`, and already how the rest of the stack talks to remote storage. **Additional context** The main design question is append semantics. Object stores have no append: the current writer holds one open handle per job for the life of the process and appends a line per event. Options, roughly in increasing order of effort: 1. **Buffer per job, write once on `JobEnd`.** Simplest, and it fits how the log is consumed today — the reader only looks at the terminal `JobEnd` record and the timeline records are unread. The cost is that a scheduler crash loses the whole job's log, and memory is held for the duration of long-running jobs. 2. **Multipart upload per job**, flushing parts as events accumulate. Preserves incremental durability, but needs careful handling of aborted uploads for jobs whose scheduler dies mid-flight. 3. **Write locally, upload on completion.** Keeps crash-durability on local disk and gets the object into remote storage once the job finishes; needs a cleanup/retry story for uploads that fail. Worth deciding this before a future UI starts consuming the incremental timeline records, since option 1 rules out live progress from a remote store. Follow-on from https://github.com/apache/datafusion-ballista/pull/2264#issuecomment-5233620333. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
