milenkovicm commented on code in PR #2265: URL: https://github.com/apache/datafusion-ballista/pull/2265#discussion_r3791290877
########## docs/source/user-guide/history-server.md: ########## @@ -0,0 +1,130 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# History Server + +The scheduler forgets a job shortly after it finishes. Completed jobs are cleaned +up after `finished_job_state_clean_up_interval_seconds`, and everything is gone +when the scheduler restarts, so by the time you want to look at a slow query it +is usually too late. + +The history server is Ballista's equivalent of the Spark History Server. When +event logging is enabled the scheduler writes a durable record of each job as it +runs, and the history server replays those records and serves the same `/api/*` +responses the live scheduler does. The existing TUI can point at it and browse +completed jobs with no scheduler running at all. + +## Enabling event logging + +Event logging is off by default. Start the scheduler with a directory to write +to: + +```shell +ballista-scheduler --event-log-dir /var/lib/ballista/history +``` + +The scheduler writes one file per job, `<job_id>.eventlog`, in +[JSON Lines](https://jsonlines.org/) format. Files are appended as the job runs +and closed when it reaches a terminal state. + +Writes happen on a background task, so the scheduler's event loop never waits on +disk. If the queue backs up, progress records are dropped rather than allowed to +stall scheduling. The terminal record is the exception: it waits for queue +capacity, because a job missing it is invisible to the history server. + +## Running the history server + +Point it at the same directory: + +```shell +ballista-history-server \ + --event-log-dir /var/lib/ballista/history \ + --bind-host 0.0.0.0 \ + --bind-port 50060 +``` + +It scans the directory at startup and indexes every completed log it finds, then +serves them over the same paths as the live scheduler: + +| Endpoint | Serves | +| ------------------------------ | ----------------------------------- | +| `GET /api/jobs` | every completed job, newest first | +| `GET /api/job/{job_id}` | one job's summary and plans | +| `GET /api/job/{job_id}/stages` | per-stage and per-task detail | +| `GET /api/job/{job_id}/config` | the session config the job ran with | +| `GET /api/job/{job_id}/dot` | the stage DAG in DOT format | + +Because the TUI talks to that same API, you can browse history with: + +```shell +BALLISTA__SCHEDULER__URL=http://localhost:50060 ballista-cli --tui +``` + +The TUI reads its URL from configuration rather than from the `--host` and +`--port` flags, which set the gRPC scheduler address used for running queries. +Pointing the TUI somewhere else means setting `BALLISTA__SCHEDULER__URL`, or +`scheduler.url` in the TUI's config file. Passing `--port 50060` alone leaves +the TUI on its default of `http://localhost:50050`, where it either finds your +live scheduler or reports that the scheduler is down. + +The directory is scanned once at startup, so restart the history server to pick Review Comment: Do you want to history sever to serve only static list of jobs discovered at startup @andygrove ? This was my main comment last time, I think it would make sense to discover new events automatically during execution rather than on startup. Anyway your call, please let me know what you think -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
