sarutak opened a new pull request, #54575: URL: https://github.com/apache/spark/pull/54575
### What changes were proposed in this pull request? This PR proposes to add multiple log directories support to SHS, allowing it to monitor event logs from multiple directories simultaneously. This PR extends `spark.history.fs.logDirectory` to accepts a comma-separated list of directories (e.g., `hdfs:///logs/prod,s3a://bucket/logs/staging`), each potentially on a different filesystem. Also, a new optional config `spark.history.fs.logDirectory.names` is added which allows users to assign display names to directories by position (e.g., `Production,Staging`). Empty entries fall back to the full path. Duplicate display names are rejected at startup. Regarding UI changes, A "Log Source" column is added to the History UI table showing the display name (or full path) for each application, with a tooltip showing the full path. <img width="1699" height="681" alt="all-log-dirs" src="https://github.com/user-attachments/assets/fc67aca3-563c-4cec-8b9c-8e44e81caa04"; /> Users can filter applications by their log directory using `Filter by Log Source` dropdown. <img width="1714" height="390" alt="filter-by-log-dir" src="https://github.com/user-attachments/assets/b590cedc-9c33-46ca-9314-fa50eb31777f"; /> The `Event log directory` section in the History UI collapses into a `<details>/<summary>` element when multiple directories are configured. <img width="431" height="99" alt="unexpand" src="https://github.com/user-attachments/assets/613ef9a0-348a-4e6a-b742-8b33b36142af"; /> <img width="423" height="187" alt="expand" src="https://github.com/user-attachments/assets/d3bbef26-13bd-408d-85e4-c0f9af2dfc5b"; /> ### Why are the changes needed? Some organization run multiple clusters and have corresponding log directory for each cluster. So if SHS supports multiple log directories, it can be used as a single end point to view event logs, which helps such organizations. ### Does this PR introduce _any_ user-facing change? Yes but will not affect existing users. ### How was this patch tested? Manually confirmed WebUI as screenshots above and added new tests. ### Was this patch authored or co-authored using generative AI tooling? Kiro CLI / Opus 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
