Hi, I have couple questions to ask regarding the Spark history server:
1. Is there a way for a cluster to selectively clean old files? For example, if we want to keep some logs from 3 days ago but also cleaned some logs from 2 days ago, is there a filter or config to do that? 2. We have over 1000 log files each day. If we want to keep those jobs for a week (7000 jobs in total), this would potentially make the load time longer. Is there any suggestion on doing this? 3. We plan to have 2 paths to long-term history server and short-term history server. We can move some log files from short-term to long-term server if we need to do some investigation on that, would this be a good idea. Do you have any input on this? Thank you in advance!
