Hi Em, > On Mar 22, 2017, at 3:41 PM, E.M. Dragowsky <[email protected]> wrote: > > Thanks, Will -- We will take a closer look at elasticsearch. > > My naive thought had been to perform the analysis on the submission scripts > fairly close in time to allocation of resources, and then only keep the > derived data.
Something like that would probably work. Note that you can run “scontrol show job <jobid> -dd” to see the script so long as the job has not completed yet. > > What is your perspective on how long to keep the index file output from > elasticsearch? > We’d like to have a separate index for each week. At the moment the plugin keeps all jobs in a single index. That is probably fine if you don’t have a large number of jobs at your site, but we often have tens of thousands of jobs complete at our site in a single day. Performance in ES is generally better if you can run queries across multiple indices.
