What about the possibility for the user to specify on which folder/file the job should run? in other to reduce the amount of data to process?
2018-05-04 14:19 GMT+01:00 Ryan Merriman <merrim...@gmail.com>: > Continuing a discussion that started in a discuss thread about exposing > Pcap query capabilities in the back end. How should we expose this feature > to users? Should it be integrated into the Alerts UI or be separate > standalone UI? > > To summarize the general points made in the other thread: > > - Adding this capability to the Alerts UI will make it more of a > composite app. Is that really what we want since we have separate UIs > for > Alerts and management? > - Would it be better to bring it in on it's own so it can be released > with qualifiers and tested with the right expectations without affecting > the Alerts UI? > - There are some use cases that begin with an infosec analyst doing a > search on alerts > followed by them going to query pcap data corresponding to the > threats they're investigating. Would having these features in the same > UI streamline this process? > > There was also mention of some features we should consider: > > - Pcap queries should be made asynchronous via the UI > - Take care that a user doesn't hit refresh or POST multiple times and > kick > off 50 mapreduce jobs > - Options for managing the YARN queue that is used > - Provide a "cancel" option that kills the MR job, or tell the user to > go to the CLI to kill their job > - Managing data if multiple users run queries > - Strategy for cleaning up jobs and implementing a TTL (I think this one > will be tricky and definitely needs discussion) > - Date range or other query limits > > A couple other features I would add: > > - Ability to paginate through results > - Ability to download results through the UI > - Realtime status of a running job in the UI > > Let me know if I missed any points or did not correctly capture them > here. What > other points do we need to consider? What other features should be > required? Nice to have? >
