I'm +1 on the merge. This is great work and congrats to those who contributed to it!
On Thu, Aug 16, 2018 at 8:27 AM Otto Fowler <ottobackwa...@gmail.com> wrote: > Looks good, thanks! > > > On August 15, 2018 at 19:38:12, Ryan Merriman (merrim...@gmail.com) wrote: > > Otto, I believe the items you requested are in the feature branch now. Is > there anything outstanding that we missed? The Jiras for the Pcap feature > branch should be up to date: > https://issues.apache.org/jira/browse/METRON-1554 > > On Mon, Aug 13, 2018 at 5:13 PM, Ryan Merriman <merrim...@gmail.com> > wrote: > > > - Date range limits on queries > > > > I will add a warning in the Job cleanup PR. That seems like an > > appropriate place for it (ie. make sure you don't cause health issues in > > your cluster). > > > > - UI should manage a queue/history of jobs > > > > I can add some documentation around killing jobs manually with the YARN > > CLI. However if they haven't set up a YARN queue, I'm not sure how you > > would view only Pcap jobs. I'm also not sure how you would get the > > application id for the job to kill because it's not displayed anywhere in > > the UI. However, I believe we are wired for a job name but REST doesn't > > set this. Maybe we could get a proper job name associated with pcap > > queries and then this would be possible to document? > > > > - Documentation/blueprint for YARN configuration > > > > You make a good point. A YARN tuning guide for Metron does sound useful. > > I will add a follow on Jira. > > > > On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler <ottobackwa...@gmail.com> > > wrote: > > > >> > >> - Date range limits on queries > >> > >> I took the point the wrong way apparently, sorry, I withdraw. I thought > >> you meant allow specifying a limit on the query, not the system imposing > a > >> limit. > >> This should be documented with a warning or something > >> > >> - UI should manage a queue/history of jobs > >> > >> I was thinking that if there where multiple users/jobs, there should > >> be some thought or documentation + script on how to manage them. > >> “To see all the jobs still running on your cluster, across users and ui > >> instances do XXXXX” > >> “If there is an issue with the jobs you can’t resolve in the UI for that > >> user, or you are an admin and want to do something then XXXXX" > >> > >> - Documentation/blueprint for YARN configuration > >> > >> I agree with what you are saying. Although, we offer guidance on storm > >> tuning, and that is conceptually the same isn’t it? That is why it comes > >> to mind. > >> Maybe this can be a follow on, in the tuning guide? > >> > >> On August 13, 2018 at 17:36:41, Ryan Merriman (merrim...@gmail.com) > >> wrote: > >> > >> - Date range limits on queries > >> > >> Can you describe what you think is needed here? Each Metron user could > >> have different volumes of pcap data spread out over different time > >> periods. Are you saying we should limit the data range to something > either > >> > >> constant or configurable? Are we sure all users would want this? Am I > >> misinterpreting this requirement? > >> > >> - UI should manage a queue/history of jobs > >> > >> What should we document here? Reading that bullet point again, it's sort > >> of vague and not very description. What I am referring to is a design > that > >> > >> provides users a way to view and manage jobs in the UI. Currently jobs > can > >> > >> only be run 1 at a time and progress is shown with a status bar, so it's > >> somewhat interactive. > >> > >> - Documentation/blueprint for YARN configuration > >> > >> > > >
