+1 on the merge as well 16.08.2018, 05:46, "Casey Stella" <ceste...@gmail.com>: > I'm +1 on the merge. This is great work and congrats to those who > contributed to it! > > On Thu, Aug 16, 2018 at 8:27 AM Otto Fowler <ottobackwa...@gmail.com> wrote: > >> Looks good, thanks! >> >> On August 15, 2018 at 19:38:12, Ryan Merriman (merrim...@gmail.com) wrote: >> >> Otto, I believe the items you requested are in the feature branch now. Is >> there anything outstanding that we missed? The Jiras for the Pcap feature >> branch should be up to date: >> https://issues.apache.org/jira/browse/METRON-1554 >> >> On Mon, Aug 13, 2018 at 5:13 PM, Ryan Merriman <merrim...@gmail.com> >> wrote: >> >> > - Date range limits on queries >> > >> > I will add a warning in the Job cleanup PR. That seems like an >> > appropriate place for it (ie. make sure you don't cause health issues in >> > your cluster). >> > >> > - UI should manage a queue/history of jobs >> > >> > I can add some documentation around killing jobs manually with the YARN >> > CLI. However if they haven't set up a YARN queue, I'm not sure how you >> > would view only Pcap jobs. I'm also not sure how you would get the >> > application id for the job to kill because it's not displayed anywhere in >> > the UI. However, I believe we are wired for a job name but REST doesn't >> > set this. Maybe we could get a proper job name associated with pcap >> > queries and then this would be possible to document? >> > >> > - Documentation/blueprint for YARN configuration >> > >> > You make a good point. A YARN tuning guide for Metron does sound useful. >> > I will add a follow on Jira. >> > >> > On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler <ottobackwa...@gmail.com> >> > wrote: >> > >> >> >> >> - Date range limits on queries >> >> >> >> I took the point the wrong way apparently, sorry, I withdraw. I thought >> >> you meant allow specifying a limit on the query, not the system imposing >> a >> >> limit. >> >> This should be documented with a warning or something >> >> >> >> - UI should manage a queue/history of jobs >> >> >> >> I was thinking that if there where multiple users/jobs, there should >> >> be some thought or documentation + script on how to manage them. >> >> “To see all the jobs still running on your cluster, across users and ui >> >> instances do XXXXX” >> >> “If there is an issue with the jobs you can’t resolve in the UI for that >> >> user, or you are an admin and want to do something then XXXXX" >> >> >> >> - Documentation/blueprint for YARN configuration >> >> >> >> I agree with what you are saying. Although, we offer guidance on storm >> >> tuning, and that is conceptually the same isn’t it? That is why it comes >> >> to mind. >> >> Maybe this can be a follow on, in the tuning guide? >> >> >> >> On August 13, 2018 at 17:36:41, Ryan Merriman (merrim...@gmail.com) >> >> wrote: >> >> >> >> - Date range limits on queries >> >> >> >> Can you describe what you think is needed here? Each Metron user could >> >> have different volumes of pcap data spread out over different time >> >> periods. Are you saying we should limit the data range to something >> either >> >> >> >> constant or configurable? Are we sure all users would want this? Am I >> >> misinterpreting this requirement? >> >> >> >> - UI should manage a queue/history of jobs >> >> >> >> What should we document here? Reading that bullet point again, it's sort >> >> of vague and not very description. What I am referring to is a design >> that >> >> >> >> provides users a way to view and manage jobs in the UI. Currently jobs >> can >> >> >> >> only be run 1 at a time and progress is shown with a status bar, so it's >> >> somewhat interactive. >> >> >> >> - Documentation/blueprint for YARN configuration >> >> >> >> >> >
------------------- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
