Thanks for the feedback Otto. I have created a sub task for documenting the Job cleanup documentation: https://issues.apache.org/jira/browse/METRON-1737. I completely agree with you there, this needs to be documented. For the others you marked "Follow on" I will create follow on tasks in Jira.
I have a few questions about a couple others you commented on - Date range limits on queries Can you describe what you think is needed here? Each Metron user could have different volumes of pcap data spread out over different time periods. Are you saying we should limit the data range to something either constant or configurable? Are we sure all users would want this? Am I misinterpreting this requirement? - UI should manage a queue/history of jobs What should we document here? Reading that bullet point again, it's sort of vague and not very description. What I am referring to is a design that provides users a way to view and manage jobs in the UI. Currently jobs can only be run 1 at a time and progress is shown with a status bar, so it's somewhat interactive. - Documentation/blueprint for YARN configuration We are setup for YARN scheduling in that we offer a configuration setting to submit a Pcap query to a specified YARN queue (this part is documented). Any YARN setup or tuning would be out of scope since everyone's YARN settings will be different and potentially expand beyond the Metron use case. I think a Hadoop admin is likely to have this knowledge and to have already set up YARN queues. Do you disagree? On Mon, Aug 13, 2018 at 8:21 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > - Job cleanup/TTL > > Documented at least, or a helper script to help yourself if you are in a > situation > > > - Expose the Query filter (vs Fixed) in the UI > > Follow on > > > - Date range limits on queries > > I don’t see how this won’t be immediately required. I would do this for > minimum viable. > > > - Pcap query as a separate UI > > Follow on > > > - UI should manage a queue/history of jobs > > Follow on, but maybe we need documentation > > > - BPF filtering > > This is going to be a PITA, follow on > > > - Sharing PCA jobs with other users > > Follow on > > > - Provide a way in the UI to populate a pcap query from an alert/metaalert > > Follow on > > > - Documentation/blueprint for YARN configuration > > Should have > > >
