We are nearing a fully functional Pcap query feature branch. I want to take a moment before we merge to review the original discussion threads and make sure the community is happy with the state of this feature branch before we accept it into master.
The original discuss threads are located at: - Back end architecture thread: https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E - UI requirements: https://lists.apache.org/thread.html/e62e361971092e49446e2012550319f06c8c31944224bcd6326718d9@%3Cdev.metron.apache.org%3E The JIRA epic can be found here: https://issues.apache.org/jira/browse/METRON-1554. The state of each task should be accurate. We expect all tasks to be finished within the next couple days (except for https://issues.apache.org/jira/browse/METRON-1561). I reviewed the original discuss threads and overall I think we accomplished a lot. We were able to create abstractions around managing and submitting jobs. We were able to configure the YARN queue for Pcap queries so we are set up for multi-tenancy in the future. We have basic guards in place to keep users from overwhelming the cluster. We were able to expose results in the UI and as a binary download. We have basic authorization in place that can be expanded later. We expect the outstanding Jira mentioned above to be converted to a follow on Jira. There are several other ideas that were brought up in the discuss threads but not done in the feature branch. They do not currently have Jiras : - Job cleanup/TTL - Expose the Query filter (vs Fixed) in the UI - Date range limits on queries - Pcap query as a separate UI - UI should manage a queue/history of jobs - BPF filtering - Sharing PCA jobs with other users - Provide a way in the UI to populate a pcap query from an alert/metaalert - Documentation/blueprint for YARN configuration I'm sure I missed some so please chime in with any you want to add. Which of these do we still feel should be done? Are there any features or changes you feel need to be done before this feature branch is merged? I will create the appropriate Jiras as needed. Ryan
