Re: [DISCUSS] Pcap UI user requirements

[zeo...@gmail.com]

Regarding the prioritization, that is what I was thinking as well, I just
wasn't as prescriptive with my suggestion.

I did look for a java implementation and failed to find one (the closest I
found was the Apache-licened bcc project ).
Perhaps someone else's google-fu is better than mine though.

Not sure that BPF has the ability to depend on state (I feel it would be
very unlikely), but if it does I've never used it.  My 10 minute read of
the original paper (admittedly > 20 years old) and some other supplementary
info seems to support my prior thoughts.

Jon

On Wed, May 9, 2018 at 2:59 PM Casey Stella  wrote:

> A couple of thoughts on cluster overuse:
> * Definitely can't pause/resume MR jobs, unfortunately
> * The traditional approach to managing overuse of cluster resources and
> prioritization in Yarn is via the scheduler.  I'd suggest rather than
> building this ourselves, we allow users to be associated with yarn queues
> so their jobs are submitted to the correct queue and don't clobber the
> cluster.
>
> Beyond that, I like the BPF idea as a filter.  Any idea if there's a java
> BPF implementation?  Also, keep in mind that our query mechanism is a map
> and a reduce job, so any filtering system which depends on state (e.g.
> previous packets by time) is going to trigger another architecture.
>
> On Mon, May 7, 2018 at 4:05 PM zeo...@gmail.com  wrote:
>
> > From my perspective PCAP is primarily used as a follow-on to an alert or
> > meta-alert - people very rarely use PCAP for initial hunting.  I know
> this
> > has been brought up by Otto, Mike, and Ryan across the two related
> threads
> > and I think it's all spot on.  Going from an alert or meta-alert to
> pulling
> > PCAP would by far the primary use case for this in every SOC I've ever
> > worked in (not necessarily a representative sample).
> >
> > I also have some additional thoughts on the feature side after doing some
> > brainstorming and talking to two of the SOCs I work most with:
> >  - Limit the size of the PCAP, not just the date range, and maybe even
> have
> > a configurable cluster-wide admin max for PCAP retrieval, set to
> 0/infinite
> > by default.
> >  - Set priority of PCAP queries.  Perhaps there's an automated
> > pcap retrieval 'just in case', which should have a lower priority than an
> > interactive request via the UI.
> >  - Ability to pause/resume (not just cancel) jobs.
> >  - Configurable cluster-wide admin max # of current PCAP queries, set to
> > 0/infinite by default.
> >  - Ability to pull PCAP live off the wire and stream it into a file.
> >  - Ability to filter PCAP by providing a BPF filter to apply in
> server-side
> > post-processing (less efficient, but very versatile).
> >  - Request what PCAP data exists in the cluster (answering "how far back
> > can I go?")
> >  - This is obvious and is probably assumed, but queries based on any set
> of
> > the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.
> >
> > Jon
> >
> > On Fri, May 4, 2018 at 9:44 AM Otto Fowler 
> > wrote:
> >
> > > That is the ‘views’ part.
> > >
> > > We can have options on the data output, if you have output full data,
> > then
> > > we can have different views and interactions for inspection and level
> of
> > > detail.
> > >
> > >
> > >
> > > On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com)
> > wrote:
> > >
> > > It can be like a report but also to investigate some case where the
> user
> > > want to see the whole packet (all the bits and bytes). Like in
> wireshark,
> > > something interactive no?
> > >
> > > 2018-05-04 14:33 GMT+01:00 Otto Fowler :
> > >
> > > > The PCAP Query seems more like PCAP Report to me. You are generating
> a
> > > > report based on parameters.
> > > > That report is something that takes some time and external process to
> > > > generate… ie you have to wait for it.
> > > >
> > > > I can almost imagine a flow where you:
> > > >
> > > > * Are in the AlertUI
> > > > * Ask to generate a PCAP report based on some selected
> > alerts/meta-alert,
> > > > possibly picking from on or more report ‘templates’
> > > > that have query options etc
> > > > * The report request is ‘queued’, that is dispatched to be be
> > > > executed/generated
> > > > * You as a user have a ‘queue’ of your report results, and when the
> > > report
> > > > is done it is queued there
> > > > * We ‘monitor’ the report/queue press through the yarn rest ( report
> > > > info/meta has the yarn details )
> > > > * You can select the report from your queue and view it either in a
> new
> > > UI
> > > > or custom component
> > > > * You can then apply a different ‘view’ to the report or work with
> the
> > > > report data
> > > > * You can print / save etc
> > > > * You can associate the report with the alerts ( again in the report
> > info
> > > )
> > > > with…. a ‘case’ or ‘ticket’ or investigation something or other
> > > >
> > > >
> > > > We can introduce extensibility into the 

Re: [DISCUSS] Pcap UI user requirements

[Casey Stella]

A couple of thoughts on cluster overuse:
* Definitely can't pause/resume MR jobs, unfortunately
* The traditional approach to managing overuse of cluster resources and
prioritization in Yarn is via the scheduler.  I'd suggest rather than
building this ourselves, we allow users to be associated with yarn queues
so their jobs are submitted to the correct queue and don't clobber the
cluster.

Beyond that, I like the BPF idea as a filter.  Any idea if there's a java
BPF implementation?  Also, keep in mind that our query mechanism is a map
and a reduce job, so any filtering system which depends on state (e.g.
previous packets by time) is going to trigger another architecture.

On Mon, May 7, 2018 at 4:05 PM zeo...@gmail.com  wrote:

> From my perspective PCAP is primarily used as a follow-on to an alert or
> meta-alert - people very rarely use PCAP for initial hunting.  I know this
> has been brought up by Otto, Mike, and Ryan across the two related threads
> and I think it's all spot on.  Going from an alert or meta-alert to pulling
> PCAP would by far the primary use case for this in every SOC I've ever
> worked in (not necessarily a representative sample).
>
> I also have some additional thoughts on the feature side after doing some
> brainstorming and talking to two of the SOCs I work most with:
>  - Limit the size of the PCAP, not just the date range, and maybe even have
> a configurable cluster-wide admin max for PCAP retrieval, set to 0/infinite
> by default.
>  - Set priority of PCAP queries.  Perhaps there's an automated
> pcap retrieval 'just in case', which should have a lower priority than an
> interactive request via the UI.
>  - Ability to pause/resume (not just cancel) jobs.
>  - Configurable cluster-wide admin max # of current PCAP queries, set to
> 0/infinite by default.
>  - Ability to pull PCAP live off the wire and stream it into a file.
>  - Ability to filter PCAP by providing a BPF filter to apply in server-side
> post-processing (less efficient, but very versatile).
>  - Request what PCAP data exists in the cluster (answering "how far back
> can I go?")
>  - This is obvious and is probably assumed, but queries based on any set of
> the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.
>
> Jon
>
> On Fri, May 4, 2018 at 9:44 AM Otto Fowler 
> wrote:
>
> > That is the ‘views’ part.
> >
> > We can have options on the data output, if you have output full data,
> then
> > we can have different views and interactions for inspection and level of
> > detail.
> >
> >
> >
> > On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com)
> wrote:
> >
> > It can be like a report but also to investigate some case where the user
> > want to see the whole packet (all the bits and bytes). Like in wireshark,
> > something interactive no?
> >
> > 2018-05-04 14:33 GMT+01:00 Otto Fowler :
> >
> > > The PCAP Query seems more like PCAP Report to me. You are generating a
> > > report based on parameters.
> > > That report is something that takes some time and external process to
> > > generate… ie you have to wait for it.
> > >
> > > I can almost imagine a flow where you:
> > >
> > > * Are in the AlertUI
> > > * Ask to generate a PCAP report based on some selected
> alerts/meta-alert,
> > > possibly picking from on or more report ‘templates’
> > > that have query options etc
> > > * The report request is ‘queued’, that is dispatched to be be
> > > executed/generated
> > > * You as a user have a ‘queue’ of your report results, and when the
> > report
> > > is done it is queued there
> > > * We ‘monitor’ the report/queue press through the yarn rest ( report
> > > info/meta has the yarn details )
> > > * You can select the report from your queue and view it either in a new
> > UI
> > > or custom component
> > > * You can then apply a different ‘view’ to the report or work with the
> > > report data
> > > * You can print / save etc
> > > * You can associate the report with the alerts ( again in the report
> info
> > )
> > > with…. a ‘case’ or ‘ticket’ or investigation something or other
> > >
> > >
> > > We can introduce extensibility into the report templates, report views
> (
> > > thinks that work with the json data of the report )
> > >
> > > Something like that.
> > >
> > >
> > > On May 4, 2018 at 09:19:15, Ryan Merriman (merrim...@gmail.com) wrote:
> > >
> > > Continuing a discussion that started in a discuss thread about exposing
> > > Pcap query capabilities in the back end. How should we expose this
> > feature
> > > to users? Should it be integrated into the Alerts UI or be separate
> > > standalone UI?
> > >
> > > To summarize the general points made in the other thread:
> > >
> > > - Adding this capability to the Alerts UI will make it more of a
> > > composite app. Is that really what we want since we have separate UIs
> for
> > > Alerts and management?
> > > - Would it be better to bring it in on it's own so it can be released
> > > with qualifiers and tested with the right expecta

Re: [DISCUSS] Pcap UI user requirements

[zeo...@gmail.com]

I had a feeling it may be that way.  Unless anyone else knows of a better
approach, it's probably most reasonable to push that into a follow-on JIRA
and not over-complicate the current activities.

Jon

On Wed, May 9, 2018 at 2:33 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> We are limited by Yarn and MapReduce applications in the case of
> pause/resume - I could be wrong, but I don't think that's something that's
> supported unless you're talking about multiple MR jobs strung together.
>
>
> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#application
>
> I don't see anything suggesting "SUSPENDED" or "PAUSED" as we have
> available in workflow engines like Oozie.
>
> "The valid application state can be one of the following:  ALL, NEW,
> NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED"
>
> Same goes for MR job commands:
>
> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html#job
>
> Mike
>
> On Mon, May 7, 2018 at 2:04 PM, zeo...@gmail.com  wrote:
>
> > From my perspective PCAP is primarily used as a follow-on to an alert or
> > meta-alert - people very rarely use PCAP for initial hunting.  I know
> this
> > has been brought up by Otto, Mike, and Ryan across the two related
> threads
> > and I think it's all spot on.  Going from an alert or meta-alert to
> pulling
> > PCAP would by far the primary use case for this in every SOC I've ever
> > worked in (not necessarily a representative sample).
> >
> > I also have some additional thoughts on the feature side after doing some
> > brainstorming and talking to two of the SOCs I work most with:
> >  - Limit the size of the PCAP, not just the date range, and maybe even
> have
> > a configurable cluster-wide admin max for PCAP retrieval, set to
> 0/infinite
> > by default.
> >  - Set priority of PCAP queries.  Perhaps there's an automated
> > pcap retrieval 'just in case', which should have a lower priority than an
> > interactive request via the UI.
> >  - Ability to pause/resume (not just cancel) jobs.
> >  - Configurable cluster-wide admin max # of current PCAP queries, set to
> > 0/infinite by default.
> >  - Ability to pull PCAP live off the wire and stream it into a file.
> >  - Ability to filter PCAP by providing a BPF filter to apply in
> server-side
> > post-processing (less efficient, but very versatile).
> >  - Request what PCAP data exists in the cluster (answering "how far back
> > can I go?")
> >  - This is obvious and is probably assumed, but queries based on any set
> of
> > the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.
> >
> > Jon
> >
> > On Fri, May 4, 2018 at 9:44 AM Otto Fowler 
> > wrote:
> >
> > > That is the ‘views’ part.
> > >
> > > We can have options on the data output, if you have output full data,
> > then
> > > we can have different views and interactions for inspection and level
> of
> > > detail.
> > >
> > >
> > >
> > > On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com)
> > wrote:
> > >
> > > It can be like a report but also to investigate some case where the
> user
> > > want to see the whole packet (all the bits and bytes). Like in
> wireshark,
> > > something interactive no?
> > >
> > > 2018-05-04 14:33 GMT+01:00 Otto Fowler :
> > >
> > > > The PCAP Query seems more like PCAP Report to me. You are generating
> a
> > > > report based on parameters.
> > > > That report is something that takes some time and external process to
> > > > generate… ie you have to wait for it.
> > > >
> > > > I can almost imagine a flow where you:
> > > >
> > > > * Are in the AlertUI
> > > > * Ask to generate a PCAP report based on some selected
> > alerts/meta-alert,
> > > > possibly picking from on or more report ‘templates’
> > > > that have query options etc
> > > > * The report request is ‘queued’, that is dispatched to be be
> > > > executed/generated
> > > > * You as a user have a ‘queue’ of your report results, and when the
> > > report
> > > > is done it is queued there
> > > > * We ‘monitor’ the report/queue press through the yarn rest ( report
> > > > info/meta has the yarn details )
> > > > * You can select the report from your queue and view it either in a
> new
> > > UI
> > > > or custom component
> > > > * You can then apply a different ‘view’ to the report or work with
> the
> > > > report data
> > > > * You can print / save etc
> > > > * You can associate the report with the alerts ( again in the report
> > info
> > > )
> > > > with…. a ‘case’ or ‘ticket’ or investigation something or other
> > > >
> > > >
> > > > We can introduce extensibility into the report templates, report
> views
> > (
> > > > thinks that work with the json data of the report )
> > > >
> > > > Something like that.
> > > >
> > > >
> > > > On May 4, 2018 at 09:19:15, Ryan Merriman (merrim...@gmail.com)
> wrote:
> > > >
> > > > Continuing a discussion that started in a discuss thread about
> exposing

Re: [DISCUSS] Pcap UI user requirements

[Michael Miklavcic]

We are limited by Yarn and MapReduce applications in the case of
pause/resume - I could be wrong, but I don't think that's something that's
supported unless you're talking about multiple MR jobs strung together.

https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#application

I don't see anything suggesting "SUSPENDED" or "PAUSED" as we have
available in workflow engines like Oozie.

"The valid application state can be one of the following:  ALL, NEW,
NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED"

Same goes for MR job commands:
https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html#job

Mike

On Mon, May 7, 2018 at 2:04 PM, zeo...@gmail.com  wrote:

> From my perspective PCAP is primarily used as a follow-on to an alert or
> meta-alert - people very rarely use PCAP for initial hunting.  I know this
> has been brought up by Otto, Mike, and Ryan across the two related threads
> and I think it's all spot on.  Going from an alert or meta-alert to pulling
> PCAP would by far the primary use case for this in every SOC I've ever
> worked in (not necessarily a representative sample).
>
> I also have some additional thoughts on the feature side after doing some
> brainstorming and talking to two of the SOCs I work most with:
>  - Limit the size of the PCAP, not just the date range, and maybe even have
> a configurable cluster-wide admin max for PCAP retrieval, set to 0/infinite
> by default.
>  - Set priority of PCAP queries.  Perhaps there's an automated
> pcap retrieval 'just in case', which should have a lower priority than an
> interactive request via the UI.
>  - Ability to pause/resume (not just cancel) jobs.
>  - Configurable cluster-wide admin max # of current PCAP queries, set to
> 0/infinite by default.
>  - Ability to pull PCAP live off the wire and stream it into a file.
>  - Ability to filter PCAP by providing a BPF filter to apply in server-side
> post-processing (less efficient, but very versatile).
>  - Request what PCAP data exists in the cluster (answering "how far back
> can I go?")
>  - This is obvious and is probably assumed, but queries based on any set of
> the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.
>
> Jon
>
> On Fri, May 4, 2018 at 9:44 AM Otto Fowler 
> wrote:
>
> > That is the ‘views’ part.
> >
> > We can have options on the data output, if you have output full data,
> then
> > we can have different views and interactions for inspection and level of
> > detail.
> >
> >
> >
> > On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com)
> wrote:
> >
> > It can be like a report but also to investigate some case where the user
> > want to see the whole packet (all the bits and bytes). Like in wireshark,
> > something interactive no?
> >
> > 2018-05-04 14:33 GMT+01:00 Otto Fowler :
> >
> > > The PCAP Query seems more like PCAP Report to me. You are generating a
> > > report based on parameters.
> > > That report is something that takes some time and external process to
> > > generate… ie you have to wait for it.
> > >
> > > I can almost imagine a flow where you:
> > >
> > > * Are in the AlertUI
> > > * Ask to generate a PCAP report based on some selected
> alerts/meta-alert,
> > > possibly picking from on or more report ‘templates’
> > > that have query options etc
> > > * The report request is ‘queued’, that is dispatched to be be
> > > executed/generated
> > > * You as a user have a ‘queue’ of your report results, and when the
> > report
> > > is done it is queued there
> > > * We ‘monitor’ the report/queue press through the yarn rest ( report
> > > info/meta has the yarn details )
> > > * You can select the report from your queue and view it either in a new
> > UI
> > > or custom component
> > > * You can then apply a different ‘view’ to the report or work with the
> > > report data
> > > * You can print / save etc
> > > * You can associate the report with the alerts ( again in the report
> info
> > )
> > > with…. a ‘case’ or ‘ticket’ or investigation something or other
> > >
> > >
> > > We can introduce extensibility into the report templates, report views
> (
> > > thinks that work with the json data of the report )
> > >
> > > Something like that.
> > >
> > >
> > > On May 4, 2018 at 09:19:15, Ryan Merriman (merrim...@gmail.com) wrote:
> > >
> > > Continuing a discussion that started in a discuss thread about exposing
> > > Pcap query capabilities in the back end. How should we expose this
> > feature
> > > to users? Should it be integrated into the Alerts UI or be separate
> > > standalone UI?
> > >
> > > To summarize the general points made in the other thread:
> > >
> > > - Adding this capability to the Alerts UI will make it more of a
> > > composite app. Is that really what we want since we have separate UIs
> for
> > > Alerts and management?
> > > - Would it be better to bring it in on it's own so it can be released
> > > with qualifiers a

Re: [DISCUSS] Pcap UI user requirements

[zeo...@gmail.com]

>From my perspective PCAP is primarily used as a follow-on to an alert or
meta-alert - people very rarely use PCAP for initial hunting.  I know this
has been brought up by Otto, Mike, and Ryan across the two related threads
and I think it's all spot on.  Going from an alert or meta-alert to pulling
PCAP would by far the primary use case for this in every SOC I've ever
worked in (not necessarily a representative sample).

I also have some additional thoughts on the feature side after doing some
brainstorming and talking to two of the SOCs I work most with:
 - Limit the size of the PCAP, not just the date range, and maybe even have
a configurable cluster-wide admin max for PCAP retrieval, set to 0/infinite
by default.
 - Set priority of PCAP queries.  Perhaps there's an automated
pcap retrieval 'just in case', which should have a lower priority than an
interactive request via the UI.
 - Ability to pause/resume (not just cancel) jobs.
 - Configurable cluster-wide admin max # of current PCAP queries, set to
0/infinite by default.
 - Ability to pull PCAP live off the wire and stream it into a file.
 - Ability to filter PCAP by providing a BPF filter to apply in server-side
post-processing (less efficient, but very versatile).
 - Request what PCAP data exists in the cluster (answering "how far back
can I go?")
 - This is obvious and is probably assumed, but queries based on any set of
the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.

Jon

On Fri, May 4, 2018 at 9:44 AM Otto Fowler  wrote:

> That is the ‘views’ part.
>
> We can have options on the data output, if you have output full data, then
> we can have different views and interactions for inspection and level of
> detail.
>
>
>
> On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com) wrote:
>
> It can be like a report but also to investigate some case where the user
> want to see the whole packet (all the bits and bytes). Like in wireshark,
> something interactive no?
>
> 2018-05-04 14:33 GMT+01:00 Otto Fowler :
>
> > The PCAP Query seems more like PCAP Report to me. You are generating a
> > report based on parameters.
> > That report is something that takes some time and external process to
> > generate… ie you have to wait for it.
> >
> > I can almost imagine a flow where you:
> >
> > * Are in the AlertUI
> > * Ask to generate a PCAP report based on some selected alerts/meta-alert,
> > possibly picking from on or more report ‘templates’
> > that have query options etc
> > * The report request is ‘queued’, that is dispatched to be be
> > executed/generated
> > * You as a user have a ‘queue’ of your report results, and when the
> report
> > is done it is queued there
> > * We ‘monitor’ the report/queue press through the yarn rest ( report
> > info/meta has the yarn details )
> > * You can select the report from your queue and view it either in a new
> UI
> > or custom component
> > * You can then apply a different ‘view’ to the report or work with the
> > report data
> > * You can print / save etc
> > * You can associate the report with the alerts ( again in the report info
> )
> > with…. a ‘case’ or ‘ticket’ or investigation something or other
> >
> >
> > We can introduce extensibility into the report templates, report views (
> > thinks that work with the json data of the report )
> >
> > Something like that.
> >
> >
> > On May 4, 2018 at 09:19:15, Ryan Merriman (merrim...@gmail.com) wrote:
> >
> > Continuing a discussion that started in a discuss thread about exposing
> > Pcap query capabilities in the back end. How should we expose this
> feature
> > to users? Should it be integrated into the Alerts UI or be separate
> > standalone UI?
> >
> > To summarize the general points made in the other thread:
> >
> > - Adding this capability to the Alerts UI will make it more of a
> > composite app. Is that really what we want since we have separate UIs for
> > Alerts and management?
> > - Would it be better to bring it in on it's own so it can be released
> > with qualifiers and tested with the right expectations without affecting
> > the Alerts UI?
> > - There are some use cases that begin with an infosec analyst doing a
> > search on alerts
> > followed by them going to query pcap data corresponding to the
> > threats they're investigating. Would having these features in the same
> > UI streamline this process?
> >
> > There was also mention of some features we should consider:
> >
> > - Pcap queries should be made asynchronous via the UI
> > - Take care that a user doesn't hit refresh or POST multiple times and
> kick
> > off 50 mapreduce jobs
> > - Options for managing the YARN queue that is used
> > - Provide a "cancel" option that kills the MR job, or tell the user to
> > go to the CLI to kill their job
> > - Managing data if multiple users run queries
> > - Strategy for cleaning up jobs and implementing a TTL (I think this one
> > will be tricky and definitely needs discussion)
> > - Date range or other query limits
> >
>

Re: [DISCUSS] Pcap UI user requirements

[Otto Fowler]

That is the ‘views’ part.

We can have options on the data output, if you have output full data, then
we can have different views and interactions for inspection and level of
detail.



On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com) wrote:

It can be like a report but also to investigate some case where the user
want to see the whole packet (all the bits and bytes). Like in wireshark,
something interactive no?

2018-05-04 14:33 GMT+01:00 Otto Fowler :

> The PCAP Query seems more like PCAP Report to me. You are generating a
> report based on parameters.
> That report is something that takes some time and external process to
> generate… ie you have to wait for it.
>
> I can almost imagine a flow where you:
>
> * Are in the AlertUI
> * Ask to generate a PCAP report based on some selected alerts/meta-alert,
> possibly picking from on or more report ‘templates’
> that have query options etc
> * The report request is ‘queued’, that is dispatched to be be
> executed/generated
> * You as a user have a ‘queue’ of your report results, and when the
report
> is done it is queued there
> * We ‘monitor’ the report/queue press through the yarn rest ( report
> info/meta has the yarn details )
> * You can select the report from your queue and view it either in a new
UI
> or custom component
> * You can then apply a different ‘view’ to the report or work with the
> report data
> * You can print / save etc
> * You can associate the report with the alerts ( again in the report info
)
> with…. a ‘case’ or ‘ticket’ or investigation something or other
>
>
> We can introduce extensibility into the report templates, report views (
> thinks that work with the json data of the report )
>
> Something like that.
>
>
> On May 4, 2018 at 09:19:15, Ryan Merriman (merrim...@gmail.com) wrote:
>
> Continuing a discussion that started in a discuss thread about exposing
> Pcap query capabilities in the back end. How should we expose this
feature
> to users? Should it be integrated into the Alerts UI or be separate
> standalone UI?
>
> To summarize the general points made in the other thread:
>
> - Adding this capability to the Alerts UI will make it more of a
> composite app. Is that really what we want since we have separate UIs for
> Alerts and management?
> - Would it be better to bring it in on it's own so it can be released
> with qualifiers and tested with the right expectations without affecting
> the Alerts UI?
> - There are some use cases that begin with an infosec analyst doing a
> search on alerts
> followed by them going to query pcap data corresponding to the
> threats they're investigating. Would having these features in the same
> UI streamline this process?
>
> There was also mention of some features we should consider:
>
> - Pcap queries should be made asynchronous via the UI
> - Take care that a user doesn't hit refresh or POST multiple times and
kick
> off 50 mapreduce jobs
> - Options for managing the YARN queue that is used
> - Provide a "cancel" option that kills the MR job, or tell the user to
> go to the CLI to kill their job
> - Managing data if multiple users run queries
> - Strategy for cleaning up jobs and implementing a TTL (I think this one
> will be tricky and definitely needs discussion)
> - Date range or other query limits
>
> A couple other features I would add:
>
> - Ability to paginate through results
> - Ability to download results through the UI
> - Realtime status of a running job in the UI
>
> Let me know if I missed any points or did not correctly capture them
> here. What
> other points do we need to consider? What other features should be
> required? Nice to have?
>

Re: [DISCUSS] Pcap UI user requirements

[Michel Sumbul]

It can be like a report but also to investigate some case where the user
want to see the whole packet (all the bits and bytes). Like in wireshark,
something interactive no?

2018-05-04 14:33 GMT+01:00 Otto Fowler :

> The PCAP Query seems more like PCAP Report to me.  You are generating a
> report based on parameters.
> That report is something that takes some time and external process to
> generate… ie you have to wait for it.
>
> I can almost imagine a flow where you:
>
> * Are in the AlertUI
> * Ask to generate a PCAP report based on some selected alerts/meta-alert,
> possibly picking from on or more report ‘templates’
> that have query options etc
> * The report request is ‘queued’, that is dispatched to be be
> executed/generated
> * You as a user have a ‘queue’ of your report results, and when the report
> is done it is queued there
> * We ‘monitor’ the report/queue press through the yarn rest ( report
> info/meta has the yarn details )
> * You can select the report from your queue and view it either in a new UI
> or custom component
> * You can then apply a different ‘view’ to the report or work with the
> report data
> * You can print / save etc
> * You can associate the report with the alerts ( again in the report info )
> with…. a ‘case’ or ‘ticket’ or investigation something or other
>
>
> We can introduce extensibility into the report templates, report views (
> thinks that work with the json data of the report )
>
> Something like that.
>
>
> On May 4, 2018 at 09:19:15, Ryan Merriman (merrim...@gmail.com) wrote:
>
> Continuing a discussion that started in a discuss thread about exposing
> Pcap query capabilities in the back end. How should we expose this feature
> to users? Should it be integrated into the Alerts UI or be separate
> standalone UI?
>
> To summarize the general points made in the other thread:
>
> - Adding this capability to the Alerts UI will make it more of a
> composite app. Is that really what we want since we have separate UIs for
> Alerts and management?
> - Would it be better to bring it in on it's own so it can be released
> with qualifiers and tested with the right expectations without affecting
> the Alerts UI?
> - There are some use cases that begin with an infosec analyst doing a
> search on alerts
> followed by them going to query pcap data corresponding to the
> threats they're investigating. Would having these features in the same
> UI streamline this process?
>
> There was also mention of some features we should consider:
>
> - Pcap queries should be made asynchronous via the UI
> - Take care that a user doesn't hit refresh or POST multiple times and kick
> off 50 mapreduce jobs
> - Options for managing the YARN queue that is used
> - Provide a "cancel" option that kills the MR job, or tell the user to
> go to the CLI to kill their job
> - Managing data if multiple users run queries
> - Strategy for cleaning up jobs and implementing a TTL (I think this one
> will be tricky and definitely needs discussion)
> - Date range or other query limits
>
> A couple other features I would add:
>
> - Ability to paginate through results
> - Ability to download results through the UI
> - Realtime status of a running job in the UI
>
> Let me know if I missed any points or did not correctly capture them
> here. What
> other points do we need to consider? What other features should be
> required? Nice to have?
>

Re: [DISCUSS] Pcap UI user requirements

[Otto Fowler]

The PCAP Query seems more like PCAP Report to me.  You are generating a
report based on parameters.
That report is something that takes some time and external process to
generate… ie you have to wait for it.

I can almost imagine a flow where you:

* Are in the AlertUI
* Ask to generate a PCAP report based on some selected alerts/meta-alert,
possibly picking from on or more report ‘templates’
that have query options etc
* The report request is ‘queued’, that is dispatched to be be
executed/generated
* You as a user have a ‘queue’ of your report results, and when the report
is done it is queued there
* We ‘monitor’ the report/queue press through the yarn rest ( report
info/meta has the yarn details )
* You can select the report from your queue and view it either in a new UI
or custom component
* You can then apply a different ‘view’ to the report or work with the
report data
* You can print / save etc
* You can associate the report with the alerts ( again in the report info )
with…. a ‘case’ or ‘ticket’ or investigation something or other


We can introduce extensibility into the report templates, report views (
thinks that work with the json data of the report )

Something like that.


On May 4, 2018 at 09:19:15, Ryan Merriman (merrim...@gmail.com) wrote:

Continuing a discussion that started in a discuss thread about exposing
Pcap query capabilities in the back end. How should we expose this feature
to users? Should it be integrated into the Alerts UI or be separate
standalone UI?

To summarize the general points made in the other thread:

- Adding this capability to the Alerts UI will make it more of a
composite app. Is that really what we want since we have separate UIs for
Alerts and management?
- Would it be better to bring it in on it's own so it can be released
with qualifiers and tested with the right expectations without affecting
the Alerts UI?
- There are some use cases that begin with an infosec analyst doing a
search on alerts
followed by them going to query pcap data corresponding to the
threats they're investigating. Would having these features in the same
UI streamline this process?

There was also mention of some features we should consider:

- Pcap queries should be made asynchronous via the UI
- Take care that a user doesn't hit refresh or POST multiple times and kick
off 50 mapreduce jobs
- Options for managing the YARN queue that is used
- Provide a "cancel" option that kills the MR job, or tell the user to
go to the CLI to kill their job
- Managing data if multiple users run queries
- Strategy for cleaning up jobs and implementing a TTL (I think this one
will be tricky and definitely needs discussion)
- Date range or other query limits

A couple other features I would add:

- Ability to paginate through results
- Ability to download results through the UI
- Realtime status of a running job in the UI

Let me know if I missed any points or did not correctly capture them
here. What
other points do we need to consider? What other features should be
required? Nice to have?

Re: [DISCUSS] Pcap UI user requirements

[Michel Sumbul]

What about the possibility for the user to specify on which folder/file the
job should run? in other to reduce the amount of data to process?

2018-05-04 14:19 GMT+01:00 Ryan Merriman :

> Continuing a discussion that started in a discuss thread about exposing
> Pcap query capabilities in the back end.  How should we expose this feature
> to users?  Should it be integrated into the Alerts UI or be separate
> standalone UI?
>
> To summarize the general points made in the other thread:
>
>- Adding this capability to the Alerts UI will make it more of a
>composite app.  Is that really what we want since we have separate UIs
> for
>Alerts and management?
>- Would it be better to bring it in on it's own so it can be released
>with qualifiers and tested with the right expectations without affecting
>the Alerts UI?
>- There are some use cases that begin with an infosec analyst doing a
>search on alerts
>followed by them going to query pcap data corresponding to the
>threats they're investigating.  Would having these features in the same
>UI streamline this process?
>
> There was also mention of some features we should consider:
>
>- Pcap queries should be made asynchronous via the UI
>- Take care that a user doesn't hit refresh or POST multiple times and
> kick
>off 50 mapreduce jobs
>- Options for managing the YARN queue that is used
>- Provide a "cancel" option that kills the MR job, or tell the user to
>go to the CLI to kill their job
>- Managing data if multiple users run queries
>- Strategy for cleaning up jobs and implementing a TTL (I think this one
>will be tricky and definitely needs discussion)
>- Date range or other query limits
>
> A couple other features I would add:
>
>- Ability to paginate through results
>- Ability to download results through the UI
>- Realtime status of a running job in the UI
>
> Let me know if I missed any points or did not correctly capture them
> here.  What
> other points do we need to consider?  What other features should be
> required?  Nice to have?
>

[DISCUSS] Pcap UI user requirements

[Ryan Merriman]

Continuing a discussion that started in a discuss thread about exposing
Pcap query capabilities in the back end.  How should we expose this feature
to users?  Should it be integrated into the Alerts UI or be separate
standalone UI?

To summarize the general points made in the other thread:

   - Adding this capability to the Alerts UI will make it more of a
   composite app.  Is that really what we want since we have separate UIs for
   Alerts and management?
   - Would it be better to bring it in on it's own so it can be released
   with qualifiers and tested with the right expectations without affecting
   the Alerts UI?
   - There are some use cases that begin with an infosec analyst doing a
   search on alerts
   followed by them going to query pcap data corresponding to the
   threats they're investigating.  Would having these features in the same
   UI streamline this process?

There was also mention of some features we should consider:

   - Pcap queries should be made asynchronous via the UI
   - Take care that a user doesn't hit refresh or POST multiple times and kick
   off 50 mapreduce jobs
   - Options for managing the YARN queue that is used
   - Provide a "cancel" option that kills the MR job, or tell the user to
   go to the CLI to kill their job
   - Managing data if multiple users run queries
   - Strategy for cleaning up jobs and implementing a TTL (I think this one
   will be tricky and definitely needs discussion)
   - Date range or other query limits

A couple other features I would add:

   - Ability to paginate through results
   - Ability to download results through the UI
   - Realtime status of a running job in the UI

Let me know if I missed any points or did not correctly capture them
here.  What
other points do we need to consider?  What other features should be
required?  Nice to have?