[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-27 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
Great.  I will get this PR merged.  I am glad to see that this one is ready 
to go.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-27 Thread justinleet 
Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/803
  
I agree. I'm fine with going ahead with this, but I'd like to see end to 
end stability being addressed as the next UI priority, which I believe 
@iraghumitra is already doing some work on.

+1


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-27 Thread merrimanr 
Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/803
  
I agree with @nickwallen.  I think we're good to merge this as long as e2e 
tests are being addressed in a separate PR.  +1


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-27 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
+1  

I'd like to see sign-off from at least one other committer (if not more) 
before this gets merged.

I previously outlined the manual functional testing that I performed.  All 
the core functionality is there.  We also have sufficient test coverage for the 
functionality that was added.

I have been able to get the e2e tests to run good enough (IMO) to move 
forward with the PR.  Based on the experimentation that I describe below, I 
believe that the test failures are due to the tests and test infrastructure and 
not the core functionality in the PR.  

The intermittent failures are a pre-existing condition to this PR.  This 
makes me comfortable enough to merge this PR and then follow-on with test 
fixes.  @iraghumitra mentioned that he is currently working on improving the 
tests and will open a separate PR.  

1. I used Brew to install the following versions of Node/NPM.
```
$ node --version
v8.9.1

$ npm --version
5.5.1
```

2. Before running the tests, truncate the `metron_update` table.  The tests 
cannot currently clear out old test data stored in HBase.

3. Enabling/disabling sets of tests in 
`metron-interface/metron-alerts/protractor.conf.js` also allowed me to see that 
all the tests can pass on an intermittent basis.  It seems that earlier test 
failures may cause later tests to fail unnecessarily.

4. Run the e2e tests using the following steps.

```
cd metron-interface/metron-alerts
npm install
./node_modules/.bin/webdriver-manager update
./node_modules/.bin/protractor
```

Thanks for working through this with me @iraghumitra.



---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-27 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
@iraghumitra Can you describe how you install Node/NPM on your development 
box?  I want to install using the same mechanism (and versions) and see if I 
can get the e2e tests all working like you.  Thanks


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-23 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
Just a quick update... I was working with @iraghumitra yesterday on getting 
the e2e tests to pass for me.  It seems that with older versions of NPM the 
tests do pass, but then only various shades of success with newer versions.  I 
don't totally understand why that is.  Still working through it.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-22 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@nickwallen This is unfortunate  I am not seeing these failures
Spec started

  login to application
✓ should display error message for invalid credentials
✓ should login for valid credentials
✓ should logout

  metron-alerts App
✓ should have all the UI elements
✓ should have all pagination controls and they should be working
✓ should have all settings controls and they should be working
✓ play pause should start polling and stop polling
✓ should select columns from table configuration
✓ should have all time-range controls
✓ should have all time range values populated - 1
✓ should have all time range values populated - 2
✓ should have all time range values populated - 3
✓ should have all time range values populated - 4
✓ should disable date picker when timestamp is present in search
✓ should have now included when to date is empty
✓ should have all time-range included while searching

  metron-alerts configure table
✓ should select columns from table configuration
✓ should rename columns from table configuration

  metron-alerts Search
✓ should display all the default values for saved searches
✓ should have all save search controls and they save search should be 
working
✓ should populate search items when selected on table
✓ should delete search items from search box
✓ should delete first search items from search box having multiple 
search fields
✓ manually entering search queries to search box and pressing enter 
key should search

  metron-alerts tree view
✓ should have all group by elements
✓ drag and drop should change group order
✓ should have group details for single group by
✓ should have group details for multiple group by
✓ should have sort working for group details for multiple sub groups
✓ should have search working for group details for multiple sub groups

  metron-alerts facets
✓ should display facets data
✓ should expand all facets
✓ should have all facet  values
✓ should collapse all facets

  metron-alerts alert status
✓ should change alert status for multiple alerts to OPEN
✓ should change alert status for multiple alerts to DISMISS
✓ should change alert status for multiple alerts to ESCALATE
✓ should change alert status for multiple alerts to RESOLVE
✓ should change alert status for multiple alerts to OPEN in tree view

  metron-alerts alert status
✓ should change alert statuses
✓ should add comments for table view
✓ should add comments for tree view

  meta-alerts workflow
✓ should have all the steps for meta alerts workflow
✓ should create a meta alert from nesting of more than one level

Executed 44 of 44 specs SUCCESS in 6 mins 31 secs.
[20:11:19] I/launcher - 0 instance(s) of WebDriver still running
[20:11:19] I/launcher - chrome #01 passed

~/incubator-metron/metron-interface/metron-alerts(METRON-1252*) » Date 

   
Thu Nov 23 00:36:54 IST 2017`

Can you try them running again, UI tests are sometimes flaky. There seems 
to be timing issue. If you run them independently they work all the time, when 
we run them together they are flaky.
I am currently working on stabilizing them.  If you notice some of the test 
cases have nothing to do with current PR :(


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-22 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
@iraghumitra Are all these e2e tests passing for you?  I am still getting 
failures.  I manually truncated the `metron_updates` table before running them 
also.
```
**
*Failures*
**

1) metron-alerts tree view drag and drop should change group order
  - Expected '0,alerts_ui_e2e,ALERTS,91' to equal 
'0,alerts_ui_e2e,ALERTS,169' First Dash Row should be correct
  - Expected '' to equal '0 US (22)' Dash Group Values should be correct 
for US
  - Expected '' to equal '0 RU (44)' Dash Group Values should be present 
for RU
  - Expected '' to equal '0 FR (25)' Dash Group Values should be present 
for FR

2) metron-alerts tree view should have sort working for group details for 
multiple sub groups
  - Failed: stale element reference: element is not attached to the page 
document
(Session info: chrome=62.0.3202.94)
(Driver info: chromedriver=2.33.506106 
(8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2),platform=Mac OS X 10.13.1 x86_64)

3) metron-alerts tree view should have search working for group details for 
multiple sub groups
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

4) meta-alerts workflow should create a meta alert from nesting of more 
than one level
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

Executed 44 of 44 specs (4 FAILED) in 8 mins 46 secs.
```


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-22 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@merrimanr Added the test case for searching an alert inside meta-alert. 
Also took the liberty to fix the timezone issue it was a one-liner.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread merrimanr 
Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/803
  
Most of the functional tests worked for me as well.  I was able to get the 
e2e tests to pass after several runs with the exception of a time picker test 
that I believe is due to a timezone issue and not related to this PR.

I was also able to test that meta alerts show up in search results when a 
search matches a child alert field.  However I don't see this case covered in 
the e2e tests.  I think this is a pretty critical function and needs a test.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
IMHO I think the functionality is good on this PR.  There are 3 functional 
tests that failed, but they are minor and can be addressed after we get this PR 
in.

My only hold-up right now is that I cannot get the e2e tests to pass.  I am 
going to rebuild Full Dev and try again.  It might have something to do with 
stale data.
- [ ] Run e2e tests.

The functional tests that I am working with...
- [ ] Validate initial state; No metaalerts exist.
- [ ] Create a metaalert from 1 level grouping
- [ ] Create a metaalert from 2 level grouping
- [ ] Create a metaalert from 3 level grouping
- [ ] Create a metaalert from 4 level grouping
- [ ] Delete a metaalert containing child alerts
- [ ] Delete one child alert from an existing metaalert
- [ ] Delete child alerts until a metaalert contains no child alerts
- [ ] Delete a metaalert with no child alerts.  (FAIL)
- [ ] Add an alert to a metaalert with no child alerts. (FAIL)
- [ ] Add one child alert to existing metaalert 
- [ ] Should not be able to add alert to a metaalert that is resolved (FAIL)
- [ ] Add multiple child alerts to an existing metaalert
- [ ] Change status of one metaalert
- [ ] Change status of multiple metaalerts 
- [ ] Rename a metaalert
- [ ] Add a comment to a metaalert
- [ ] Delete a comment from a metaalert



---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@nickwallen I will raise a jira to track this. Even if we are able to 
reproduce this issue it is unlikely bcoz this PR.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@nickwallen can you raise a jira to track this. I guess even if we are able 
to reproduce I highly doubt if it is bcoz of changes in this PR.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread justinleet 
Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/803
  
@nickwallen I agree, unless I'm missing something, it seems like a bug with 
recent searches, since you didn't say searches were weird during the initial 
attempts.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
@justinleet @iraghumitra I am glad that no one else can reproduce.  I will 
work on trying to get a set of steps to reproduce this.  Let's not worry about 
this on the current PR.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@nickwallen That looks wicked can you share your manual steps that lead to 
this?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread justinleet 
Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/803
  
@nickwallen I haven't been able to get into the same sort of state, but I 
know you've used/tested the UI more than me.  Do you know a repeatable way to 
reproduce this?  Or have you been able to determine if it's preexisting, 
assuming you've looked into it.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
Something seems to be wrong with saved searches.  When I run through a 
manual test script of creating metaalerts, deleting and navigating, most (if 
not all) of my saved searches are these giant query strings that are incorrect. 
 

![screen shot 2017-11-21 at 10 52 47 
am](https://user-images.githubusercontent.com/2475409/33082288-68416002-ceaa-11e7-98ec-62dd5065b047.png)



---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@merrimanr Thanks for 
https://github.com/iraghumitra/incubator-metron/pull/5. It was a pretty 
significant issue. I merged the fix 👍 


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
>  I am not refreshing on purpose. If I refresh the UI after any operation 
users would lose the current context and see new alerts (If available).

Ok @iraghumitra.  Since you have a valid concern about whether this is the 
right approach, then let's NOT change this in your PR.  We can track this 
separately and come to the right resolution after your PR goes in.  Thanks.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@nickwallen I am not refreshing on purpose. If I refresh the UI after any 
operation users would lose the current context and see new alerts (If 
available). If you think this is not a major concern I can add a call to 
refresh the UI.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
Hi @iraghumitra - The UI does not refresh itself after a metaalert is 
deleted.  If this is something easy to fix, let's tackle it.  Otherwise, I can 
create a JIRA to track this and we can fix it as a separate PR.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
I noticed that when I am running e2e test's, the comments that were added 
in the previous run are still visible at times. The e2e tests delete's  all the 
comments that it added and the only way I can see this happening is if in the 
previous run of tests add comment was successful and remove comment fails. The 
best fix for this would be to clean the metron_update table before running the 
tests. Since we don't have a rest API's for Hbase I am looking at finding 
alternatives. For the time being, if  you bump into this issue I request you to 
truncate the meton_update table manually.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-21 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@justinleet Thanks for taking time to test the PR. I was using an incorrect 
index to find the alert to delete when I implemented the new API's I fixed it 
now. Also added tests for all the issues I fixed so far.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-20 Thread justinleet 
Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/803
  
I made a metaalert with four entries

```
957f20a3-d67b-407a-a593-09bdcbca19df
b18e0949-9ac5-48e2-945f-74f9609667db
8fb8f6cf-861f-4337-8d34-1becc9cecad9
0c6543c8-c5b3-4540-ba83-b338b1aa52f0
```

When I delete an alert, e.g. `0c6543c8-c5b3-4540-ba83-b338b1aa52f0`, the 
wrong alert is removed (in this case `957f20a3-d67b-407a-a593-09bdcbca19df`).

The wrong alert is passed
```
{
  "metaAlertGuid": "f5cd050c-7a7d-4562-bfc1-1ca5796fa1e4",
  "alerts": [
{
  "guid": "957f20a3-d67b-407a-a593-09bdcbca19df",
  "sensorType": "snort",
  "index": ""
}
  ]
}
```

It appears to always submit a request to remove the first alert, not the 
alert I choose in the UI (although I haven't extensively validated that)


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-20 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@justinleet my bad. The search query to fetch all the alerts in a group was 
returning a nested object since I was passing 'source: type' twice in the 
fields. I don't know why I was getting a nested object if I pass the same 
column name twice but for now, I fixed it in the UI.

The test spec for the above issue is in flight...


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-20 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@justinleet my bad. The search query to fetch all the alerts in a group was 
returning a nested object since I was passing 'source: type' twice in the 
fields. I don't know why I was getting a nested object if I pass the same 
column name twice but for now, I fixed it in the UI.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-20 Thread merrimanr 
Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/803
  
I've verified the bug reported by Justin happens when you create a meta 
alert from a group that is nested by more than 1 level.  Creating a meta alert 
from a top level group works.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-20 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
That's weird @justinleet .  The create request is working for me.  I'll 
mess with it some more and try to replicate what you are seeing.

I am seeing a separate issue on the REST UI side.
```
17/11/20 16:55:40 ERROR dao.ElasticsearchColumnMetadataDao: Field type 
mismatch: snort_index_2017.11.20.16.threat:triage:score has type float while 
metaalert_index.threat:triage:score has type double.  Defaulting type to other.
17/11/20 16:55:40 ERROR dao.ElasticsearchColumnMetadataDao: Field type 
mismatch: bro_index_2017.11.20.16.id has type string while 
snort_index_2017.11.20.16.id has type integer.  Defaulting type to other.
17/11/20 16:55:40 ERROR dao.ElasticsearchRequestSubmitter: Failed to 
execute search; error='NotSerializableExceptionWrapper: class_cast_exception: 
java.lang.Double cannot be cast to java.lang.Float', 
search='{"from":0,"size":25,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"should":[{"query_string":{"query":"*"}},{"nested":{"query":{"query_string":{"query":"*"}},"path":"alert"}}]}},{"bool":{"should":[{"term":{"status":"active"}},{"bool":{"must_not":{"exists":{"field":"status"]}}],"must_not":{"exists":{"field":"metaalerts"}},"_source":{"includes":[],"excludes":[]},"sort":[{"threat:triage:score":{"order":"desc","missing":"_last","unmapped_type":"other"}}],"track_scores":true,"aggregations":{"source:type_count":{"terms":{"field":"source:type"}},"ip_src_addr_count":{"terms":{"field":"ip_src_addr"}},"ip_dst_addr_count":{"terms":{"field":"ip_dst_addr"}},"host_count":{"terms":{"field":"host"}},"enrichments:geo:ip_dst_addr:country_count":{"terms":{"field":"enrichm
 ents:geo:ip_dst_addr:country"'
Failed to execute phase [query], [reduce] ; shardFailures 
{[0KqVPgyOT2KKCjTKZYIl3Q][bro_index_2017.11.20.16][0]: 
RemoteTransportException[[node1][192.168.66.121:9300][indices:data/read/search[phase/query]]];
 nested: SearchParseException[failed to parse search source 
[{"from":0,"size":25,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"should":[{"query_string":{"query":"*"}},{"nested":{"query":{"query_string":{"query":"*"}},"path":"alert"}}]}},{"bool":{"should":[{"term":{"status":"active"}},{"bool":{"must_not":{"exists":{"field":"status"]}}],"must_not":{"exists":{"field":"metaalerts"}},"_source":{"includes":[],"excludes":[]},"sort":[{"threat:triage:score":{"order":"desc","missing":"_last","unmapped_type":"other"}}],"track_scores":true,"aggregations":{"source:type_count":{"terms":{"field":"source:type"}},"ip_src_addr_count":{"terms":{"field":"ip_src_addr"}},"ip_dst_addr_count":{"terms":{"field":"ip_dst_addr"}},"host_count":{"terms":{"field":"host"}},"enrichmen
 
ts:geo:ip_dst_addr:country_count":{"terms":{"field":"enrichments:geo:ip_dst_addr:country"]];
 nested: IllegalArgumentException[No mapper found for type [other]]; }
at 
org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:176)
at 
org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onResponse(AbstractSearchAsyncAction.java:147)
at 
org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onResponse(AbstractSearchAsyncAction.java:144)
at 
org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:41)
at 
org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:819)
at 
org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:803)
at 
org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:793)
at 
org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:58)
at 
org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:134)
at 
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:369)
at 
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
at 
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
at 
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
at 
org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
at 
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecuto

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-20 Thread justinleet 
Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/803
  
@iraghumitra looks like the new API isn't being used quite right.

Sample from the dev tools
```
{
  "alerts": [
{
  "guid": "50a0c1f6-8a55-4cdd-a031-81c53174ad7b",
  "sensorType": [
"snort"
  ],
  "index": "snort_index_2017.11.20.15"
},
...
```

This should be, I believe,
```
{
  "alerts": [
{
  "guid": "50a0c1f6-8a55-4cdd-a031-81c53174ad7b",
  "sensorType": "snort",
  "index": "snort_index_2017.11.20.15"
},
...
  ],
  "groups": [
"source:type"
  ]
}
```


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-20 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
Merged the PR with master and used new API's for creating meta-alerts. 
Please feel free to review and let me know the feedback.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-20 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
@iraghumitra I see that you merged some changes.  Is this ready to test?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-17 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
Hi @iraghumitra - I'd like to see your work get in ASAP.  Can you merge 
with master when you get a chance?  

Also, I think you need to make some updates based on recent PRs that have 
gone in.  @justinleet pointed out that at a minimum the UI has to change for 
#824.  There may be others too.




---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-02 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@merrimanr Looks like the following rest-api's are not working as expected
 - When a meta alert is created and when we try to group by an IP 
address(ip_dst_addr/ip_src_addr) the IP address is shown as string instead of IP
 - When a meta alert is created the alert details of an alert under 
meta-alert are missing when we query using findone

The spec that is failing is unable to find state field in the details view 
bcoz of findone issue  that i mentioned hence the failure. (The 
alert-detail-status.e2e-spec.ts set's the status of the alert that it is trying 
to test to 
[NEW](https://github.com/apache/metron/blob/383d798c7827e916068f89ff362a6ecccb550241/metron-interface/metron-alerts/e2e/alert-details/alert-status/alert-details-status.e2e-spec.ts#L55)
 before starting. So I feel the issue might not be bcoz of #796. Anyhow, I will 
check this again)

I am trying to address the usage of sleep in specs by removing the 'fade' 
transition for dialogs. We don't need this since the dialog comes on top of the 
window and the css transition is not needed.

Also, i noticed that sometimes when we delete a meta alert not all the 
alerts in the meta alert are returned back the alerts pool.Ex: If we have 100 
alerts and create a meta-alert with 50 alerts and then delete the meta alert. 
The alert search shows only 60 alerts.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-31 Thread merrimanr 
Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/803
  
Yes HBase is running.  The first set of status tests are working (they 
would not if HBase were down).


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-31 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@merrimanr is HBase running these errors are typically seen when HBase is 
down.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-30 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@merrimanr METRON-1272 has broken few of our assertions and these are 
cascaded failures bcoz of them. I am working on fixing them.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-30 Thread merrimanr 
Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/803
  
I am consistently getting these e2e test failures:
```
1) meta-alerts workflow should have all the steps for meta alerts workflow
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

2) metron-alerts alert status should change alert statuses
  - Expected 'RESOLVE' to equal 'OPEN'.
  - Expected 'NEW' to equal 'OPEN'.
  - Expected 'RESOLVE' to equal 'DISMISS'.
  - Expected 'NEW' to equal 'DISMISS'.
  - Expected 'RESOLVE' to equal 'ESCALATE'.
  - Expected 'NEW' to equal 'ESCALATE'.
  - Expected 'NEW' to equal 'RESOLVE'.

3) metron-alerts alert status should add comments for table view
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

4) metron-alerts alert status should add comments for tree view
  - Failed: unknown error: Element   is not clickable at point (2485, 311). Other 
element would receive the click:  
(Session info: chrome=61.0.3163.100)
(Driver info: chromedriver=2.33.506106 
(8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2),platform=Mac OS X 10.11.6 x86_64)
```

Not sure if 3) and 4) are cascading errors caused by previous errors.  I 
suspect 2) was introduced by https://github.com/apache/metron/pull/796.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-26 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
I had to force push the last merge since I was facing issues with github. 


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-23 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
I added a single test case which covers the entire workflow of meta-alert. 
I couldn't think of a way to decompose it into smaller cases without making 
them dependent on each other. Any suggestions on the existing tests or for any 
more test case are most welcomed.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-23 Thread iraghumitra 
Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/803
  
@nickwallen  your observation is right. As mentioned in the Limitations 
section of CC, as of now we are not displaying the meta-alerts in the group 
view. Hence I didn't see a need to fire a refresh. 

However looking at the screenshot it occurred to me that we need to refresh 
the UI to at least get the group numbers correct. This is a simple change, I 
will add this functionality once de-duping of alerts in meta alert is available 
otherwise you will see the group back which might cause even more confusion for 
now.



---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-23 Thread nickwallen 
Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
I was testing #811 along with the changes in this PR and noted an issue.  
It appears that the UI does not refresh itself after a meta-alert is created.

1. First, I isolate 10 alerts with a specific host name that I want to work 
with.  I have turned off ingest so that no additional alerts should appear 
during the running of this example.  I can see that the facet counts are what I 
would expect.

![screen shot 2017-10-23 at 3 01 45 
pm](https://user-images.githubusercontent.com/2475409/31908592-6de23998-b805-11e7-8201-c9983bfdc476.png)

2. Next, I group by host so that I can create my meta-alert.

![screen shot 2017-10-23 at 3 01 58 
pm](https://user-images.githubusercontent.com/2475409/31908646-97cf304e-b805-11e7-9ca4-14bb0269fc0c.png)

![screen shot 2017-10-23 at 3 02 09 
pm](https://user-images.githubusercontent.com/2475409/31908665-a1bea3e6-b805-11e7-96e1-572faaf38e7a.png)

3. Immediately after creating the meta-alert, I do not immediately see it.  
I think this is a problem with the UI itself not refreshing after creating the 
alert.  

![screen shot 2017-10-23 at 3 02 37 
pm](https://user-images.githubusercontent.com/2475409/31908754-edf3aebe-b805-11e7-86ff-838a1572a27d.png)

4. If I then trigger another search, I do see the meta-alert.  I would 
expect to see it directly without having to manually trigger another search.  
Ignore the facet counts being 0.  I think that is a problem with #811.

![screen shot 2017-10-23 at 3 03 14 
pm](https://user-images.githubusercontent.com/2475409/31908976-a74e13cc-b806-11e7-860b-8a86aac8d36a.png)



---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-21 Thread justinleet 
Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/803
  
Potentially we want to expose some abstraction for the ES options for 
missing field sorting (that I admittedly don't know exist in Solr).


https://www.elastic.co/guide/en/elasticsearch/reference/2.3/search-request-sort.html#_missing_values


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-21 Thread merrimanr 
Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/803
  
I just submitted a 
[PR](https://github.com/iraghumitra/incubator-metron/pull/3) against this PR 
that addresses all of the bugs reported above except 1:

- when metaalerts and alerts are in the same result set, sorting on fields 
other than timestamp causes metaalerts to be excluded

I think we need more discussion on what should happen in that scenario 
(sorting on fields that doesn't exist in metaalerts).  Let me know what you 
think.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-20 Thread james-sirota 
Github user james-sirota commented on the issue:

https://github.com/apache/metron/pull/803
  
I filed the following follow-on PRs per your comments:

https://issues.apache.org/jira/browse/METRON-1268
https://issues.apache.org/jira/browse/METRON-1269




---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-19 Thread james-sirota 
Github user james-sirota commented on the issue:

https://github.com/apache/metron/pull/803
  
You should not have empty meta alerts.  That does not make sense 


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-19 Thread merrimanr 
Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/803
  
I did an initial review of this and I see several things we need to work 
through.  It's a pretty significant feature so that's not surprising.  I know 
there are some other PRs being worked on that this PR depends on 
(https://github.com/apache/metron/pull/806).  Some questions I have:
- In this PR description, what does 'Change the state of meta alert' mean?
- Can I create a new empty metaalert?
- Can I create a metaalert by manually selecting alerts?
- How do I create a metaalert from a group that is >1 levels deep? 
- How do I see all metaalerts? I was able to query with "_exists_:alert.*" 
but that is not intuitive.
- Can I only remove 1 alert from a metaalert at a time?
- Would it be useful to assign a name to your metaalert in the confirmation 
form rather than having to find it after you create it and rename in the detail 
view?
- Not a fan of vertical scrolling through alerts in metaalert detail, would 
it be possible to add pagination?

Some initial bugs I found (commented on some of these):
- metaalert index name throughout code is wrong, should be 'metaalert'
- metaalerts have alert_status set to 'NEW'
- when you group by ip address fields, then expand group, count goes to 0 
and corrects after the next search (not sure if this existed before this PR)
- clicking on a metaalert fires this findOne call:
```
{
  "guid": "ca80f4fc-0cdb-431c-b972-c460dad022ee",
  "sensorType": "undefined"
}
```
- not obvious from a user perspective what happens when I remove all alerts 
from a metaalert (I can see rest call that sets metaalert status to inactive)
- missing space in alert merge confirmation and 1 alert displays as 
'1alerts'
- need to add metaalert index to e2e npm environment
- when metaalerts and alerts are in the same result set, sorting on fields 
other than timestamp causes metaalerts to be excluded
- when I select an alert and then select Add to Alert in bulk actions, 
metaalerts display 0 alerts  ie.
 (0) 
- adding a comment to metaalert is failing because sensorType is undefined 
in patch request:
```
{
  "patch": [
{
  "op": "add",
  "path": "/comments",
  "value": [
{
  "comment": "test",
  "username": "user",
  "timestamp": 1508446386940
}
  ]
}
  ],
  "guid": "b9479340-316b-46db-baa5-0a0376ff015a",
  "index": "metaalert_index",
  "sensorType": "undefined"
}
```
- when I change/assign a metaalert name, reverts back to id in search 
result list on next search
- comment icons in the list view are not appearing after I add a comment 
even though comments are displayed in the metaalert default view (likely 
related to failed add comment call, being written only to hbase and not ES)

I'm also assuming more e2e tests are coming soon.


---