[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...

2018-02-28 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/940
  
@cestella Thanks, Casey. Wouldn't be still hard to tune this solution? 
Still, thread pool tuning and probably the race condition between these threads 
and normal Strom workers makes the tuning hard for a production platform with 
tons of feeds/topologies. Storm resource management is very basic at this stage 
to absorb spikes, and having a separate thread pool transfers the complexity 
from one place to another place. 


---


[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...

2018-02-22 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/940
  
Is there any document somewhere to show how the previous approach was 
implemented? I would like to understand the previous architecture in details. 
Becuase some of the pros/cons didn't make sense to me. Maybe I can help to 
predict what the impact will be. Thanks. 


---


[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

2018-02-01 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/915
  
@cestella Definitely there will be value for having an ability to turn it 
on and off especially since you have already implemented that. However, won't 
be a sort of premature disk optimization? We are storing original_string and 
lots of other things that generally we may not really use them. Are those 
timestamp fields really troublemaker?


---


[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

2018-01-30 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/915
  
@simonellistonball We are using threat intel joiner timestamp currently. I 
am not sure indexing timestamp is added to the latest version or not, but we 
need the ES indexing one as well.


---


[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

2018-01-24 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/907
  
@cestella  Would it be possible to target Storm 1.2 (which was released 2 
days ago) as well?


---


[GitHub] metron issue #879: METRON-1378: Create a summarizer

2018-01-14 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/879
  
@cestella Is there any document or description regarding this feature? How 
would the performance be comparable with normal HBase enrichment?


---


[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

2017-11-27 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/840
  
Yes, I agree. It completely makes sense to minimize the scope and work on 
stabilizing this version at this moment. 


---


[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

2017-11-27 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/840
  
Is this the best time to ask for changing field name convention to avoid 
dot or colon? We are externally using Hive external tables on HDFS data, due to 
Hive limitations we need to change the Metron field convention. I heard there 
is a long term plan in future to use ORC files instead of JSON and maybe Hive 
table can be supported directly. If this is right maybe this is the best time 
we can move towards changing field seperators accordingly.


---


[GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...

2017-11-09 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/831
  
That sounds great. We had a huge headache to find good tuning parameters 
for "indexing" topology. 


---


[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

2017-11-09 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/840
  
It might not be completely related to this PR, but since _timestamp is 
removed in ES 5.x, can we add a specific time of indexing at indexing bolt to 
capture time of indexing. It is useful for the purpose of benchmarking as well 
as evaluation of SLA.


---


[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-27 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra Have you found any way to manage multiple Elasticsearch 
endpoints for load-balancing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-13 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra You're welcome.

As a load balancer. For Elasticsearch client, if you provide a list of 
endpoints, it acts as a load balancer to make sure one of them will not get 
overwhelmed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-13 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra I've just checked it again. It's working now. Perfect. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-12 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra We are using ASA and CEF parsers. Can't you get the field 
names dynamically from Elasticsearch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-11 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra Also, I was wondering whether you have found any way to set 
multiple Elasticsearch urls or not?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-11 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra I've tested your latest commit and it is much better now. 
However, I cannot see all the fields in the customise visible fields panel.


![image](https://user-images.githubusercontent.com/8438293/28059526-dcb212a2-6667-11e7-94b1-9e65eee43be5.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-10 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra I am going to test your latest code to make sure the mentioned 
issues haven't been resolved yet. I was using an older version of your build, 
so it might be outdated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-06 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra I cannot see any error in JS console, except the following 
warning which I don't think is really important.
Angular is running in the development mode. Call enableProdMode() to enable 
the production mode.

More details for the customise functionality for the visible fields:
- I can see only a few number of available fields not all of them in that 
window. 
- "Score" field doesn't do anything. I tried to remove it, and it didn't 
work. It doesn't represent the treat triage score. It seems it is just a hard 
coded field.

Regarding the " retrieves all fields related to events" issue that I have 
mentioned, no I meant it is retrieving all fields not all of the events. So in 
RDBMS terminology, it is like "select *" right now instead of selecting only 
those fields that are visible in GUI. It is affecting query fetch time 
significantly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-05 Thread mraliagha
Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
Hi,

There are few issues that we have faced during our testing. However, I am 
not sure they have been fixed already or not. I am just going to note them.

- The play/pause button doesn't work properly and creates and infinite loop 
sometimes which act as a DOS attack on Elasticsearch. 
- Alert-UI configuration only accepts a single URL for Elasticsearch 
endpoint. It doesn't accept a list of URLs.
- The customise functionality for managing visible fields in UI doesn't 
work.
- The Alert-UI search functionality retrieves all fields related to events 
rather than only target the feasible ones.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---