Re: [Discuss] Minimal General-purpose Monitoring Engine Proposal

2015-11-10 Thread Zhang, Edward (GDI Hadoop)
We just need a better name for “AlertExecutorService”. This name scared people off :-) if you can visualize the draft topology in UI, AlertExecutorService is pretty straightforward. On 11/10/15, 23:28, "Jilin" wrote: >For UI side we can do the configuration of AlertStreamService >& AlertStream

Re: [Discuss] Minimal General-purpose Monitoring Engine Proposal

2015-11-10 Thread Jilin
For UI side we can do the configuration of AlertStreamService & AlertStreamSchemaService management.(Enhance the metadata page). Which could let admin to manage the Stream. But for *AlertExecutorService*, we should consider how to config it. I think this entity set in backend is better than in fro

[jira] [Created] (EAGLE-23) External data join API as part of EAGLE stream API

2015-11-10 Thread Edward Zhang (JIRA)
Edward Zhang created EAGLE-23: - Summary: External data join API as part of EAGLE stream API Key: EAGLE-23 URL: https://issues.apache.org/jira/browse/EAGLE-23 Project: Eagle Issue Type: Improvemen

[jira] [Created] (EAGLE-22) External data join at scale

2015-11-10 Thread Edward Zhang (JIRA)
Edward Zhang created EAGLE-22: - Summary: External data join at scale Key: EAGLE-22 URL: https://issues.apache.org/jira/browse/EAGLE-22 Project: Eagle Issue Type: Improvement Reporter:

Re: [Discuss] Minimal General-purpose Monitoring Engine Proposal

2015-11-10 Thread Zhang, Edward (GDI Hadoop)
I figured out a graceful way of onboarding a new datasource (Kafka) which enables general purpose monitoring framework, please review and give feedback. (If we do good design on this, then we can easily onboard large number of hadoop jmx metrics :-) ) 1. In UI, Role Modeler describes kafka settin

[jira] [Commented] (EAGLE-16) HBase security log integration for HBase security monitoring

2015-11-10 Thread Don Bosco Durai (JIRA)
[ https://issues.apache.org/jira/browse/EAGLE-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1506#comment-1506 ] Don Bosco Durai commented on EAGLE-16: -- Because of the volume of logs in HBase and Kafk

[jira] [Created] (EAGLE-21) Balance policy computation in runtime to avoid policy evaluation screw

2015-11-10 Thread Edward Zhang (JIRA)
Edward Zhang created EAGLE-21: - Summary: Balance policy computation in runtime to avoid policy evaluation screw Key: EAGLE-21 URL: https://issues.apache.org/jira/browse/EAGLE-21 Project: Eagle I

Re: [Discuss] Minimal General-purpose Monitoring Engine Proposal

2015-11-10 Thread Zhang, Edward (GDI Hadoop)
I think over this problem again, now I want to bring up some considerations we should take into account while we do design. 1. Simple general-purpose monitoring framework and how we model stream are two different problems Simple general-purpose monitoring framework tries to onboard a new data s

[jira] [Commented] (EAGLE-16) HBase security log integration for HBase security monitoring

2015-11-10 Thread Edward Zhang (JIRA)
[ https://issues.apache.org/jira/browse/EAGLE-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14999512#comment-14999512 ] Edward Zhang commented on EAGLE-16: --- Right, we can develop adaptor for Ranger-enabled data

[jira] [Commented] (EAGLE-11) Eagle and Splunk Integration for security activity analysis

2015-11-10 Thread Edward Zhang (JIRA)
[ https://issues.apache.org/jira/browse/EAGLE-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14999509#comment-14999509 ] Edward Zhang commented on EAGLE-11: --- Also, Syslog is a more popular way for Splunk to acce

[jira] [Commented] (EAGLE-13) Eagle machine learning no longer needs hourly aggregation

2015-11-10 Thread Edward Zhang (JIRA)
[ https://issues.apache.org/jira/browse/EAGLE-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14999465#comment-14999465 ] Edward Zhang commented on EAGLE-13: --- pull request created, please review. https://github.c

Eagle JIRA privilege request

2015-11-10 Thread Edward Zhang
Hi Eagle mentors, Could you please grant me and other committers of JIRA privilege to assign JIRA ticket? Today we only have read access by default. My jira username is: yonzhang2012 My apache id is : yonzhang2012 Thanks Edward Zhang

Re: Best Practice for Code Collaboration in Apache Way

2015-11-10 Thread Julian Hyde
I think using http://github.com/apache/incubator-eagle with pull requests would be perfect. Some projects accept also patches but I think it's simpler & better if your community is OK with just doing pull requests. On Tue, Nov 10, 2015 at 1:00 AM, Hao Chen wrote: > Hi Eagle Mentors, > > I would l

[jira] [Commented] (EAGLE-11) Eagle and Splunk Integration for security activity analysis

2015-11-10 Thread SunLibin (JIRA)
[ https://issues.apache.org/jira/browse/EAGLE-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998324#comment-14998324 ] SunLibin commented on EAGLE-11: --- Good idea, Probably we can streaming eagle alerts to kafka S

[jira] [Created] (EAGLE-20) Support Filebeat as Log Shipper for better performance

2015-11-10 Thread Hao Chen (JIRA)
Hao Chen created EAGLE-20: - Summary: Support Filebeat as Log Shipper for better performance Key: EAGLE-20 URL: https://issues.apache.org/jira/browse/EAGLE-20 Project: Eagle Issue Type: New Feature

Best Practice for Code Collaboration in Apache Way

2015-11-10 Thread Hao Chen
Hi Eagle Mentors, I would like to consult about what is the Best Practice for Code Collaboration in Apache Way? Currently we are using github https://github.com/eBay/Eagle with pull request. What practice should we follow after migrating into Apache code repository? -- Hao

Re: [Discuss] Minimal General-purpose Monitoring Engine Proposal

2015-11-10 Thread Hao Chen
As the stream schema model, you could refer to: stream { name = "MonitoredStream" executor = "MonitoredStream" attributes = [ { name = "value", type = "double", // more attribute properties }, // more attributes definition

Re: [Discuss] Minimal General-purpose Monitoring Engine Proposal

2015-11-10 Thread 蒋吉麟
Agree. Use api create stream schema is not user friendly. If not familiar with create entity structure, it's not easy to use. I think we can design the UI for simple add the stream schema. :) 2015-11-10 16:06 GMT+08:00 Hao Chen : > *Jirap* > > https://issues.apache.org/jira/browse/EAGLE-5 > > *Us

Re: [Discuss] User Profile ML Model Visualization Design

2015-11-10 Thread Hao Chen
Jira Created: https://issues.apache.org/jira/browse/EAGLE-19 -- Hao On Tue, Nov 10, 2015 at 4:19 PM, Hao Chen wrote: > Eagle Team, > > Currently we have basic visualization for two ml algorithm: KDE and > EigenDecomposition, but the current visualization chart seems not very > expressive eno

[jira] [Created] (EAGLE-19) User Profile ML Model Visualization for KDE and EigenDecomposition

2015-11-10 Thread Hao Chen (JIRA)
Hao Chen created EAGLE-19: - Summary: User Profile ML Model Visualization for KDE and EigenDecomposition Key: EAGLE-19 URL: https://issues.apache.org/jira/browse/EAGLE-19 Project: Eagle Issue Type: N

[Discuss] User Profile ML Model Visualization Design

2015-11-10 Thread Hao Chen
Eagle Team, Currently we have basic visualization for two ml algorithm: KDE and EigenDecomposition, but the current visualization chart seems not very expressive enough, so let's discuss a more reasonable design. Regards, Hao

[jira] [Updated] (EAGLE-5) Minimal general-purpose monitoring engine

2015-11-10 Thread Hao Chen (JIRA)
[ https://issues.apache.org/jira/browse/EAGLE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Chen updated EAGLE-5: - Description: We should decouple the core framework and features from security monitoring case, so that we could brin

[Discuss] Minimal General-purpose Monitoring Engine Proposal

2015-11-10 Thread Hao Chen
*Jirap* https://issues.apache.org/jira/browse/EAGLE-5 *Use Cases* Currently Eagle supports very complex data processing pipeline for hadoop audit/security logs, but I think we reuse some valuable components in Eagle: 1) distributed policy engine 2) highly abstracted streaming program API 3)