Hi Colm, appreciate your suggestions/ efforts in looking into this project, putting my comments inline...
a) There is already a JIRA to bump the version here, although the PR does not apply as it is too old: https://issues.apache.org/jira/browse/EAGLE-1025 . I can submit a new PR, but should the version be 0.6.0 or 0.5.1? *since there are still minor issues, i would say, we put up 0.5.1 as next version. I've updated/rebased the PR ( https://github.com/apache/eagle/pull/936 <https://github.com/apache/eagle/pull/936> )* b) The issues that are "resolved" for the 0.5.1 release in JIRA are actually already fixed in 0.5.0, so they should be updated ( https://issues.apache.org/jira/projects/EAGLE/versions/12341128). However, the following two issues are resolved even though they are not merged to master? https://issues.apache.org/jira/browse/EAGLE-1051 . - * this was pending from developer;s response but i think this is reviewed, so I have merged it.* https://issues.apache.org/jira/browse/EAGLE-1068 . - * this is reopened now. I dont think this is done yet. Also this is big change.* Like I said I can submit PRs but I'm not convinced there is any activity on the project. Where are the rest of the committers? *let me give you some ocontext on this. so there were lot of development happened during last releases, and most of applications that were added are being used in production at multiple enterprise companies, but we are out of ideas on new apps, so at this point we are only focusing on bug fixes and tech upgrades until we get some new ideas to brainstorm and add.* *I think current community's thinking is based on their own industries use-cases, but there is definitely room for new features and integration with other monitoring and security components like grafana and rangers.* *Thanks,* *Jayesh* On Tue, Jan 30, 2018 at 8:11 AM, Colm O hEigeartaigh <[email protected]> wrote: > Hi Jayesh, > > Dev suggestions: > > a) There is already a JIRA to bump the version here, although the PR does > not apply as it is too old: https://issues.apache.org/ > jira/browse/EAGLE-1025. > I can submit a new PR, but should the version be 0.6.0 or 0.5.1? > b) The issues that are "resolved" for the 0.5.1 release in JIRA are > actually already fixed in 0.5.0, so they should be updated ( > https://issues.apache.org/jira/projects/EAGLE/versions/12341128). However, > the following two issues are resolved even though they are not merged to > master? > https://issues.apache.org/jira/browse/EAGLE-1051 > https://issues.apache.org/jira/browse/EAGLE-1068 > > Like I said I can submit PRs but I'm not convinced there is any activity on > the project. Where are the rest of the committers? > > Multiple Publisher issue: > > If I assign two publishers for one policy, the alert only goes to the first > policy. In the logs I see: > > 2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator [INFO] > Alert > event is skipped because it's duplicated: Alert {site=sandbox, > stream=eagle_output,timestamp=2018-01-30 > 00:00:11,300,data={securityZone=NA, dst=null, sensitivityType=NA, > src=/apps/hbase/data/archive/data/default/ambarismoketest, allowed=true, > host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM, > timestamp=1517270411300}, policyId=test, > createdBy=alertBolt3-evaluator_stage1, metaVersion=null} > > It looks like this deduplicator is not working properly, as I'm guessing it > should only be used to de-duplicate events for a single publisher? > > Incognito mode: Already tried it but with the same result. Could I ask you > to try the docker image to see if the UI is working correctly for you > there? > > Colm. > > On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya <[email protected]> > wrote: > > > Hi Colm, > > > > Thanks for the list of dev suggestions, I think we should take care of > > those. even better if you can provide PR with the changes or at keast can > > you please create a ticket so we can track it? > > > > for other issues. > > > > - I dont have any issue with multiple publisher, but if there is any > error > > updating the publisher info in storm topology, i might try restarting the > > topology and see if that works. > > - for us, chrome works as fine as firefox. can u try incognito mode? > just > > to be sure to have clean cache? > > > > Thanks > > Jayesh > > > > > > On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh < > [email protected]> > > wrote: > > > > > Thanks again for your feedback. Jayesh, adding AlertEagleStorePlugin > did > > > the trick, I can now see alerts in the UI, thanks! By the way, I can't > > > configure two Alert Publishers, or else the Alert DeDuplicator bins the > > > alert. Is this a known issue? > > > > > > Could I ask which browser people are using with the UI? There appears > to > > be > > > a bug with Chromium where it doesn't list the pages under Auth.isAdmin > > > even though I am logged on as an administrator. It works OK in Firefox. > > > Even with Firefox though, I only see a limited number of links in the > > > left-hand column - I can't get back to the "integration" page. Can > > someone > > > else confirm this please? > > > > > > Could I suggest the devs do some basic house-keeping tasks: > > > > > > a) "Release" version 0.5.0 in JIRA (it's still listed as "unreleased"). > > > b) Figure out whether the next version will be 0.5.1 or 0.6.0 and > update > > > the versions on Master accordingly with 0.5.1-SNAPSHOT or > 0.6.0-SNAPSHOT. > > > There are some issues marked here as resolved for 0.5.1 - > > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128), > > however > > > I > > > don't see a branch for 0.5.x? > > > > > > Colm. > > > > > > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <[email protected]> > > > wrote: > > > > > > > Hi, > > > > > > > > we do use eagle 0.5 in production although we dont use all the > > available > > > > hadoop applications. > > > > > > > > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a fix > > for > > > > email issue we found while our testing. should be merged soon after a > > > > rebase. > > > > > > > > @Colm, did you tried adding storage publisher > (AlertEagleStorePlugin)? > > to > > > > see alerts on UI ? > > > > > > > > Thanks > > > > Jayesh > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang < > [email protected]> > > > > wrote: > > > > > > > >> Eagle 0.5 was deployed in production as far as I know, but it may > not > > be > > > >> exact the current version in master branch. > > > >> > > > >> Thanks for your investigation, seems there is still some bug in 0.5, > > but > > > >> this particular issue seems is due to dependent components version > > > conflict. > > > >> > > > >> @Jayesh is this Jira ready for merge to master? > https://issues.apache > > . > > > >> org/jira/browse/EAGLE-968 > > > >> > > > >> > > > >> Thanks > > > >> Edward > > > >> > > > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh < > > > [email protected] > > > >> > wrote: > > > >> > > > >>> OK I've made some more progress. I wasn't seeing any email alerts > due > > > to > > > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure > a > > > >>> Kafka > > > >>> alert, I can see the alerts flowing into my topic. It's still not > > clear > > > >>> to > > > >>> me however where the policy "output" is going. I also don't see any > > > >>> alerts > > > >>> in the UI window. > > > >>> > > > >>> Could I ask what the status of the project is in general? There > have > > > been > > > >>> no commits to master since November, so I'm not sure if there is > any > > > >>> point > > > >>> in submitting Pull Requests for outstanding bugs? Are recent > versions > > > of > > > >>> Apache Eagle used in production? > > > >>> > > > >>> Colm. > > > >>> > > > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh < > > > >>> [email protected]> > > > >>> wrote: > > > >>> > > > >>> > > > > >>> > I've done that but I'm not seeing any alerts, which is why I want > > to > > > >>> find > > > >>> > out what the "output" of a policy is and where I can check this. > > > >>> > > > > >>> > Colm. > > > >>> > > > > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN < > [email protected] > > > > > > >>> wrote: > > > >>> > > > > >>> >> Create and add a publisher to see the output. > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> Regards, > > > >>> >> Sudha jenslin > > > >>> >> > > > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" < > > [email protected] > > > > > > > >>> >> wrote: > > > >>> >> > > > >>> >> Thanks - the error was due to a problem running Storm with Java > > 1.8. > > > >>> I've > > > >>> >> abandoned the docker image for now, and I'm trying to get it > > working > > > >>> >> locally. > > > >>> >> > > > >>> >> There are two things I'm not clear on currently, if someone > could > > > >>> fill me > > > >>> >> in: > > > >>> >> > > > >>> >> a) For the 'Hdfs Audit Log Monitor' application, the Kafka > > Consumer > > > >>> Topic > > > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog > Event > > > >>> Sink' > > > >>> >> it > > > >>> >> also specifies 'hdfs_audit_event_sandbox'. However the > > documentation > > > >>> for > > > >>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'? > > > >>> >> > > > >>> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA > > > >>> M_SANDBOX" > > > >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run > > the > > > >>> >> application, I can see cleansed log data appearing in > > > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that > > > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary? > > > >>> >> > > > >>> >> b) It's unclear to me where the output data goes when you > create a > > > >>> policy. > > > >>> >> E.g. say I have: > > > >>> >> > > > >>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains( > src,'/hb > > > >>> ase')] > > > >>> >> select * group by user insert into hdfs_audit_log_enriched_ > > > stream_out > > > >>> >> > > > >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a > > > Kafka > > > >>> >> topic?). How can I check the output to make sure the policy is > > > working > > > >>> >> correctly? > > > >>> >> > > > >>> >> Thanks, > > > >>> >> > > > >>> >> Colm. > > > >>> >> > > > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang < > > > >>> [email protected]> > > > >>> >> wrote: > > > >>> >> > > > >>> >> > There is a data preparation stage between data source(HDFS > audit > > > >>> log) > > > >>> >> and > > > >>> >> > Alert Engine. This stage is running in Storm and transform the > > raw > > > >>> HDFS > > > >>> >> log > > > >>> >> > into something which can be alerted. > > > >>> >> > > > > >>> >> > The input for data preparation is hdfs_audit_log_sandbox topic > > and > > > >>> >> output > > > >>> >> > is > > > >>> >> > hdfs_audit_log_enriched_sandbox. > > > >>> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox > > and > > > >>> >> output > > > >>> >> > is > > > >>> >> > hdfs_audit_log_alert_sandbox. > > > >>> >> > > > > >>> >> > Seems in your case, the data preparation staging is not > working. > > > We > > > >>> >> > probably need look at Storm console and figure out if that > part > > is > > > >>> >> working. > > > >>> >> > > > > >>> >> > Thanks > > > >>> >> > Edward > > > >>> >> > > > > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh < > > > >>> >> [email protected]> > > > >>> >> > wrote: > > > >>> >> > > > > >>> >> > > Hi Jayesh, > > > >>> >> > > > > > >>> >> > > Many thanks for your feedback! I was able to make a little > > > further > > > >>> >> > headway. > > > >>> >> > > There are two configuration problems with the official > docker > > > >>> image: > > > >>> >> > > > > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and " > > > >>> server.eagle.apache.org" > > > >>> >> > (this > > > >>> >> > > only occurs in the instructions for running the docker > image. > > > The > > > >>> >> version > > > >>> >> > > that can be started via the script in the eagle source is > OK). > > > >>> I'll > > > >>> >> > submit > > > >>> >> > > a PR to fix this once I get a basic use-case working. > > > >>> >> > > b) For the audit case, it automatically logs HDFS audit logs > > to > > > >>> the > > > >>> >> KAFKA > > > >>> >> > > topic sandbox_hdfs_audit_log instead of the expected > > > >>> >> > hdfs_audit_log_sandbox > > > >>> >> > > > > > >>> >> > > I've fixed these things locally and I can verify that > > everything > > > >>> is > > > >>> >> > started > > > >>> >> > > correctly in Ambari. I log into the docker container and > > create > > > >>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox > > > >>> topics, > > > >>> >> and > > > >>> >> > > verify that the HDFS audit logs are flowing into the first > > > topic. > > > >>> >> Then in > > > >>> >> > > the UI I start the Alert Engine and then the HDFS Audit Log > > > >>> Monitor > > > >>> >> > > application (changing localhost:6667 to > > > >>> server.eagle.apache.org:6667 > > > >>> >> ). > > > >>> >> > > Both > > > >>> >> > > applications start up correctly and show "running". > > > >>> >> > > > > > >>> >> > > I then create a policy with an email alert along the lines > of > > > from > > > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains( > src,'/h > > > >>> base')] > > > >>> >> > select > > > >>> >> > > * group by user insert into hdfs_audit_log_enriched_ > > > stream_out". > > > >>> >> However > > > >>> >> > > at > > > >>> >> > > this point I'm stuck - nothing appears in the alert window. > Is > > > >>> there > > > >>> >> > > anything obvious I'm doing wrong, or how can I get access to > > > logs > > > >>> to > > > >>> >> > figure > > > >>> >> > > out what the problem is? Other topics such as > > > >>> >> "hdfs_audit_event_sandbox" > > > >>> >> > > are mentioned in the streams window, but the documentation > > > doesn't > > > >>> >> say to > > > >>> >> > > create them. > > > >>> >> > > > > > >>> >> > > The UI is buggy though on both Firefox and Chromium on > Linux. > > > What > > > >>> >> > > browser/platform are people using with the UI? > > > >>> >> > > > > > >>> >> > > Colm. > > > >>> >> > > > > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya < > > > >>> [email protected] > > > >>> >> > > > > >>> >> > > wrote: > > > >>> >> > > > > > >>> >> > > > Hi Colm, > > > >>> >> > > > > > > >>> >> > > > Please find my comments inline. > > > >>> >> > > > > > > >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not > the > > > >>> >> released > > > >>> >> > > > version. > > > >>> >> > > > - this is because we uploaded docker image before apache > > > >>> release. > > > >>> >> > > actually > > > >>> >> > > > this is same codebase apache-eagle-0.5, and it can be > fixed > > > >>> easily > > > >>> >> by > > > >>> >> > > just > > > >>> >> > > > rebuilding docker image. there should not be any mismatch > > due > > > to > > > >>> >> this. > > > >>> >> > > > > > > >>> >> > > > b) Aside from the above, the official docker image uses a > > mix > > > >>> of " > > > >>> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" > as > > > the > > > >>> host > > > >>> >> > > name. > > > >>> >> > > > The HBase service doesn't start by default in Ambari as a > > > >>> result. > > > >>> >> > > > - the only places it uses sandbox is in example script > which > > > you > > > >>> >> will > > > >>> >> > > have > > > >>> >> > > > to update anyway, which i agree that it would be good to > > keep > > > it > > > >>> >> > > > consistent. > > > >>> >> > > > > > > >>> >> > > > c) The UI seems quite buggy. On both chromium and > firefox, I > > > >>> only > > > >>> >> see > > > >>> >> > > > links to "Sandbox" and "Alert" on the left hand-side. > Once I > > > >>> click > > > >>> >> on > > > >>> >> > > > "Alert" I have no way of going back to see the > > applications. I > > > >>> don't > > > >>> >> > see > > > >>> >> > > > the links to "integration" or "sites" as in the picture > > here: > > > >>> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx- > > monito > > > >>> ring > > > >>> >> > > > - when hbase is as deep storage is used, and if eagle app > > has > > > >>> issue > > > >>> >> > > > connecting to hbase, the UI becomes unresponsive. > > > >>> >> > > > > > > >>> >> > > > d) In chromium, the button to create a new policy does not > > > >>> exist - I > > > >>> >> > can > > > >>> >> > > > only see it on Firefox. > > > >>> >> > > > - i have seen when you logged in, you will see admin > > actions. > > > >>> but if > > > >>> >> > this > > > >>> >> > > > still an issue, can you please file UI bug? > > > >>> >> > > > > > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case > > > >>> working, > > > >>> >> but > > > >>> >> > > it > > > >>> >> > > > seems to be stuck in "Initialized". > > > >>> >> > > > this eagle docs has example on how to setup the app. pls > let > > > us > > > >>> >> know if > > > >>> >> > > > you find any gaps. > > > >>> >> > > > > > > >>> >> > > > Thanks for trying out, and sharing your findings, > > > >>> >> > > > Jayesh > > > >>> >> > > > > > > >>> >> > > > > > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh < > > > >>> >> > > [email protected]> > > > >>> >> > > > wrote: > > > >>> >> > > > > > > >>> >> > > >> Hi all, > > > >>> >> > > >> > > > >>> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 > to > > no > > > >>> >> avail. > > > >>> >> > > Here > > > >>> >> > > >> are the problems I've run into so far: > > > >>> >> > > >> > > > >>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not > > the > > > >>> >> released > > > >>> >> > > >> version. > > > >>> >> > > >> > > > >>> >> > > >> b) Aside from the above, the official docker image uses a > > mix > > > >>> of " > > > >>> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" > as > > > the > > > >>> >> host > > > >>> >> > > >> name. The HBase service doesn't start by default in > Ambari > > > as a > > > >>> >> > result. > > > >>> >> > > >> > > > >>> >> > > >> c) The UI seems quite buggy. On both chromium and > firefox, > > I > > > >>> only > > > >>> >> see > > > >>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. > Once > > I > > > >>> click > > > >>> >> on > > > >>> >> > > >> "Alert" I have no way of going back to see the > > applications. > > > I > > > >>> >> don't > > > >>> >> > see > > > >>> >> > > >> the links to "integration" or "sites" as in the picture > > here: > > > >>> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx- > > monito > > > >>> ring > > > >>> >> > > >> > > > >>> >> > > >> d) In chromium, the button to create a new policy does > not > > > >>> exist - > > > >>> >> I > > > >>> >> > can > > > >>> >> > > >> only see it on Firefox. > > > >>> >> > > >> > > > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" > use-case > > > >>> working, > > > >>> >> > but > > > >>> >> > > >> it seems to be stuck in "Initialized". > > > >>> >> > > >> > > > >>> >> > > >> Could someone fill me in on what the "recommended" way is > > to > > > >>> start > > > >>> >> > > Apache > > > >>> >> > > >> Eagle so that I can play around with the functionality > that > > > it > > > >>> >> offers? > > > >>> >> > > >> Clearly the docker approach is buggy. Also, what browser > > > >>> should be > > > >>> >> > used? > > > >>> >> > > >> > > > >>> >> > > >> Thanks, > > > >>> >> > > >> > > > >>> >> > > >> Colm. > > > >>> >> > > >> > > > >>> >> > > >> > > > >>> >> > > >> -- > > > >>> >> > > >> Colm O hEigeartaigh > > > >>> >> > > >> > > > >>> >> > > >> Talend Community Coder > > > >>> >> > > >> http://coders.talend.com > > > >>> >> > > >> > > > >>> >> > > > > > > >>> >> > > > > > > >>> >> > > > > > >>> >> > > > > > >>> >> > > -- > > > >>> >> > > Colm O hEigeartaigh > > > >>> >> > > > > > >>> >> > > Talend Community Coder > > > >>> >> > > http://coders.talend.com > > > >>> >> > > > > > >>> >> > > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> -- > > > >>> >> Colm O hEigeartaigh > > > >>> >> > > > >>> >> Talend Community Coder > > > >>> >> http://coders.talend.com > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> > > > > >>> > > > > >>> > -- > > > >>> > Colm O hEigeartaigh > > > >>> > > > > >>> > Talend Community Coder > > > >>> > http://coders.talend.com > > > >>> > > > > >>> > > > >>> > > > >>> > > > >>> -- > > > >>> Colm O hEigeartaigh > > > >>> > > > >>> Talend Community Coder > > > >>> http://coders.talend.com > > > >>> > > > >> > > > >> > > > > > > > > > > > > > -- > > > Colm O hEigeartaigh > > > > > > Talend Community Coder > > > http://coders.talend.com > > > > > > > > > -- > Colm O hEigeartaigh > > Talend Community Coder > http://coders.talend.com >
