Hi Jayesh, Dev suggestions:
a) There is already a JIRA to bump the version here, although the PR does not apply as it is too old: https://issues.apache.org/jira/browse/EAGLE-1025. I can submit a new PR, but should the version be 0.6.0 or 0.5.1? b) The issues that are "resolved" for the 0.5.1 release in JIRA are actually already fixed in 0.5.0, so they should be updated ( https://issues.apache.org/jira/projects/EAGLE/versions/12341128). However, the following two issues are resolved even though they are not merged to master? https://issues.apache.org/jira/browse/EAGLE-1051 https://issues.apache.org/jira/browse/EAGLE-1068 Like I said I can submit PRs but I'm not convinced there is any activity on the project. Where are the rest of the committers? Multiple Publisher issue: If I assign two publishers for one policy, the alert only goes to the first policy. In the logs I see: 2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator [INFO] Alert event is skipped because it's duplicated: Alert {site=sandbox, stream=eagle_output,timestamp=2018-01-30 00:00:11,300,data={securityZone=NA, dst=null, sensitivityType=NA, src=/apps/hbase/data/archive/data/default/ambarismoketest, allowed=true, host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM, timestamp=1517270411300}, policyId=test, createdBy=alertBolt3-evaluator_stage1, metaVersion=null} It looks like this deduplicator is not working properly, as I'm guessing it should only be used to de-duplicate events for a single publisher? Incognito mode: Already tried it but with the same result. Could I ask you to try the docker image to see if the UI is working correctly for you there? Colm. On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya <[email protected]> wrote: > Hi Colm, > > Thanks for the list of dev suggestions, I think we should take care of > those. even better if you can provide PR with the changes or at keast can > you please create a ticket so we can track it? > > for other issues. > > - I dont have any issue with multiple publisher, but if there is any error > updating the publisher info in storm topology, i might try restarting the > topology and see if that works. > - for us, chrome works as fine as firefox. can u try incognito mode? just > to be sure to have clean cache? > > Thanks > Jayesh > > > On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh <[email protected]> > wrote: > > > Thanks again for your feedback. Jayesh, adding AlertEagleStorePlugin did > > the trick, I can now see alerts in the UI, thanks! By the way, I can't > > configure two Alert Publishers, or else the Alert DeDuplicator bins the > > alert. Is this a known issue? > > > > Could I ask which browser people are using with the UI? There appears to > be > > a bug with Chromium where it doesn't list the pages under Auth.isAdmin > > even though I am logged on as an administrator. It works OK in Firefox. > > Even with Firefox though, I only see a limited number of links in the > > left-hand column - I can't get back to the "integration" page. Can > someone > > else confirm this please? > > > > Could I suggest the devs do some basic house-keeping tasks: > > > > a) "Release" version 0.5.0 in JIRA (it's still listed as "unreleased"). > > b) Figure out whether the next version will be 0.5.1 or 0.6.0 and update > > the versions on Master accordingly with 0.5.1-SNAPSHOT or 0.6.0-SNAPSHOT. > > There are some issues marked here as resolved for 0.5.1 - > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128), > however > > I > > don't see a branch for 0.5.x? > > > > Colm. > > > > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <[email protected]> > > wrote: > > > > > Hi, > > > > > > we do use eagle 0.5 in production although we dont use all the > available > > > hadoop applications. > > > > > > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a fix > for > > > email issue we found while our testing. should be merged soon after a > > > rebase. > > > > > > @Colm, did you tried adding storage publisher (AlertEagleStorePlugin)? > to > > > see alerts on UI ? > > > > > > Thanks > > > Jayesh > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <[email protected]> > > > wrote: > > > > > >> Eagle 0.5 was deployed in production as far as I know, but it may not > be > > >> exact the current version in master branch. > > >> > > >> Thanks for your investigation, seems there is still some bug in 0.5, > but > > >> this particular issue seems is due to dependent components version > > conflict. > > >> > > >> @Jayesh is this Jira ready for merge to master? https://issues.apache > . > > >> org/jira/browse/EAGLE-968 > > >> > > >> > > >> Thanks > > >> Edward > > >> > > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh < > > [email protected] > > >> > wrote: > > >> > > >>> OK I've made some more progress. I wasn't seeing any email alerts due > > to > > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a > > >>> Kafka > > >>> alert, I can see the alerts flowing into my topic. It's still not > clear > > >>> to > > >>> me however where the policy "output" is going. I also don't see any > > >>> alerts > > >>> in the UI window. > > >>> > > >>> Could I ask what the status of the project is in general? There have > > been > > >>> no commits to master since November, so I'm not sure if there is any > > >>> point > > >>> in submitting Pull Requests for outstanding bugs? Are recent versions > > of > > >>> Apache Eagle used in production? > > >>> > > >>> Colm. > > >>> > > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh < > > >>> [email protected]> > > >>> wrote: > > >>> > > >>> > > > >>> > I've done that but I'm not seeing any alerts, which is why I want > to > > >>> find > > >>> > out what the "output" of a policy is and where I can check this. > > >>> > > > >>> > Colm. > > >>> > > > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <[email protected] > > > > >>> wrote: > > >>> > > > >>> >> Create and add a publisher to see the output. > > >>> >> > > >>> >> > > >>> >> > > >>> >> Regards, > > >>> >> Sudha jenslin > > >>> >> > > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" < > [email protected] > > > > > >>> >> wrote: > > >>> >> > > >>> >> Thanks - the error was due to a problem running Storm with Java > 1.8. > > >>> I've > > >>> >> abandoned the docker image for now, and I'm trying to get it > working > > >>> >> locally. > > >>> >> > > >>> >> There are two things I'm not clear on currently, if someone could > > >>> fill me > > >>> >> in: > > >>> >> > > >>> >> a) For the 'Hdfs Audit Log Monitor' application, the Kafka > Consumer > > >>> Topic > > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event > > >>> Sink' > > >>> >> it > > >>> >> also specifies 'hdfs_audit_event_sandbox'. However the > documentation > > >>> for > > >>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'? > > >>> >> > > >>> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA > > >>> M_SANDBOX" > > >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run > the > > >>> >> application, I can see cleansed log data appearing in > > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that > > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary? > > >>> >> > > >>> >> b) It's unclear to me where the output data goes when you create a > > >>> policy. > > >>> >> E.g. say I have: > > >>> >> > > >>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hb > > >>> ase')] > > >>> >> select * group by user insert into hdfs_audit_log_enriched_ > > stream_out > > >>> >> > > >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a > > Kafka > > >>> >> topic?). How can I check the output to make sure the policy is > > working > > >>> >> correctly? > > >>> >> > > >>> >> Thanks, > > >>> >> > > >>> >> Colm. > > >>> >> > > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang < > > >>> [email protected]> > > >>> >> wrote: > > >>> >> > > >>> >> > There is a data preparation stage between data source(HDFS audit > > >>> log) > > >>> >> and > > >>> >> > Alert Engine. This stage is running in Storm and transform the > raw > > >>> HDFS > > >>> >> log > > >>> >> > into something which can be alerted. > > >>> >> > > > >>> >> > The input for data preparation is hdfs_audit_log_sandbox topic > and > > >>> >> output > > >>> >> > is > > >>> >> > hdfs_audit_log_enriched_sandbox. > > >>> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox > and > > >>> >> output > > >>> >> > is > > >>> >> > hdfs_audit_log_alert_sandbox. > > >>> >> > > > >>> >> > Seems in your case, the data preparation staging is not working. > > We > > >>> >> > probably need look at Storm console and figure out if that part > is > > >>> >> working. > > >>> >> > > > >>> >> > Thanks > > >>> >> > Edward > > >>> >> > > > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh < > > >>> >> [email protected]> > > >>> >> > wrote: > > >>> >> > > > >>> >> > > Hi Jayesh, > > >>> >> > > > > >>> >> > > Many thanks for your feedback! I was able to make a little > > further > > >>> >> > headway. > > >>> >> > > There are two configuration problems with the official docker > > >>> image: > > >>> >> > > > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and " > > >>> server.eagle.apache.org" > > >>> >> > (this > > >>> >> > > only occurs in the instructions for running the docker image. > > The > > >>> >> version > > >>> >> > > that can be started via the script in the eagle source is OK). > > >>> I'll > > >>> >> > submit > > >>> >> > > a PR to fix this once I get a basic use-case working. > > >>> >> > > b) For the audit case, it automatically logs HDFS audit logs > to > > >>> the > > >>> >> KAFKA > > >>> >> > > topic sandbox_hdfs_audit_log instead of the expected > > >>> >> > hdfs_audit_log_sandbox > > >>> >> > > > > >>> >> > > I've fixed these things locally and I can verify that > everything > > >>> is > > >>> >> > started > > >>> >> > > correctly in Ambari. I log into the docker container and > create > > >>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox > > >>> topics, > > >>> >> and > > >>> >> > > verify that the HDFS audit logs are flowing into the first > > topic. > > >>> >> Then in > > >>> >> > > the UI I start the Alert Engine and then the HDFS Audit Log > > >>> Monitor > > >>> >> > > application (changing localhost:6667 to > > >>> server.eagle.apache.org:6667 > > >>> >> ). > > >>> >> > > Both > > >>> >> > > applications start up correctly and show "running". > > >>> >> > > > > >>> >> > > I then create a policy with an email alert along the lines of > > from > > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/h > > >>> base')] > > >>> >> > select > > >>> >> > > * group by user insert into hdfs_audit_log_enriched_ > > stream_out". > > >>> >> However > > >>> >> > > at > > >>> >> > > this point I'm stuck - nothing appears in the alert window. Is > > >>> there > > >>> >> > > anything obvious I'm doing wrong, or how can I get access to > > logs > > >>> to > > >>> >> > figure > > >>> >> > > out what the problem is? Other topics such as > > >>> >> "hdfs_audit_event_sandbox" > > >>> >> > > are mentioned in the streams window, but the documentation > > doesn't > > >>> >> say to > > >>> >> > > create them. > > >>> >> > > > > >>> >> > > The UI is buggy though on both Firefox and Chromium on Linux. > > What > > >>> >> > > browser/platform are people using with the UI? > > >>> >> > > > > >>> >> > > Colm. > > >>> >> > > > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya < > > >>> [email protected] > > >>> >> > > > >>> >> > > wrote: > > >>> >> > > > > >>> >> > > > Hi Colm, > > >>> >> > > > > > >>> >> > > > Please find my comments inline. > > >>> >> > > > > > >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the > > >>> >> released > > >>> >> > > > version. > > >>> >> > > > - this is because we uploaded docker image before apache > > >>> release. > > >>> >> > > actually > > >>> >> > > > this is same codebase apache-eagle-0.5, and it can be fixed > > >>> easily > > >>> >> by > > >>> >> > > just > > >>> >> > > > rebuilding docker image. there should not be any mismatch > due > > to > > >>> >> this. > > >>> >> > > > > > >>> >> > > > b) Aside from the above, the official docker image uses a > mix > > >>> of " > > >>> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as > > the > > >>> host > > >>> >> > > name. > > >>> >> > > > The HBase service doesn't start by default in Ambari as a > > >>> result. > > >>> >> > > > - the only places it uses sandbox is in example script which > > you > > >>> >> will > > >>> >> > > have > > >>> >> > > > to update anyway, which i agree that it would be good to > keep > > it > > >>> >> > > > consistent. > > >>> >> > > > > > >>> >> > > > c) The UI seems quite buggy. On both chromium and firefox, I > > >>> only > > >>> >> see > > >>> >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I > > >>> click > > >>> >> on > > >>> >> > > > "Alert" I have no way of going back to see the > applications. I > > >>> don't > > >>> >> > see > > >>> >> > > > the links to "integration" or "sites" as in the picture > here: > > >>> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx- > monito > > >>> ring > > >>> >> > > > - when hbase is as deep storage is used, and if eagle app > has > > >>> issue > > >>> >> > > > connecting to hbase, the UI becomes unresponsive. > > >>> >> > > > > > >>> >> > > > d) In chromium, the button to create a new policy does not > > >>> exist - I > > >>> >> > can > > >>> >> > > > only see it on Firefox. > > >>> >> > > > - i have seen when you logged in, you will see admin > actions. > > >>> but if > > >>> >> > this > > >>> >> > > > still an issue, can you please file UI bug? > > >>> >> > > > > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case > > >>> working, > > >>> >> but > > >>> >> > > it > > >>> >> > > > seems to be stuck in "Initialized". > > >>> >> > > > this eagle docs has example on how to setup the app. pls let > > us > > >>> >> know if > > >>> >> > > > you find any gaps. > > >>> >> > > > > > >>> >> > > > Thanks for trying out, and sharing your findings, > > >>> >> > > > Jayesh > > >>> >> > > > > > >>> >> > > > > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh < > > >>> >> > > [email protected]> > > >>> >> > > > wrote: > > >>> >> > > > > > >>> >> > > >> Hi all, > > >>> >> > > >> > > >>> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to > no > > >>> >> avail. > > >>> >> > > Here > > >>> >> > > >> are the problems I've run into so far: > > >>> >> > > >> > > >>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not > the > > >>> >> released > > >>> >> > > >> version. > > >>> >> > > >> > > >>> >> > > >> b) Aside from the above, the official docker image uses a > mix > > >>> of " > > >>> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as > > the > > >>> >> host > > >>> >> > > >> name. The HBase service doesn't start by default in Ambari > > as a > > >>> >> > result. > > >>> >> > > >> > > >>> >> > > >> c) The UI seems quite buggy. On both chromium and firefox, > I > > >>> only > > >>> >> see > > >>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once > I > > >>> click > > >>> >> on > > >>> >> > > >> "Alert" I have no way of going back to see the > applications. > > I > > >>> >> don't > > >>> >> > see > > >>> >> > > >> the links to "integration" or "sites" as in the picture > here: > > >>> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx- > monito > > >>> ring > > >>> >> > > >> > > >>> >> > > >> d) In chromium, the button to create a new policy does not > > >>> exist - > > >>> >> I > > >>> >> > can > > >>> >> > > >> only see it on Firefox. > > >>> >> > > >> > > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case > > >>> working, > > >>> >> > but > > >>> >> > > >> it seems to be stuck in "Initialized". > > >>> >> > > >> > > >>> >> > > >> Could someone fill me in on what the "recommended" way is > to > > >>> start > > >>> >> > > Apache > > >>> >> > > >> Eagle so that I can play around with the functionality that > > it > > >>> >> offers? > > >>> >> > > >> Clearly the docker approach is buggy. Also, what browser > > >>> should be > > >>> >> > used? > > >>> >> > > >> > > >>> >> > > >> Thanks, > > >>> >> > > >> > > >>> >> > > >> Colm. > > >>> >> > > >> > > >>> >> > > >> > > >>> >> > > >> -- > > >>> >> > > >> Colm O hEigeartaigh > > >>> >> > > >> > > >>> >> > > >> Talend Community Coder > > >>> >> > > >> http://coders.talend.com > > >>> >> > > >> > > >>> >> > > > > > >>> >> > > > > > >>> >> > > > > >>> >> > > > > >>> >> > > -- > > >>> >> > > Colm O hEigeartaigh > > >>> >> > > > > >>> >> > > Talend Community Coder > > >>> >> > > http://coders.talend.com > > >>> >> > > > > >>> >> > > > >>> >> > > >>> >> > > >>> >> > > >>> >> -- > > >>> >> Colm O hEigeartaigh > > >>> >> > > >>> >> Talend Community Coder > > >>> >> http://coders.talend.com > > >>> >> > > >>> >> > > >>> >> > > >>> > > > >>> > > > >>> > -- > > >>> > Colm O hEigeartaigh > > >>> > > > >>> > Talend Community Coder > > >>> > http://coders.talend.com > > >>> > > > >>> > > >>> > > >>> > > >>> -- > > >>> Colm O hEigeartaigh > > >>> > > >>> Talend Community Coder > > >>> http://coders.talend.com > > >>> > > >> > > >> > > > > > > > > > -- > > Colm O hEigeartaigh > > > > Talend Community Coder > > http://coders.talend.com > > > -- Colm O hEigeartaigh Talend Community Coder http://coders.talend.com
