Thanks again for your feedback. Jayesh, adding AlertEagleStorePlugin did the trick, I can now see alerts in the UI, thanks! By the way, I can't configure two Alert Publishers, or else the Alert DeDuplicator bins the alert. Is this a known issue?
Could I ask which browser people are using with the UI? There appears to be a bug with Chromium where it doesn't list the pages under Auth.isAdmin even though I am logged on as an administrator. It works OK in Firefox. Even with Firefox though, I only see a limited number of links in the left-hand column - I can't get back to the "integration" page. Can someone else confirm this please? Could I suggest the devs do some basic house-keeping tasks: a) "Release" version 0.5.0 in JIRA (it's still listed as "unreleased"). b) Figure out whether the next version will be 0.5.1 or 0.6.0 and update the versions on Master accordingly with 0.5.1-SNAPSHOT or 0.6.0-SNAPSHOT. There are some issues marked here as resolved for 0.5.1 - https://issues.apache.org/jira/projects/EAGLE/versions/12341128), however I don't see a branch for 0.5.x? Colm. On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <jay...@apache.org> wrote: > Hi, > > we do use eagle 0.5 in production although we dont use all the available > hadoop applications. > > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a fix for > email issue we found while our testing. should be merged soon after a > rebase. > > @Colm, did you tried adding storage publisher (AlertEagleStorePlugin)? to > see alerts on UI ? > > Thanks > Jayesh > > > > > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <yonzhang2...@gmail.com> > wrote: > >> Eagle 0.5 was deployed in production as far as I know, but it may not be >> exact the current version in master branch. >> >> Thanks for your investigation, seems there is still some bug in 0.5, but >> this particular issue seems is due to dependent components version conflict. >> >> @Jayesh is this Jira ready for merge to master? https://issues.apache. >> org/jira/browse/EAGLE-968 >> >> >> Thanks >> Edward >> >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <cohei...@apache.org >> > wrote: >> >>> OK I've made some more progress. I wasn't seeing any email alerts due to >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a >>> Kafka >>> alert, I can see the alerts flowing into my topic. It's still not clear >>> to >>> me however where the policy "output" is going. I also don't see any >>> alerts >>> in the UI window. >>> >>> Could I ask what the status of the project is in general? There have been >>> no commits to master since November, so I'm not sure if there is any >>> point >>> in submitting Pull Requests for outstanding bugs? Are recent versions of >>> Apache Eagle used in production? >>> >>> Colm. >>> >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh < >>> cohei...@apache.org> >>> wrote: >>> >>> > >>> > I've done that but I'm not seeing any alerts, which is why I want to >>> find >>> > out what the "output" of a policy is and where I can check this. >>> > >>> > Colm. >>> > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sjens...@gmail.com> >>> wrote: >>> > >>> >> Create and add a publisher to see the output. >>> >> >>> >> >>> >> >>> >> Regards, >>> >> Sudha jenslin >>> >> >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <cohei...@apache.org> >>> >> wrote: >>> >> >>> >> Thanks - the error was due to a problem running Storm with Java 1.8. >>> I've >>> >> abandoned the docker image for now, and I'm trying to get it working >>> >> locally. >>> >> >>> >> There are two things I'm not clear on currently, if someone could >>> fill me >>> >> in: >>> >> >>> >> a) For the 'Hdfs Audit Log Monitor' application, the Kafka Consumer >>> Topic >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event >>> Sink' >>> >> it >>> >> also specifies 'hdfs_audit_event_sandbox'. However the documentation >>> for >>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'? >>> >> >>> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA >>> M_SANDBOX" >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the >>> >> application, I can see cleansed log data appearing in >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary? >>> >> >>> >> b) It's unclear to me where the output data goes when you create a >>> policy. >>> >> E.g. say I have: >>> >> >>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hb >>> ase')] >>> >> select * group by user insert into hdfs_audit_log_enriched_stream_out >>> >> >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka >>> >> topic?). How can I check the output to make sure the policy is working >>> >> correctly? >>> >> >>> >> Thanks, >>> >> >>> >> Colm. >>> >> >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang < >>> yonzhang2...@gmail.com> >>> >> wrote: >>> >> >>> >> > There is a data preparation stage between data source(HDFS audit >>> log) >>> >> and >>> >> > Alert Engine. This stage is running in Storm and transform the raw >>> HDFS >>> >> log >>> >> > into something which can be alerted. >>> >> > >>> >> > The input for data preparation is hdfs_audit_log_sandbox topic and >>> >> output >>> >> > is >>> >> > hdfs_audit_log_enriched_sandbox. >>> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and >>> >> output >>> >> > is >>> >> > hdfs_audit_log_alert_sandbox. >>> >> > >>> >> > Seems in your case, the data preparation staging is not working. We >>> >> > probably need look at Storm console and figure out if that part is >>> >> working. >>> >> > >>> >> > Thanks >>> >> > Edward >>> >> > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh < >>> >> cohei...@apache.org> >>> >> > wrote: >>> >> > >>> >> > > Hi Jayesh, >>> >> > > >>> >> > > Many thanks for your feedback! I was able to make a little further >>> >> > headway. >>> >> > > There are two configuration problems with the official docker >>> image: >>> >> > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and " >>> server.eagle.apache.org" >>> >> > (this >>> >> > > only occurs in the instructions for running the docker image. The >>> >> version >>> >> > > that can be started via the script in the eagle source is OK). >>> I'll >>> >> > submit >>> >> > > a PR to fix this once I get a basic use-case working. >>> >> > > b) For the audit case, it automatically logs HDFS audit logs to >>> the >>> >> KAFKA >>> >> > > topic sandbox_hdfs_audit_log instead of the expected >>> >> > hdfs_audit_log_sandbox >>> >> > > >>> >> > > I've fixed these things locally and I can verify that everything >>> is >>> >> > started >>> >> > > correctly in Ambari. I log into the docker container and create >>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox >>> topics, >>> >> and >>> >> > > verify that the HDFS audit logs are flowing into the first topic. >>> >> Then in >>> >> > > the UI I start the Alert Engine and then the HDFS Audit Log >>> Monitor >>> >> > > application (changing localhost:6667 to >>> server.eagle.apache.org:6667 >>> >> ). >>> >> > > Both >>> >> > > applications start up correctly and show "running". >>> >> > > >>> >> > > I then create a policy with an email alert along the lines of from >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/h >>> base')] >>> >> > select >>> >> > > * group by user insert into hdfs_audit_log_enriched_stream_out". >>> >> However >>> >> > > at >>> >> > > this point I'm stuck - nothing appears in the alert window. Is >>> there >>> >> > > anything obvious I'm doing wrong, or how can I get access to logs >>> to >>> >> > figure >>> >> > > out what the problem is? Other topics such as >>> >> "hdfs_audit_event_sandbox" >>> >> > > are mentioned in the streams window, but the documentation doesn't >>> >> say to >>> >> > > create them. >>> >> > > >>> >> > > The UI is buggy though on both Firefox and Chromium on Linux. What >>> >> > > browser/platform are people using with the UI? >>> >> > > >>> >> > > Colm. >>> >> > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya < >>> jay...@apache.org >>> >> > >>> >> > > wrote: >>> >> > > >>> >> > > > Hi Colm, >>> >> > > > >>> >> > > > Please find my comments inline. >>> >> > > > >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the >>> >> released >>> >> > > > version. >>> >> > > > - this is because we uploaded docker image before apache >>> release. >>> >> > > actually >>> >> > > > this is same codebase apache-eagle-0.5, and it can be fixed >>> easily >>> >> by >>> >> > > just >>> >> > > > rebuilding docker image. there should not be any mismatch due to >>> >> this. >>> >> > > > >>> >> > > > b) Aside from the above, the official docker image uses a mix >>> of " >>> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the >>> host >>> >> > > name. >>> >> > > > The HBase service doesn't start by default in Ambari as a >>> result. >>> >> > > > - the only places it uses sandbox is in example script which you >>> >> will >>> >> > > have >>> >> > > > to update anyway, which i agree that it would be good to keep it >>> >> > > > consistent. >>> >> > > > >>> >> > > > c) The UI seems quite buggy. On both chromium and firefox, I >>> only >>> >> see >>> >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I >>> click >>> >> on >>> >> > > > "Alert" I have no way of going back to see the applications. I >>> don't >>> >> > see >>> >> > > > the links to "integration" or "sites" as in the picture here: >>> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monito >>> ring >>> >> > > > - when hbase is as deep storage is used, and if eagle app has >>> issue >>> >> > > > connecting to hbase, the UI becomes unresponsive. >>> >> > > > >>> >> > > > d) In chromium, the button to create a new policy does not >>> exist - I >>> >> > can >>> >> > > > only see it on Firefox. >>> >> > > > - i have seen when you logged in, you will see admin actions. >>> but if >>> >> > this >>> >> > > > still an issue, can you please file UI bug? >>> >> > > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case >>> working, >>> >> but >>> >> > > it >>> >> > > > seems to be stuck in "Initialized". >>> >> > > > this eagle docs has example on how to setup the app. pls let us >>> >> know if >>> >> > > > you find any gaps. >>> >> > > > >>> >> > > > Thanks for trying out, and sharing your findings, >>> >> > > > Jayesh >>> >> > > > >>> >> > > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh < >>> >> > > cohei...@apache.org> >>> >> > > > wrote: >>> >> > > > >>> >> > > >> Hi all, >>> >> > > >> >>> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no >>> >> avail. >>> >> > > Here >>> >> > > >> are the problems I've run into so far: >>> >> > > >> >>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the >>> >> released >>> >> > > >> version. >>> >> > > >> >>> >> > > >> b) Aside from the above, the official docker image uses a mix >>> of " >>> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the >>> >> host >>> >> > > >> name. The HBase service doesn't start by default in Ambari as a >>> >> > result. >>> >> > > >> >>> >> > > >> c) The UI seems quite buggy. On both chromium and firefox, I >>> only >>> >> see >>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I >>> click >>> >> on >>> >> > > >> "Alert" I have no way of going back to see the applications. I >>> >> don't >>> >> > see >>> >> > > >> the links to "integration" or "sites" as in the picture here: >>> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monito >>> ring >>> >> > > >> >>> >> > > >> d) In chromium, the button to create a new policy does not >>> exist - >>> >> I >>> >> > can >>> >> > > >> only see it on Firefox. >>> >> > > >> >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case >>> working, >>> >> > but >>> >> > > >> it seems to be stuck in "Initialized". >>> >> > > >> >>> >> > > >> Could someone fill me in on what the "recommended" way is to >>> start >>> >> > > Apache >>> >> > > >> Eagle so that I can play around with the functionality that it >>> >> offers? >>> >> > > >> Clearly the docker approach is buggy. Also, what browser >>> should be >>> >> > used? >>> >> > > >> >>> >> > > >> Thanks, >>> >> > > >> >>> >> > > >> Colm. >>> >> > > >> >>> >> > > >> >>> >> > > >> -- >>> >> > > >> Colm O hEigeartaigh >>> >> > > >> >>> >> > > >> Talend Community Coder >>> >> > > >> http://coders.talend.com >>> >> > > >> >>> >> > > > >>> >> > > > >>> >> > > >>> >> > > >>> >> > > -- >>> >> > > Colm O hEigeartaigh >>> >> > > >>> >> > > Talend Community Coder >>> >> > > http://coders.talend.com >>> >> > > >>> >> > >>> >> >>> >> >>> >> >>> >> -- >>> >> Colm O hEigeartaigh >>> >> >>> >> Talend Community Coder >>> >> http://coders.talend.com >>> >> >>> >> >>> >> >>> > >>> > >>> > -- >>> > Colm O hEigeartaigh >>> > >>> > Talend Community Coder >>> > http://coders.talend.com >>> > >>> >>> >>> >>> -- >>> Colm O hEigeartaigh >>> >>> Talend Community Coder >>> http://coders.talend.com >>> >> >> > -- Colm O hEigeartaigh Talend Community Coder http://coders.talend.com
