OK cool. I have submitted a few other PRs: https://github.com/apache/eagle/pull/986 https://github.com/apache/eagle/pull/983 https://github.com/apache/eagle/pull/985
The first one is a fix for the issue reported on the dev list recently, where you can't start the eagle-server.sh script from the same directory. I changed it to get the actual directory name using "readlink -f". I tested it works OK when run via "./eagle-server.sh start" and "bin/eagle-server.sh start". The other two are checkstyle fixes for two different modules. Colm. On Fri, Feb 2, 2018 at 5:25 PM, Jayesh Senjaliya <[email protected]> wrote: > Ya slackpublisher has some issue and we have fixed it and also changed the > implementation to use rest api instad of sdk. > We will push the patch soon. > > Thanks > Jayesh > > On Fri, Feb 2, 2018 at 9:02 AM Colm O hEigeartaigh <[email protected]> > wrote: > > > Thanks. I've submitted an initial PR to fix the Slack > > ClassNotFoundException issue here: > > > > https://issues.apache.org/jira/browse/EAGLE-879 > > https://github.com/apache/eagle/pull/984 > > > > However, no actual Slack messages are sent. The problem is that > > AlertSlackPublisher only sends the message if the event contains a > > "severity" that matches the configured "severity" on the publisher. > However > > the event contains neither "severity" (or "message") so the message never > > gets sent. This is for an HDFS audit log - I'm not sure if there are > other > > scenarios where there is a "severity" column in the event? Either that or > > it looks like the SlackPublisher was written before the event column > > headers were changed. > > > > Colm. > > > > On Fri, Feb 2, 2018 at 12:54 AM, Jayesh Senjaliya <[email protected]> > > wrote: > > > >> resolved those tickets now. > >> > >> I have asked the developer to rebase the PR #941, if he doesnt get to it > >> by > >> this week, i will take care of, its long pending one. > >> Thanks for verifying though. > >> > >> - Jayesh > >> > >> On Thu, Feb 1, 2018 at 8:56 AM, Colm O hEigeartaigh < > [email protected]> > >> wrote: > >> > >> > Thanks Jayesh. I have two more PRs awaiting review: > >> > > >> > https://github.com/apache/eagle/pull/981 > >> > https://github.com/apache/eagle/pull/982 > >> > > >> > Thanks for the JIRA privileges, I can now assign issues to me + change > >> the > >> > versions. However, I can't "resolve" JIRAs that weren't reported by me > >> > which is annoying. These 3 JIRAs should be resolved as they are > already > >> > merged: > >> > > >> > https://issues.apache.org/jira/browse/EAGLE-445 > >> > https://issues.apache.org/jira/browse/EAGLE-476 > >> > https://issues.apache.org/jira/browse/EAGLE-331 > >> > > >> > In addition, I tested the fix for the Email issue and it works > >> correctly. > >> > The PR (https://github.com/apache/eagle/pull/941) just needs to have > >> the > >> > extra commits stripped away - I attached a version of the patch on the > >> > JIRA. > >> > > >> > Colm. > >> > > >> > On Wed, Jan 31, 2018 at 10:08 PM, Jayesh Senjaliya <[email protected] > > > >> > wrote: > >> > > >> > > Thanks for the PRs. I have merged them. > >> > > > >> > > welcome to the developer community Colm. I have also added you to > jira > >> > > project so can assign the tasks to yourself. > >> > > > >> > > lets create ticket to fix the dedup functionality, I m actually > >> surprised > >> > > we havent hit this issue yet. we do use multiple publishers but > >> someone > >> > can > >> > > verify this. > >> > > > >> > > Thanks > >> > > Jayesh > >> > > > >> > > > >> > > > >> > > On Wed, Jan 31, 2018 at 9:25 AM, Colm O hEigeartaigh < > >> > [email protected]> > >> > > wrote: > >> > > > >> > >> Thanks Jayesh. I've created a JIRA here for some admin work for > some > >> > >> issues > >> > >> that were incorrectly flagged as "fix for" 0.5.1/0.6.0: > >> > >> > >> > >> https://issues.apache.org/jira/browse/EAGLE-1076 > >> > >> > >> > >> I've submitted the following (fairly trivial) pull requests. Could > I > >> ask > >> > >> that you or one of the other committers review? > >> > >> > >> > >> https://github.com/apache/eagle/pull/978 > >> > >> https://github.com/apache/eagle/pull/979 > >> > >> https://github.com/apache/eagle/pull/980 > >> > >> > >> > >> It would be good to try to inject some energy into the project. We > >> need > >> > >> more than one active committer though. > >> > >> > >> > >> Just in terms of the Alert Deduplication issue. The > >> DefaultDeDuplicator > >> > >> works per "output" in the policy rule. So if you have more than one > >> > >> AlertPublisher, I think it is guaranteed to only publish to one of > >> them. > >> > >> Instead, surely it would make more sense to work per publisher? > >> > >> > >> > >> Colm. > >> > >> > >> > >> On Tue, Jan 30, 2018 at 10:39 PM, Jayesh Senjaliya < > >> [email protected]> > >> > >> wrote: > >> > >> > >> > >> > Hi Colm, > >> > >> > > >> > >> > appreciate your suggestions/ efforts in looking into this > project, > >> > >> > putting my comments inline... > >> > >> > > >> > >> > a) There is already a JIRA to bump the version here, although the > >> PR > >> > >> does > >> > >> > not apply as it is too old: https://issues.apache.org/ > >> > >> > jira/browse/EAGLE-1025 > >> > >> > . > >> > >> > I can submit a new PR, but should the version be 0.6.0 or 0.5.1? > >> > >> > > >> > >> > *since there are still minor issues, i would say, we put up 0.5.1 > >> as > >> > >> next > >> > >> > version. I've updated/rebased the PR ( > >> > >> > https://github.com/apache/eagle/pull/936 > >> > >> > <https://github.com/apache/eagle/pull/936> )* > >> > >> > > >> > >> > > >> > >> > b) The issues that are "resolved" for the 0.5.1 release in JIRA > are > >> > >> > actually already fixed in 0.5.0, so they should be updated ( > >> > >> > https://issues.apache.org/jira/projects/EAGLE/versions/12341128 > ). > >> > >> However, > >> > >> > the following two issues are resolved even though they are not > >> merged > >> > to > >> > >> > master? > >> > >> > https://issues.apache.org/jira/browse/EAGLE-1051 . - * this > was > >> > >> pending > >> > >> > from developer;s response but i think this is reviewed, so I have > >> > merged > >> > >> > it.* > >> > >> > https://issues.apache.org/jira/browse/EAGLE-1068 . - * this > is > >> > >> reopened > >> > >> > now. I dont think this is done yet. Also this is big change.* > >> > >> > > >> > >> > > >> > >> > Like I said I can submit PRs but I'm not convinced there is any > >> > >> activity on > >> > >> > the project. Where are the rest of the committers? > >> > >> > > >> > >> > *let me give you some ocontext on this. so there were lot of > >> > development > >> > >> > happened during last releases, and most of applications that were > >> > added > >> > >> are > >> > >> > being used in production at multiple enterprise companies, but we > >> are > >> > >> out > >> > >> > of ideas on new apps, so at this point we are only focusing on > bug > >> > fixes > >> > >> > and tech upgrades until we get some new ideas to brainstorm and > >> add.* > >> > >> > > >> > >> > *I think current community's thinking is based on their own > >> industries > >> > >> > use-cases, but there is definitely room for new features and > >> > integration > >> > >> > with other monitoring and security components like grafana and > >> > rangers.* > >> > >> > > >> > >> > > >> > >> > *Thanks,* > >> > >> > *Jayesh* > >> > >> > > >> > >> > > >> > >> > > >> > >> > On Tue, Jan 30, 2018 at 8:11 AM, Colm O hEigeartaigh < > >> > >> [email protected]> > >> > >> > wrote: > >> > >> > > >> > >> > > Hi Jayesh, > >> > >> > > > >> > >> > > Dev suggestions: > >> > >> > > > >> > >> > > a) There is already a JIRA to bump the version here, although > >> the PR > >> > >> does > >> > >> > > not apply as it is too old: https://issues.apache.org/ > >> > >> > > jira/browse/EAGLE-1025. > >> > >> > > I can submit a new PR, but should the version be 0.6.0 or > 0.5.1? > >> > >> > > b) The issues that are "resolved" for the 0.5.1 release in JIRA > >> are > >> > >> > > actually already fixed in 0.5.0, so they should be updated ( > >> > >> > > https://issues.apache.org/jira/projects/EAGLE/versions/ > 12341128 > >> ). > >> > >> > However, > >> > >> > > the following two issues are resolved even though they are not > >> > merged > >> > >> to > >> > >> > > master? > >> > >> > > https://issues.apache.org/jira/browse/EAGLE-1051 > >> > >> > > https://issues.apache.org/jira/browse/EAGLE-1068 > >> > >> > > > >> > >> > > Like I said I can submit PRs but I'm not convinced there is any > >> > >> activity > >> > >> > on > >> > >> > > the project. Where are the rest of the committers? > >> > >> > > > >> > >> > > Multiple Publisher issue: > >> > >> > > > >> > >> > > If I assign two publishers for one policy, the alert only goes > to > >> > the > >> > >> > first > >> > >> > > policy. In the logs I see: > >> > >> > > > >> > >> > > 2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator > >> > [INFO] > >> > >> > > Alert > >> > >> > > event is skipped because it's duplicated: Alert {site=sandbox, > >> > >> > > stream=eagle_output,timestamp=2018-01-30 > >> > >> > > 00:00:11,300,data={securityZone=NA, dst=null, > sensitivityType=NA, > >> > >> > > src=/apps/hbase/data/archive/data/default/ambarismoketest, > >> > >> allowed=true, > >> > >> > > host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM, > >> > >> > > timestamp=1517270411300}, policyId=test, > >> > >> > > createdBy=alertBolt3-evaluator_stage1, metaVersion=null} > >> > >> > > > >> > >> > > It looks like this deduplicator is not working properly, as I'm > >> > >> guessing > >> > >> > it > >> > >> > > should only be used to de-duplicate events for a single > >> publisher? > >> > >> > > > >> > >> > > Incognito mode: Already tried it but with the same result. > Could > >> I > >> > ask > >> > >> > you > >> > >> > > to try the docker image to see if the UI is working correctly > for > >> > you > >> > >> > > there? > >> > >> > > > >> > >> > > Colm. > >> > >> > > > >> > >> > > On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya < > >> > [email protected]> > >> > >> > > wrote: > >> > >> > > > >> > >> > > > Hi Colm, > >> > >> > > > > >> > >> > > > Thanks for the list of dev suggestions, I think we should > take > >> > care > >> > >> of > >> > >> > > > those. even better if you can provide PR with the changes or > at > >> > >> keast > >> > >> > can > >> > >> > > > you please create a ticket so we can track it? > >> > >> > > > > >> > >> > > > for other issues. > >> > >> > > > > >> > >> > > > - I dont have any issue with multiple publisher, but if there > >> is > >> > any > >> > >> > > error > >> > >> > > > updating the publisher info in storm topology, i might try > >> > >> restarting > >> > >> > the > >> > >> > > > topology and see if that works. > >> > >> > > > - for us, chrome works as fine as firefox. can u try > incognito > >> > >> mode? > >> > >> > > just > >> > >> > > > to be sure to have clean cache? > >> > >> > > > > >> > >> > > > Thanks > >> > >> > > > Jayesh > >> > >> > > > > >> > >> > > > > >> > >> > > > On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh < > >> > >> > > [email protected]> > >> > >> > > > wrote: > >> > >> > > > > >> > >> > > > > Thanks again for your feedback. Jayesh, adding > >> > >> AlertEagleStorePlugin > >> > >> > > did > >> > >> > > > > the trick, I can now see alerts in the UI, thanks! By the > >> way, I > >> > >> > can't > >> > >> > > > > configure two Alert Publishers, or else the Alert > >> DeDuplicator > >> > >> bins > >> > >> > the > >> > >> > > > > alert. Is this a known issue? > >> > >> > > > > > >> > >> > > > > Could I ask which browser people are using with the UI? > There > >> > >> appears > >> > >> > > to > >> > >> > > > be > >> > >> > > > > a bug with Chromium where it doesn't list the pages under > >> > >> > Auth.isAdmin > >> > >> > > > > even though I am logged on as an administrator. It works OK > >> in > >> > >> > Firefox. > >> > >> > > > > Even with Firefox though, I only see a limited number of > >> links > >> > in > >> > >> the > >> > >> > > > > left-hand column - I can't get back to the "integration" > >> page. > >> > Can > >> > >> > > > someone > >> > >> > > > > else confirm this please? > >> > >> > > > > > >> > >> > > > > Could I suggest the devs do some basic house-keeping tasks: > >> > >> > > > > > >> > >> > > > > a) "Release" version 0.5.0 in JIRA (it's still listed as > >> > >> > "unreleased"). > >> > >> > > > > b) Figure out whether the next version will be 0.5.1 or > 0.6.0 > >> > and > >> > >> > > update > >> > >> > > > > the versions on Master accordingly with 0.5.1-SNAPSHOT or > >> > >> > > 0.6.0-SNAPSHOT. > >> > >> > > > > There are some issues marked here as resolved for 0.5.1 - > >> > >> > > > > > >> https://issues.apache.org/jira/projects/EAGLE/versions/12341128 > >> > ), > >> > >> > > > however > >> > >> > > > > I > >> > >> > > > > don't see a branch for 0.5.x? > >> > >> > > > > > >> > >> > > > > Colm. > >> > >> > > > > > >> > >> > > > > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya < > >> > >> [email protected] > >> > >> > > > >> > >> > > > > wrote: > >> > >> > > > > > >> > >> > > > > > Hi, > >> > >> > > > > > > >> > >> > > > > > we do use eagle 0.5 in production although we dont use > all > >> the > >> > >> > > > available > >> > >> > > > > > hadoop applications. > >> > >> > > > > > > >> > >> > > > > > EAGLE-968 <https://issues.apache.org/ > jira/browse/EAGLE-968 > >> > > >> > is > >> > >> a > >> > >> > fix > >> > >> > > > for > >> > >> > > > > > email issue we found while our testing. should be merged > >> soon > >> > >> > after a > >> > >> > > > > > rebase. > >> > >> > > > > > > >> > >> > > > > > @Colm, did you tried adding storage publisher > >> > >> > > (AlertEagleStorePlugin)? > >> > >> > > > to > >> > >> > > > > > see alerts on UI ? > >> > >> > > > > > > >> > >> > > > > > Thanks > >> > >> > > > > > Jayesh > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang < > >> > >> > > [email protected]> > >> > >> > > > > > wrote: > >> > >> > > > > > > >> > >> > > > > >> Eagle 0.5 was deployed in production as far as I know, > >> but it > >> > >> may > >> > >> > > not > >> > >> > > > be > >> > >> > > > > >> exact the current version in master branch. > >> > >> > > > > >> > >> > >> > > > > >> Thanks for your investigation, seems there is still some > >> bug > >> > in > >> > >> > 0.5, > >> > >> > > > but > >> > >> > > > > >> this particular issue seems is due to dependent > components > >> > >> version > >> > >> > > > > conflict. > >> > >> > > > > >> > >> > >> > > > > >> @Jayesh is this Jira ready for merge to master? > >> > >> > > https://issues.apache > >> > >> > > > . > >> > >> > > > > >> org/jira/browse/EAGLE-968 > >> > >> > > > > >> > >> > >> > > > > >> > >> > >> > > > > >> Thanks > >> > >> > > > > >> Edward > >> > >> > > > > >> > >> > >> > > > > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh < > >> > >> > > > > [email protected] > >> > >> > > > > >> > wrote: > >> > >> > > > > >> > >> > >> > > > > >>> OK I've made some more progress. I wasn't seeing any > >> email > >> > >> alerts > >> > >> > > due > >> > >> > > > > to > >> > >> > > > > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once > I > >> > >> > configure > >> > >> > > a > >> > >> > > > > >>> Kafka > >> > >> > > > > >>> alert, I can see the alerts flowing into my topic. It's > >> > still > >> > >> not > >> > >> > > > clear > >> > >> > > > > >>> to > >> > >> > > > > >>> me however where the policy "output" is going. I also > >> don't > >> > >> see > >> > >> > any > >> > >> > > > > >>> alerts > >> > >> > > > > >>> in the UI window. > >> > >> > > > > >>> > >> > >> > > > > >>> Could I ask what the status of the project is in > general? > >> > >> There > >> > >> > > have > >> > >> > > > > been > >> > >> > > > > >>> no commits to master since November, so I'm not sure if > >> > there > >> > >> is > >> > >> > > any > >> > >> > > > > >>> point > >> > >> > > > > >>> in submitting Pull Requests for outstanding bugs? Are > >> recent > >> > >> > > versions > >> > >> > > > > of > >> > >> > > > > >>> Apache Eagle used in production? > >> > >> > > > > >>> > >> > >> > > > > >>> Colm. > >> > >> > > > > >>> > >> > >> > > > > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh < > >> > >> > > > > >>> [email protected]> > >> > >> > > > > >>> wrote: > >> > >> > > > > >>> > >> > >> > > > > >>> > > >> > >> > > > > >>> > I've done that but I'm not seeing any alerts, which > is > >> > why I > >> > >> > want > >> > >> > > > to > >> > >> > > > > >>> find > >> > >> > > > > >>> > out what the "output" of a policy is and where I can > >> check > >> > >> > this. > >> > >> > > > > >>> > > >> > >> > > > > >>> > Colm. > >> > >> > > > > >>> > > >> > >> > > > > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN < > >> > >> > > [email protected] > >> > >> > > > > > >> > >> > > > > >>> wrote: > >> > >> > > > > >>> > > >> > >> > > > > >>> >> Create and add a publisher to see the output. > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> Regards, > >> > >> > > > > >>> >> Sudha jenslin > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" < > >> > >> > > > [email protected] > >> > >> > > > > > > >> > >> > > > > >>> >> wrote: > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> Thanks - the error was due to a problem running > Storm > >> > with > >> > >> > Java > >> > >> > > > 1.8. > >> > >> > > > > >>> I've > >> > >> > > > > >>> >> abandoned the docker image for now, and I'm trying > to > >> get > >> > >> it > >> > >> > > > working > >> > >> > > > > >>> >> locally. > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> There are two things I'm not clear on currently, if > >> > someone > >> > >> > > could > >> > >> > > > > >>> fill me > >> > >> > > > > >>> >> in: > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> a) For the 'Hdfs Audit Log Monitor' application, > the > >> > Kafka > >> > >> > > > Consumer > >> > >> > > > > >>> Topic > >> > >> > > > > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for > >> > >> Auditlog > >> > >> > > Event > >> > >> > > > > >>> Sink' > >> > >> > > > > >>> >> it > >> > >> > > > > >>> >> also specifies 'hdfs_audit_event_sandbox'. However > the > >> > >> > > > documentation > >> > >> > > > > >>> for > >> > >> > > > > >>> >> the application mentions > >> 'hdfs_audit_log_enriched_sandb > >> > >> ox'? > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> When I click on "STREAMS", the > >> > >> "HDFS_AUDIT_LOG_ENRICHED_STREA > >> > >> > > > > >>> M_SANDBOX" > >> > >> > > > > >>> >> uses the topic "hdfs_audit_event_sandbox". And > indeed > >> > when > >> > >> I > >> > >> > run > >> > >> > > > the > >> > >> > > > > >>> >> application, I can see cleansed log data appearing > in > >> > >> > > > > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here > that > >> > >> > > > > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or > >> > >> > necessary? > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> b) It's unclear to me where the output data goes > when > >> you > >> > >> > > create a > >> > >> > > > > >>> policy. > >> > >> > > > > >>> >> E.g. say I have: > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> from HDFS_AUDIT_LOG_ENRICHED_ > >> > STREAM_SANDBOX[str:contains( > >> > >> > > src,'/hb > >> > >> > > > > >>> ase')] > >> > >> > > > > >>> >> select * group by user insert into > >> > hdfs_audit_log_enriched_ > >> > >> > > > > stream_out > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> Where is "hdfs_audit_log_enriched_stream_out" > defined > >> > (is > >> > >> it > >> > >> > a > >> > >> > > > > Kafka > >> > >> > > > > >>> >> topic?). How can I check the output to make sure the > >> > >> policy is > >> > >> > > > > working > >> > >> > > > > >>> >> correctly? > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> Thanks, > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> Colm. > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang < > >> > >> > > > > >>> [email protected]> > >> > >> > > > > >>> >> wrote: > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> > There is a data preparation stage between data > >> > >> source(HDFS > >> > >> > > audit > >> > >> > > > > >>> log) > >> > >> > > > > >>> >> and > >> > >> > > > > >>> >> > Alert Engine. This stage is running in Storm and > >> > >> transform > >> > >> > the > >> > >> > > > raw > >> > >> > > > > >>> HDFS > >> > >> > > > > >>> >> log > >> > >> > > > > >>> >> > into something which can be alerted. > >> > >> > > > > >>> >> > > >> > >> > > > > >>> >> > The input for data preparation is > >> > hdfs_audit_log_sandbox > >> > >> > topic > >> > >> > > > and > >> > >> > > > > >>> >> output > >> > >> > > > > >>> >> > is > >> > >> > > > > >>> >> > hdfs_audit_log_enriched_sandbox. > >> > >> > > > > >>> >> > The input for Alert Engine is > >> hdfs_audit_log_enriched_ > >> > >> > sandbox > >> > >> > > > and > >> > >> > > > > >>> >> output > >> > >> > > > > >>> >> > is > >> > >> > > > > >>> >> > hdfs_audit_log_alert_sandbox. > >> > >> > > > > >>> >> > > >> > >> > > > > >>> >> > Seems in your case, the data preparation staging > is > >> not > >> > >> > > working. > >> > >> > > > > We > >> > >> > > > > >>> >> > probably need look at Storm console and figure out > >> if > >> > >> that > >> > >> > > part > >> > >> > > > is > >> > >> > > > > >>> >> working. > >> > >> > > > > >>> >> > > >> > >> > > > > >>> >> > Thanks > >> > >> > > > > >>> >> > Edward > >> > >> > > > > >>> >> > > >> > >> > > > > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O > >> hEigeartaigh < > >> > >> > > > > >>> >> [email protected]> > >> > >> > > > > >>> >> > wrote: > >> > >> > > > > >>> >> > > >> > >> > > > > >>> >> > > Hi Jayesh, > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > Many thanks for your feedback! I was able to > make > >> a > >> > >> little > >> > >> > > > > further > >> > >> > > > > >>> >> > headway. > >> > >> > > > > >>> >> > > There are two configuration problems with the > >> > official > >> > >> > > docker > >> > >> > > > > >>> image: > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and " > >> > >> > > > > >>> server.eagle.apache.org" > >> > >> > > > > >>> >> > (this > >> > >> > > > > >>> >> > > only occurs in the instructions for running the > >> > docker > >> > >> > > image. > >> > >> > > > > The > >> > >> > > > > >>> >> version > >> > >> > > > > >>> >> > > that can be started via the script in the eagle > >> > source > >> > >> is > >> > >> > > OK). > >> > >> > > > > >>> I'll > >> > >> > > > > >>> >> > submit > >> > >> > > > > >>> >> > > a PR to fix this once I get a basic use-case > >> working. > >> > >> > > > > >>> >> > > b) For the audit case, it automatically logs > HDFS > >> > audit > >> > >> > logs > >> > >> > > > to > >> > >> > > > > >>> the > >> > >> > > > > >>> >> KAFKA > >> > >> > > > > >>> >> > > topic sandbox_hdfs_audit_log instead of the > >> expected > >> > >> > > > > >>> >> > hdfs_audit_log_sandbox > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > I've fixed these things locally and I can verify > >> that > >> > >> > > > everything > >> > >> > > > > >>> is > >> > >> > > > > >>> >> > started > >> > >> > > > > >>> >> > > correctly in Ambari. I log into the docker > >> container > >> > >> and > >> > >> > > > create > >> > >> > > > > >>> >> > > hdfs_audit_log_sandbox and > >> hdfs_audit_log_enriched_ > >> > >> > sandbox > >> > >> > > > > >>> topics, > >> > >> > > > > >>> >> and > >> > >> > > > > >>> >> > > verify that the HDFS audit logs are flowing into > >> the > >> > >> first > >> > >> > > > > topic. > >> > >> > > > > >>> >> Then in > >> > >> > > > > >>> >> > > the UI I start the Alert Engine and then the > HDFS > >> > Audit > >> > >> > Log > >> > >> > > > > >>> Monitor > >> > >> > > > > >>> >> > > application (changing localhost:6667 to > >> > >> > > > > >>> server.eagle.apache.org:6667 > >> > >> > > > > >>> >> ). > >> > >> > > > > >>> >> > > Both > >> > >> > > > > >>> >> > > applications start up correctly and show > >> "running". > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > I then create a policy with an email alert along > >> the > >> > >> lines > >> > >> > > of > >> > >> > > > > from > >> > >> > > > > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_ > >> > STREAM_SANDBOX[str:contains( > >> > >> > > src,'/h > >> > >> > > > > >>> base')] > >> > >> > > > > >>> >> > select > >> > >> > > > > >>> >> > > * group by user insert into > >> hdfs_audit_log_enriched_ > >> > >> > > > > stream_out". > >> > >> > > > > >>> >> However > >> > >> > > > > >>> >> > > at > >> > >> > > > > >>> >> > > this point I'm stuck - nothing appears in the > >> alert > >> > >> > window. > >> > >> > > Is > >> > >> > > > > >>> there > >> > >> > > > > >>> >> > > anything obvious I'm doing wrong, or how can I > get > >> > >> access > >> > >> > to > >> > >> > > > > logs > >> > >> > > > > >>> to > >> > >> > > > > >>> >> > figure > >> > >> > > > > >>> >> > > out what the problem is? Other topics such as > >> > >> > > > > >>> >> "hdfs_audit_event_sandbox" > >> > >> > > > > >>> >> > > are mentioned in the streams window, but the > >> > >> documentation > >> > >> > > > > doesn't > >> > >> > > > > >>> >> say to > >> > >> > > > > >>> >> > > create them. > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > The UI is buggy though on both Firefox and > >> Chromium > >> > on > >> > >> > > Linux. > >> > >> > > > > What > >> > >> > > > > >>> >> > > browser/platform are people using with the UI? > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > Colm. > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh > >> Senjaliya < > >> > >> > > > > >>> [email protected] > >> > >> > > > > >>> >> > > >> > >> > > > > >>> >> > > wrote: > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > > Hi Colm, > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > > Please find my comments inline. > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > > a) The official docker image uses > 0.5.0-SNAPSHOT > >> > and > >> > >> not > >> > >> > > the > >> > >> > > > > >>> >> released > >> > >> > > > > >>> >> > > > version. > >> > >> > > > > >>> >> > > > - this is because we uploaded docker image > >> before > >> > >> apache > >> > >> > > > > >>> release. > >> > >> > > > > >>> >> > > actually > >> > >> > > > > >>> >> > > > this is same codebase apache-eagle-0.5, and it > >> can > >> > be > >> > >> > > fixed > >> > >> > > > > >>> easily > >> > >> > > > > >>> >> by > >> > >> > > > > >>> >> > > just > >> > >> > > > > >>> >> > > > rebuilding docker image. there should not be > any > >> > >> > mismatch > >> > >> > > > due > >> > >> > > > > to > >> > >> > > > > >>> >> this. > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > > b) Aside from the above, the official docker > >> image > >> > >> uses > >> > >> > a > >> > >> > > > mix > >> > >> > > > > >>> of " > >> > >> > > > > >>> >> > > > server.eagle.apache.org" and " > >> > >> sandbox.eagle.apache.org" > >> > >> > > as > >> > >> > > > > the > >> > >> > > > > >>> host > >> > >> > > > > >>> >> > > name. > >> > >> > > > > >>> >> > > > The HBase service doesn't start by default in > >> > Ambari > >> > >> as > >> > >> > a > >> > >> > > > > >>> result. > >> > >> > > > > >>> >> > > > - the only places it uses sandbox is in > example > >> > >> script > >> > >> > > which > >> > >> > > > > you > >> > >> > > > > >>> >> will > >> > >> > > > > >>> >> > > have > >> > >> > > > > >>> >> > > > to update anyway, which i agree that it would > be > >> > >> good to > >> > >> > > > keep > >> > >> > > > > it > >> > >> > > > > >>> >> > > > consistent. > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > > c) The UI seems quite buggy. On both chromium > >> and > >> > >> > > firefox, I > >> > >> > > > > >>> only > >> > >> > > > > >>> >> see > >> > >> > > > > >>> >> > > > links to "Sandbox" and "Alert" on the left > >> > hand-side. > >> > >> > > Once I > >> > >> > > > > >>> click > >> > >> > > > > >>> >> on > >> > >> > > > > >>> >> > > > "Alert" I have no way of going back to see the > >> > >> > > > applications. I > >> > >> > > > > >>> don't > >> > >> > > > > >>> >> > see > >> > >> > > > > >>> >> > > > the links to "integration" or "sites" as in > the > >> > >> picture > >> > >> > > > here: > >> > >> > > > > >>> >> > > > http://eagle.apache.org/docs/l > >> > >> atest/applications/#jmx- > >> > >> > > > monito > >> > >> > > > > >>> ring > >> > >> > > > > >>> >> > > > - when hbase is as deep storage is used, and > if > >> > eagle > >> > >> > app > >> > >> > > > has > >> > >> > > > > >>> issue > >> > >> > > > > >>> >> > > > connecting to hbase, the UI becomes > >> unresponsive. > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > > d) In chromium, the button to create a new > >> policy > >> > >> does > >> > >> > not > >> > >> > > > > >>> exist - I > >> > >> > > > > >>> >> > can > >> > >> > > > > >>> >> > > > only see it on Firefox. > >> > >> > > > > >>> >> > > > - i have seen when you logged in, you will see > >> > admin > >> > >> > > > actions. > >> > >> > > > > >>> but if > >> > >> > > > > >>> >> > this > >> > >> > > > > >>> >> > > > still an issue, can you please file UI bug? > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log > >> Monitor" > >> > >> > use-case > >> > >> > > > > >>> working, > >> > >> > > > > >>> >> but > >> > >> > > > > >>> >> > > it > >> > >> > > > > >>> >> > > > seems to be stuck in "Initialized". > >> > >> > > > > >>> >> > > > this eagle docs has example on how to setup > the > >> > app. > >> > >> pls > >> > >> > > let > >> > >> > > > > us > >> > >> > > > > >>> >> know if > >> > >> > > > > >>> >> > > > you find any gaps. > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > > Thanks for trying out, and sharing your > >> findings, > >> > >> > > > > >>> >> > > > Jayesh > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O > >> > hEigeartaigh > >> > >> < > >> > >> > > > > >>> >> > > [email protected]> > >> > >> > > > > >>> >> > > > wrote: > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > >> Hi all, > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> I'm trying to play around a bit with Apache > >> Eagle > >> > >> 0.5.0 > >> > >> > > to > >> > >> > > > no > >> > >> > > > > >>> >> avail. > >> > >> > > > > >>> >> > > Here > >> > >> > > > > >>> >> > > >> are the problems I've run into so far: > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> a) The official docker image uses > >> 0.5.0-SNAPSHOT > >> > and > >> > >> > not > >> > >> > > > the > >> > >> > > > > >>> >> released > >> > >> > > > > >>> >> > > >> version. > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> b) Aside from the above, the official docker > >> image > >> > >> > uses a > >> > >> > > > mix > >> > >> > > > > >>> of " > >> > >> > > > > >>> >> > > >> server.eagle.apache.org" and " > >> > >> sandbox.eagle.apache.org > >> > >> > " > >> > >> > > as > >> > >> > > > > the > >> > >> > > > > >>> >> host > >> > >> > > > > >>> >> > > >> name. The HBase service doesn't start by > >> default > >> > in > >> > >> > > Ambari > >> > >> > > > > as a > >> > >> > > > > >>> >> > result. > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> c) The UI seems quite buggy. On both chromium > >> and > >> > >> > > firefox, > >> > >> > > > I > >> > >> > > > > >>> only > >> > >> > > > > >>> >> see > >> > >> > > > > >>> >> > > >> links to "Sandbox" and "Alert" on the left > >> > >> hand-side. > >> > >> > > Once > >> > >> > > > I > >> > >> > > > > >>> click > >> > >> > > > > >>> >> on > >> > >> > > > > >>> >> > > >> "Alert" I have no way of going back to see > the > >> > >> > > > applications. > >> > >> > > > > I > >> > >> > > > > >>> >> don't > >> > >> > > > > >>> >> > see > >> > >> > > > > >>> >> > > >> the links to "integration" or "sites" as in > the > >> > >> picture > >> > >> > > > here: > >> > >> > > > > >>> >> > > >> http://eagle.apache.org/docs/l > >> > >> atest/applications/#jmx- > >> > >> > > > monito > >> > >> > > > > >>> ring > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> d) In chromium, the button to create a new > >> policy > >> > >> does > >> > >> > > not > >> > >> > > > > >>> exist - > >> > >> > > > > >>> >> I > >> > >> > > > > >>> >> > can > >> > >> > > > > >>> >> > > >> only see it on Firefox. > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log > >> Monitor" > >> > >> > > use-case > >> > >> > > > > >>> working, > >> > >> > > > > >>> >> > but > >> > >> > > > > >>> >> > > >> it seems to be stuck in "Initialized". > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> Could someone fill me in on what the > >> "recommended" > >> > >> way > >> > >> > is > >> > >> > > > to > >> > >> > > > > >>> start > >> > >> > > > > >>> >> > > Apache > >> > >> > > > > >>> >> > > >> Eagle so that I can play around with the > >> > >> functionality > >> > >> > > that > >> > >> > > > > it > >> > >> > > > > >>> >> offers? > >> > >> > > > > >>> >> > > >> Clearly the docker approach is buggy. Also, > >> what > >> > >> > browser > >> > >> > > > > >>> should be > >> > >> > > > > >>> >> > used? > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> Thanks, > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> Colm. > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> -- > >> > >> > > > > >>> >> > > >> Colm O hEigeartaigh > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > >> Talend Community Coder > >> > >> > > > > >>> >> > > >> http://coders.talend.com > >> > >> > > > > >>> >> > > >> > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > > > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > -- > >> > >> > > > > >>> >> > > Colm O hEigeartaigh > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > Talend Community Coder > >> > >> > > > > >>> >> > > http://coders.talend.com > >> > >> > > > > >>> >> > > > >> > >> > > > > >>> >> > > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> -- > >> > >> > > > > >>> >> Colm O hEigeartaigh > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> Talend Community Coder > >> > >> > > > > >>> >> http://coders.talend.com > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> > >> > >> > > > > >>> >> > >> > >> > > > > >>> > > >> > >> > > > > >>> > > >> > >> > > > > >>> > -- > >> > >> > > > > >>> > Colm O hEigeartaigh > >> > >> > > > > >>> > > >> > >> > > > > >>> > Talend Community Coder > >> > >> > > > > >>> > http://coders.talend.com > >> > >> > > > > >>> > > >> > >> > > > > >>> > >> > >> > > > > >>> > >> > >> > > > > >>> > >> > >> > > > > >>> -- > >> > >> > > > > >>> Colm O hEigeartaigh > >> > >> > > > > >>> > >> > >> > > > > >>> Talend Community Coder > >> > >> > > > > >>> http://coders.talend.com > >> > >> > > > > >>> > >> > >> > > > > >> > >> > >> > > > > >> > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > -- > >> > >> > > > > Colm O hEigeartaigh > >> > >> > > > > > >> > >> > > > > Talend Community Coder > >> > >> > > > > http://coders.talend.com > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > -- > >> > >> > > Colm O hEigeartaigh > >> > >> > > > >> > >> > > Talend Community Coder > >> > >> > > http://coders.talend.com > >> > >> > > > >> > >> > > >> > >> > >> > >> > >> > >> > >> > >> -- > >> > >> Colm O hEigeartaigh > >> > >> > >> > >> Talend Community Coder > >> > >> http://coders.talend.com > >> > >> > >> > > > >> > > > >> > > >> > > >> > -- > >> > Colm O hEigeartaigh > >> > > >> > Talend Community Coder > >> > http://coders.talend.com > >> > > >> > > > > > > > > -- > > Colm O hEigeartaigh > > > > Talend Community Coder > > http://coders.talend.com > > > -- Colm O hEigeartaigh Talend Community Coder http://coders.talend.com
