[jira] [Created] (EAGLE-381) Improve app attempt id fetcher for spark job feeder

[Huizhi Lu (JIRA)]

Huizhi Lu created EAGLE-381:
---

 Summary: Improve app attempt id fetcher for spark job feeder
 Key: EAGLE-381
 URL: https://issues.apache.org/jira/browse/EAGLE-381
 Project: Eagle
  Issue Type: Bug
Affects Versions: v0.5.0
Reporter: Huizhi Lu
Assignee: Huizhi Lu
 Fix For: v0.5.0


About the attempt_id, assuming attempt_id starting from 1 may not be completely 
correct. This will not work in HDP Sandbox 2.4. Please see the attempt id in 
sandbox. Spark version is 1.6.0.
I am confused why the attempt id are in different naming rule.

We need to fix this issue. I will take care of it.


hdp-2.4
[ {
  "id" : "application_1468625664674_0003",
  "name" : "Spark Pi",
  "attempts" : [ {
"attemptId" : "appattempt_1468625664674_0003_01",
"startTime" : "2016-07-16T00:13:16.320GMT",
"endTime" : "2016-07-16T00:13:24.279GMT",
"sparkUser" : "spark",
"completed" : true
  } ]
}, {
  "id" : "application_1468625664674_0002",
  "name" : "Spark Pi",
  "attempts" : [ {
"startTime" : "2016-07-16T00:12:01.386GMT",
"endTime" : "2016-07-16T00:12:18.181GMT",
"sparkUser" : "spark",
"completed" : true
  } ]

Ares.
  "id" : "application_1464382345557_264985",
  "name" : "",
  "attempts" : [ {
"attemptId" : "1",
"startTime" : "2016-07-15T09:01:25.032GMT",
"endTime" : "2016-07-15T09:58:29.941GMT",
"sparkUser" : "xxx",
"completed" : true
  } ]




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Apache Ranger integration for Audit Logs...

[Edward Zhang]

Thanks Bosco.

For new features, please do under develop branch where Eagle 0.5 is
targeted.

And in develop branch, we have different programming paradigm than before.
Alert engine is separated into a general storm topology and the
applications which prepare data is separate storm topologies so that output
from application will be input to alert engine.

So you can write application based on whatever framework you like, storm,
spark etc. But in Eagle 0.5 we will only support storm based application,
where Eagle provides a framework to manage application lifecycle.

But at beginning, probably you just need write plain storm topology to
process data from Apache Ranger.

Thanks
Edward

On Fri, Jul 15, 2016 at 2:00 PM, Don Bosco Durai  wrote:

> I have some spare time and was planning to work on this. If no one
> currently looking into this JIRA, then can you assign it to me?
>
> https://issues.apache.org/jira/browse/EAGLE-59
>
>
> Thanks
>
> Bosco
>
>
> On 11/29/15, 8:43 PM, "Don Bosco Durai"  wrote:
>
> Edward
>
> Thanks. I will look into HdfsAuditLogProcessorMain class.
>
> I will upload the sample files in the JIRA.
>
>
>
> Thanks
>
> Bosco
>
>
> On 11/29/15, 7:56 PM, "Zhang, Edward (GDI Hadoop)" 
> wrote:
>
> >One more thing, Bosco, could you please copy some sample hdfs audit
> log,
> >hbase log and hive log to here?
> >
> >I realize with Ranger data source, we probably still need some minor
> code
> >development as follows
> >1. Substitute existing eagle data source(raw hdfs audit log) with
> Ranger
> >data source, for example, in HdfsAuditLogProcessorMain, modify the
> code to
> >use different log deserializer
> >2. Ensure output of Ranger log deserializer is compatible to existing
> >eagle data source.
> >
> >With the above code change, we can automatically get all capabilities
> like
> >sensitivity data join, user hadoop command reassembly, hive query
> >semantics parsing etc.
> >
> >Thanks
> >Edward Zhang
> >
> >On 11/29/15, 18:52, "Zhang, Edward (GDI Hadoop)" 
> wrote:
> >
> >>Hi Bosco,
> >>
> >>Thanks for creating this ticket. It is very helpful if EAGLE can use
> >>Ranger as data source and automatically get monitoring capability in
> 9
> >>Hadoop components.
> >>
> >>If a datasource is not from Kafka, and needs a lot of
> pre-processing, it
> >>is not trivial to integrate that data source.
> >>
> >>Ranger¹s data source should be uniform in syntax and the integration
> >>should be straightforward, if we have a uniform deserializer.
> >>
> >>I think we can document the steps of integrating a new datasource.
> >>
> >>Thanks
> >>Edward Zhang
> >>
> >>On 11/29/15, 12:00, "Don Bosco Durai"  wrote:
> >>
> >>>Hi Eagle team
> >>>
> >>>I am excited to see all the activities on this project. I have
> created a
> >>>JIRA (https://issues.apache.org/jira/browse/EAGLE-59) to track the
> >>>integration with Apache Ranger.
> >>>
> >>>One way to integrate is for Ranger to send the audit logs in the
> same way
> >>>as native log format to Kafka. However, Ranger already is doing the
> >>>normalization of the audit format for all the components. So
> >>>reconstructing might not be a good way to go.
> >>>
> >>>I am still getting familiar with the internals of Apache Eagle, but
> if
> >>>someone can help me or document how a 3rd party source can be
> integrated
> >>>with Apache Eagle, then it will be great. Also, what is the change
> >>>required on the analytics side to support new data sources? E.g. If
> we
> >>>integrate with Ranger Audit Logs, we would get audit logs from
> around 9
> >>>components right away. How can we use it?
> >>>
> >>>If you are okay, I am willing to work on this JIRA.
> >>>
> >>>Thanks
> >>>
> >>>Bosco
> >>>
> >>>
> >>
> >
>
>
>
>
>

[jira] [Updated] (EAGLE-59) Add support for Audit Logs from Apache Ranger

[Edward Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-59?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Zhang updated EAGLE-59:
--
Assignee: Don Bosco Durai

> Add support for Audit Logs from Apache Ranger
> -
>
> Key: EAGLE-59
> URL: https://issues.apache.org/jira/browse/EAGLE-59
> Project: Eagle
>  Issue Type: New Feature
>Reporter: Don Bosco Durai
>Assignee: Don Bosco Durai
>
> Apache Ranger centralizes the audit logs for most of the Hadoop components. 
> The logs are also standardized in common format. It might be good if we can 
> ensure the audit format is supported by Apache Eagle.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Apache Ranger integration for Audit Logs...

[Don Bosco Durai]

I have some spare time and was planning to work on this. If no one currently 
looking into this JIRA, then can you assign it to me? 

https://issues.apache.org/jira/browse/EAGLE-59


Thanks

Bosco


On 11/29/15, 8:43 PM, "Don Bosco Durai"  wrote:

Edward

Thanks. I will look into HdfsAuditLogProcessorMain class.

I will upload the sample files in the JIRA. 



Thanks

Bosco


On 11/29/15, 7:56 PM, "Zhang, Edward (GDI Hadoop)"  
wrote:

>One more thing, Bosco, could you please copy some sample hdfs audit log,
>hbase log and hive log to here?
>
>I realize with Ranger data source, we probably still need some minor code
>development as follows
>1. Substitute existing eagle data source(raw hdfs audit log) with Ranger
>data source, for example, in HdfsAuditLogProcessorMain, modify the code to
>use different log deserializer
>2. Ensure output of Ranger log deserializer is compatible to existing
>eagle data source.
>
>With the above code change, we can automatically get all capabilities like
>sensitivity data join, user hadoop command reassembly, hive query
>semantics parsing etc.
>
>Thanks
>Edward Zhang
>
>On 11/29/15, 18:52, "Zhang, Edward (GDI Hadoop)"  wrote:
>
>>Hi Bosco,
>>
>>Thanks for creating this ticket. It is very helpful if EAGLE can use
>>Ranger as data source and automatically get monitoring capability in 9
>>Hadoop components.
>>
>>If a datasource is not from Kafka, and needs a lot of pre-processing, it
>>is not trivial to integrate that data source.
>>
>>Ranger¹s data source should be uniform in syntax and the integration
>>should be straightforward, if we have a uniform deserializer.
>>
>>I think we can document the steps of integrating a new datasource.
>>
>>Thanks
>>Edward Zhang
>>
>>On 11/29/15, 12:00, "Don Bosco Durai"  wrote:
>>
>>>Hi Eagle team
>>>
>>>I am excited to see all the activities on this project. I have created a
>>>JIRA (https://issues.apache.org/jira/browse/EAGLE-59) to track the
>>>integration with Apache Ranger.
>>>
>>>One way to integrate is for Ranger to send the audit logs in the same way
>>>as native log format to Kafka. However, Ranger already is doing the
>>>normalization of the audit format for all the components. So
>>>reconstructing might not be a good way to go.
>>>
>>>I am still getting familiar with the internals of Apache Eagle, but if
>>>someone can help me or document how a 3rd party source can be integrated
>>>with Apache Eagle, then it will be great. Also, what is the change
>>>required on the analytics side to support new data sources? E.g. If we
>>>integrate with Ranger Audit Logs, we would get audit logs from around 9
>>>components right away. How can we use it?
>>>
>>>If you are okay, I am willing to work on this JIRA.
>>>
>>>Thanks
>>>
>>>Bosco
>>> 
>>>
>>
>

[jira] [Resolved] (EAGLE-379) update site/doc with new content of 0.4.0-incubating

[Michael Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Wu resolved EAGLE-379.
--
Resolution: Done

open PR: https://github.com/eaglemonitoring/eaglemonitoring.github.io/pull/57

> update site/doc with new content of 0.4.0-incubating
> 
>
> Key: EAGLE-379
> URL: https://issues.apache.org/jira/browse/EAGLE-379
> Project: Eagle
>  Issue Type: Task
>Reporter: Michael Wu
>Assignee: Michael Wu
>
> base code repository is: 
> https://github.com/eaglemonitoring/eaglemonitoring.github.io
> we'd deploy the new content after 0.4.0-incubating release announced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (EAGLE-380) fix eagle site/doc branding issue

[Michael Wu (JIRA)]

Michael Wu created EAGLE-380:


 Summary: fix eagle site/doc branding issue
 Key: EAGLE-380
 URL: https://issues.apache.org/jira/browse/EAGLE-380
 Project: Eagle
  Issue Type: Task
Reporter: Michael Wu
Assignee: Michael Wu


add "apache" and "incubating" in documents, either in page title, or in the 
first mention of "eagle".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Why?

[Edward Zhang]

I feel the same. Eagle project today needs more discussion in Eagle dev DL.
I do see many discussions and code reviews within individual emails instead
of going through Eagle dev DL. And some users also ask questions to
individual email directly :-)

Could I suggest Eagle committers and community please discuss important
plans and issues in Eagle dev DL to have public record for people to review
at any time?

Thanks
Edward


On Fri, Jul 15, 2016 at 10:07 AM, Julian Hyde  wrote:

> I am seeing a few JIRA cases which are basically just check-in comments.
> They are created just before the commit, they explain what was done in the
> commit, do not explain why, do not link to any previous or future work.
>
> An example of this is EAGLE-378. It arrives a couple of days after I had a
> conversation with Michael [1] about cleaning up included jars, yet it seems
> to be doing exactly the opposite.
>
> Is the Eagle project operating commit-then-review or review-then-commit?
> It seems to be operating commit-then-review, but if so, there’s not enough
> information in the public record for people to review what is happening.
>
> As my math teacher used to say: don’t just write down the answer, you need
> to show your working!
>
> Julian
>
> [1] https://issues.apache.org/jira/browse/EAGLE-378 <
> https://issues.apache.org/jira/browse/EAGLE-378>
>
> [2] https://issues.apache.org/jira/browse/EAGLE-377 <
> https://issues.apache.org/jira/browse/EAGLE-377>

Why?

[Julian Hyde]

I am seeing a few JIRA cases which are basically just check-in comments. They 
are created just before the commit, they explain what was done in the commit, 
do not explain why, do not link to any previous or future work.

An example of this is EAGLE-378. It arrives a couple of days after I had a 
conversation with Michael [1] about cleaning up included jars, yet it seems to 
be doing exactly the opposite.

Is the Eagle project operating commit-then-review or review-then-commit? It 
seems to be operating commit-then-review, but if so, there’s not enough 
information in the public record for people to review what is happening.

As my math teacher used to say: don’t just write down the answer, you need to 
show your working!

Julian

[1] https://issues.apache.org/jira/browse/EAGLE-378 


[2] https://issues.apache.org/jira/browse/EAGLE-377 

[jira] [Created] (EAGLE-379) update site/doc with new content of 0.4.0-incubating

[Michael Wu (JIRA)]

Michael Wu created EAGLE-379:


 Summary: update site/doc with new content of 0.4.0-incubating
 Key: EAGLE-379
 URL: https://issues.apache.org/jira/browse/EAGLE-379
 Project: Eagle
  Issue Type: Task
Reporter: Michael Wu
Assignee: Michael Wu


base code repository is: 
https://github.com/eaglemonitoring/eaglemonitoring.github.io

we'd deploy the new content after 0.4.0-incubating release announced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (EAGLE-378) create a git patch for including tomcat jar dependencies back to 0.4.0-incubating

[Michael Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Wu closed EAGLE-378.


> create a git patch for including tomcat jar dependencies back to 
> 0.4.0-incubating
> -
>
> Key: EAGLE-378
> URL: https://issues.apache.org/jira/browse/EAGLE-378
> Project: Eagle
>  Issue Type: Task
>Reporter: Michael Wu
>Assignee: Michael Wu
>
> ./eagle-assembly/src/main/lib/tomcat/bin/bootstrap.jar
> ./eagle-assembly/src/main/lib/tomcat/bin/commons-daemon.jar
> ./eagle-assembly/src/main/lib/tomcat/bin/tomcat-juli.jar



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (EAGLE-378) create a git patch for including tomcat jar dependencies back to 0.4.0-incubating

[Michael Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379083#comment-15379083
 ] 

Michael Wu commented on EAGLE-378:
--

patch created: 
https://patch-diff.githubusercontent.com/raw/apache/incubator-eagle/pull/268.patch

> create a git patch for including tomcat jar dependencies back to 
> 0.4.0-incubating
> -
>
> Key: EAGLE-378
> URL: https://issues.apache.org/jira/browse/EAGLE-378
> Project: Eagle
>  Issue Type: Task
>Reporter: Michael Wu
>Assignee: Michael Wu
>
> ./eagle-assembly/src/main/lib/tomcat/bin/bootstrap.jar
> ./eagle-assembly/src/main/lib/tomcat/bin/commons-daemon.jar
> ./eagle-assembly/src/main/lib/tomcat/bin/tomcat-juli.jar



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (EAGLE-378) create a git patch for including tomcat jar dependencies back to 0.4.0-incubating

[Michael Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Wu resolved EAGLE-378.
--
Resolution: Fixed

> create a git patch for including tomcat jar dependencies back to 
> 0.4.0-incubating
> -
>
> Key: EAGLE-378
> URL: https://issues.apache.org/jira/browse/EAGLE-378
> Project: Eagle
>  Issue Type: Task
>Reporter: Michael Wu
>Assignee: Michael Wu
>
> ./eagle-assembly/src/main/lib/tomcat/bin/bootstrap.jar
> ./eagle-assembly/src/main/lib/tomcat/bin/commons-daemon.jar
> ./eagle-assembly/src/main/lib/tomcat/bin/tomcat-juli.jar



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (EAGLE-378) create a git patch for including tomcat jar dependencies back to 0.4.0-incubating

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379065#comment-15379065
 ] 

ASF GitHub Bot commented on EAGLE-378:
--

Github user anyway1021 closed the pull request at:

https://github.com/apache/incubator-eagle/pull/268


> create a git patch for including tomcat jar dependencies back to 
> 0.4.0-incubating
> -
>
> Key: EAGLE-378
> URL: https://issues.apache.org/jira/browse/EAGLE-378
> Project: Eagle
>  Issue Type: Task
>Reporter: Michael Wu
>Assignee: Michael Wu
>
> ./eagle-assembly/src/main/lib/tomcat/bin/bootstrap.jar
> ./eagle-assembly/src/main/lib/tomcat/bin/commons-daemon.jar
> ./eagle-assembly/src/main/lib/tomcat/bin/tomcat-juli.jar



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-eagle issue #268: [EAGLE-378] -DO NOT DELETE- add tomcat depending...

[anyway1021]

Github user anyway1021 commented on the issue:

https://github.com/apache/incubator-eagle/pull/268
  
just close this PR for it's not possible to be merged, it's just for 
downloading .jar files and place them into directory: 
./eagle-assembly/src/main/lib/tomcat/bin/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #268: [EAGLE-378] -DO NOT DELETE- add tomcat de...

[anyway1021]

Github user anyway1021 closed the pull request at:

https://github.com/apache/incubator-eagle/pull/268


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (EAGLE-378) create a git patch for including tomcat jar dependencies back to 0.4.0-incubating

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/EAGLE-378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379057#comment-15379057
 ] 

ASF GitHub Bot commented on EAGLE-378:
--

GitHub user anyway1021 opened a pull request:

https://github.com/apache/incubator-eagle/pull/268

[EAGLE-378] -DO NOT DELETE- add tomcat depending jars back to 
0.4.0-incubating

./eagle-assembly/src/main/lib/tomcat/bin/bootstrap.jar
./eagle-assembly/src/main/lib/tomcat/bin/commons-daemon.jar
./eagle-assembly/src/main/lib/tomcat/bin/tomcat-juli.jar

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anyway1021/incubator-eagle EAGLE-378

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-eagle/pull/268.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #268


commit e5803e01c4cc4b1bd347c7adc6449af6d3588714
Author: anyway1021 
Date:   2016-07-15T08:58:19Z

add tomcat depending jars back to 0.4.0-incubating




> create a git patch for including tomcat jar dependencies back to 
> 0.4.0-incubating
> -
>
> Key: EAGLE-378
> URL: https://issues.apache.org/jira/browse/EAGLE-378
> Project: Eagle
>  Issue Type: Task
>Reporter: Michael Wu
>Assignee: Michael Wu
>
> ./eagle-assembly/src/main/lib/tomcat/bin/bootstrap.jar
> ./eagle-assembly/src/main/lib/tomcat/bin/commons-daemon.jar
> ./eagle-assembly/src/main/lib/tomcat/bin/tomcat-juli.jar



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (EAGLE-376) skip faulty hbase related UT code

[Michael Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/EAGLE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Wu resolved EAGLE-376.
--
Resolution: Fixed

> skip faulty hbase related UT code
> -
>
> Key: EAGLE-376
> URL: https://issues.apache.org/jira/browse/EAGLE-376
> Project: Eagle
>  Issue Type: Task
>Reporter: Michael Wu
>Assignee: Michael Wu
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)