[jira] [Assigned] (ATLAS-3929) Kafka hook for Atlas to capture production/consumption

2020-08-31 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass reassigned ATLAS-3929:
--

Assignee: Viktor Somogyi-Vass

> Kafka hook for Atlas to capture production/consumption
> --
>
> Key: ATLAS-3929
> URL: https://issues.apache.org/jira/browse/ATLAS-3929
> Project: Atlas
>  Issue Type: Improvement
>  Components: kafka-integration
>Reporter: Dennis Jaheruddin
>    Assignee: Viktor Somogyi-Vass
>Priority: Major
>
> Currently we already have hooks for many engines (such as Spark, Nifi, 
> Flink). We also have a hook into the classical storage (HDFS). However, with 
> streaming becoming more and more dominant, we should also get a hook into the 
> streaming storage: Kafka.
> Currently producers and consumers need to explicitly take atlas into account, 
> but for stronger governance and auditability, it should be possible to 
> capture every interaction with Kafka, regardless of how consumption or 
> production is done.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72729: ATLAS-3910: Use KafkaZkClient instead of ZkUtils

2020-08-24 Thread Viktor Somogyi-Vass

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72729/#review221686
---


Ship it!




Ship It!

- Viktor Somogyi-Vass


On Aug. 11, 2020, 7:42 a.m., Viktor Somogyi-Vass wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72729/
> ---
> 
> (Updated Aug. 11, 2020, 7:42 a.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3910: Use KafkaZkClient instead of ZkUtils
> 
> 
> Diffs
> -
> 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  40b1fee71e57175cd7ddf8a3696ef147282295c2 
>   
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java
>  c8cc85cda1f62b2d1a0a49c61abf06d9beb6fdb7 
>   notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java 
> c69574103e360fcfb7dfb525bb06043a1fd647c3 
>   notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java 
> 293784770502299875177bd4a4a9c4b32c6b5a9a 
> 
> 
> Diff: https://reviews.apache.org/r/72729/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Viktor Somogyi-Vass
> 
>



Re: Review Request 72729: ATLAS-3910: Use KafkaZkClient instead of ZkUtils

2020-08-11 Thread Viktor Somogyi-Vass

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72729/
---

(Updated Aug. 11, 2020, 7:42 a.m.)


Review request for atlas and Madhan Neethiraj.


Repository: atlas


Description
---

ATLAS-3910: Use KafkaZkClient instead of ZkUtils


Diffs (updated)
-

  
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
 40b1fee71e57175cd7ddf8a3696ef147282295c2 
  
addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java
 c8cc85cda1f62b2d1a0a49c61abf06d9beb6fdb7 
  notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java 
c69574103e360fcfb7dfb525bb06043a1fd647c3 
  notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java 
293784770502299875177bd4a4a9c4b32c6b5a9a 


Diff: https://reviews.apache.org/r/72729/diff/2/

Changes: https://reviews.apache.org/r/72729/diff/1-2/


Testing
---


Thanks,

Viktor Somogyi-Vass



Re: Asking for JIRA permissions

2020-08-06 Thread Viktor Somogyi-Vass
Thanks Ramesh!

On Thu, Aug 6, 2020 at 4:03 PM Ramesh Mani  wrote:

> Vitor,
>
> I am reviewing this change and will get back soon.
>
> Thanks,
> Ramesh
>
> On Thu, Aug 6, 2020 at 6:50 AM Viktor Somogyi-Vass
>  wrote:
>
> > Who would be the target audience? I've added the atlas group but do I
> need
> > to add specific people?
> > This is the review: https://reviews.apache.org/r/72729/
> >
> > Thanks,
> > Viktor
> >
> > On Mon, Aug 3, 2020 at 4:27 PM Viktor Somogyi-Vass <
> > viktor.somo...@cloudera.com> wrote:
> >
> > > Hi Madhan,
> > >
> > > Thanks, I'll submit my patch through Reviewboard then.
> > >
> > > Viktor
> > >
> > > On Tue, Jul 28, 2020 at 9:00 AM Madhan Neethiraj 
> > > wrote:
> > >
> > >> Viktor,
> > >>
> > >> Thanks for your interest in contributing to Apache Atlas. I added you
> as
> > >> a contributor to Apache Atlas. You can now assign JIRAs to yourself,
> and
> > >> add patches.
> > >>
> > >> Review board (https://reviews.apache.org/) is preferred by most Atlas
> > >> folks to review and provide feedback on the patches.
> > >>
> > >> Regards,
> > >> Madhan
> > >>
> > >>
> > >>
> > >>
> > >> On 6/25/20, 2:56 AM, "Viktor Somogyi-Vass" <
> > viktor.somo...@cloudera.com.INVALID>
> > >> wrote:
> > >>
> > >> Hi devs,
> > >>
> > >> I created a JIRA [1] that I'd like to resolve (and have a PR
> > already)
> > >> but
> > >> I'd like to assign it to myself. Could you please grant me the
> > >> required permissions?
> > >> Also, do you accept PR's against the master branch or does it have
> > to
> > >> be
> > >> provided via JIRA in git patch format?
> > >>
> > >> [1]
> > >> https://issues.apache.org/jira/browse/ATLAS-3864
> > >>
> > >> Thanks,
> > >> Viktor
> > >>
> > >>
> > >>
> >
>


Re: Asking for JIRA permissions

2020-08-06 Thread Viktor Somogyi-Vass
Who would be the target audience? I've added the atlas group but do I need
to add specific people?
This is the review: https://reviews.apache.org/r/72729/

Thanks,
Viktor

On Mon, Aug 3, 2020 at 4:27 PM Viktor Somogyi-Vass <
viktor.somo...@cloudera.com> wrote:

> Hi Madhan,
>
> Thanks, I'll submit my patch through Reviewboard then.
>
> Viktor
>
> On Tue, Jul 28, 2020 at 9:00 AM Madhan Neethiraj 
> wrote:
>
>> Viktor,
>>
>> Thanks for your interest in contributing to Apache Atlas. I added you as
>> a contributor to Apache Atlas. You can now assign JIRAs to yourself, and
>> add patches.
>>
>> Review board (https://reviews.apache.org/) is preferred by most Atlas
>> folks to review and provide feedback on the patches.
>>
>> Regards,
>> Madhan
>>
>>
>>
>>
>> On 6/25/20, 2:56 AM, "Viktor Somogyi-Vass" 
>> 
>> wrote:
>>
>> Hi devs,
>>
>> I created a JIRA [1] that I'd like to resolve (and have a PR already)
>> but
>> I'd like to assign it to myself. Could you please grant me the
>> required permissions?
>> Also, do you accept PR's against the master branch or does it have to
>> be
>> provided via JIRA in git patch format?
>>
>> [1]
>> https://issues.apache.org/jira/browse/ATLAS-3864
>>
>> Thanks,
>> Viktor
>>
>>
>>


Review Request 72729: ATLAS-3910: Use KafkaZkClient instead of ZkUtils

2020-08-04 Thread Viktor Somogyi-Vass

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72729/
---

Review request for atlas.


Repository: atlas


Description
---

ATLAS-3910: Use KafkaZkClient instead of ZkUtils


Diffs
-

  
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
 40b1fee71e57175cd7ddf8a3696ef147282295c2 
  
addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java
 c8cc85cda1f62b2d1a0a49c61abf06d9beb6fdb7 
  notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java 
c69574103e360fcfb7dfb525bb06043a1fd647c3 
  notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java 
293784770502299875177bd4a4a9c4b32c6b5a9a 


Diff: https://reviews.apache.org/r/72729/diff/1/


Testing
---


Thanks,

Viktor Somogyi-Vass



Re: Asking for JIRA permissions

2020-08-03 Thread Viktor Somogyi-Vass
Hi Madhan,

Thanks, I'll submit my patch through Reviewboard then.

Viktor

On Tue, Jul 28, 2020 at 9:00 AM Madhan Neethiraj  wrote:

> Viktor,
>
> Thanks for your interest in contributing to Apache Atlas. I added you as a
> contributor to Apache Atlas. You can now assign JIRAs to yourself, and add
> patches.
>
> Review board (https://reviews.apache.org/) is preferred by most Atlas
> folks to review and provide feedback on the patches.
>
> Regards,
> Madhan
>
>
>
>
> On 6/25/20, 2:56 AM, "Viktor Somogyi-Vass" 
> 
> wrote:
>
> Hi devs,
>
> I created a JIRA [1] that I'd like to resolve (and have a PR already)
> but
> I'd like to assign it to myself. Could you please grant me the
> required permissions?
> Also, do you accept PR's against the master branch or does it have to
> be
> provided via JIRA in git patch format?
>
> [1]
> https://issues.apache.org/jira/browse/ATLAS-3864
>
> Thanks,
> Viktor
>
>
>


[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible

2020-08-03 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3864:
---
Description: 
Zookeeper access should be restricted wherever possible as going forward Kafka 
won't use Zookeeper (see 
[KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]).
Also the Zookeeper classes in Kafka aren't public apis so usage should be 
avoided wherever possible. As a tactical fix (see ATLAS-3910) though it is 
possible to continue using KafkaZkClient as that class likely won't change much 
until the completion of KIP-500.

  was:
Zookeeper access should be restricted wherever possible as going forward Kafka 
won't use Zookeeper (see 
[KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]).
Also the Zookeeper classes in Kafka aren't public apis so usage should be 
avoided wherever possible. As a tactical fix though it is possible to continue 


> Atlas should use AdminClient where possible
> ---
>
> Key: ATLAS-3864
> URL: https://issues.apache.org/jira/browse/ATLAS-3864
> Project: Atlas
>  Issue Type: Improvement
>    Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> Zookeeper access should be restricted wherever possible as going forward 
> Kafka won't use Zookeeper (see 
> [KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]).
> Also the Zookeeper classes in Kafka aren't public apis so usage should be 
> avoided wherever possible. As a tactical fix (see ATLAS-3910) though it is 
> possible to continue using KafkaZkClient as that class likely won't change 
> much until the completion of KIP-500.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible

2020-08-03 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3864:
---
Description: 
Zookeeper access should be restricted wherever possible as going forward Kafka 
won't use Zookeeper (see 
[KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]).
Also the Zookeeper classes in Kafka aren't public apis so usage should be 
avoided wherever possible. As a tactical fix though it is possible to continue 

  was:Zookeeper access should be restricted wherever possible as going forward 
Kafka won't use Zookeeper (see KIP-500).


> Atlas should use AdminClient where possible
> ---
>
> Key: ATLAS-3864
> URL: https://issues.apache.org/jira/browse/ATLAS-3864
> Project: Atlas
>  Issue Type: Improvement
>    Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> Zookeeper access should be restricted wherever possible as going forward 
> Kafka won't use Zookeeper (see 
> [KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]).
> Also the Zookeeper classes in Kafka aren't public apis so usage should be 
> avoided wherever possible. As a tactical fix though it is possible to 
> continue 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible

2020-08-03 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3864:
---
Description: Zookeeper access should be restricted wherever possible as 
going forward Kafka won't use Zookeeper (see KIP-500).  (was: Kafka removed 
ZkUtils as it's an internal class and replaced it with KafkaZkClient. To be 
compatible with newer Kafka versions Atlas should use AdminClient and migrate 
from ZkUtils to KafkaZkClient.)

> Atlas should use AdminClient where possible
> ---
>
> Key: ATLAS-3864
> URL: https://issues.apache.org/jira/browse/ATLAS-3864
> Project: Atlas
>  Issue Type: Improvement
>    Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> Zookeeper access should be restricted wherever possible as going forward 
> Kafka won't use Zookeeper (see KIP-500).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3910) Use KafkaZkClient instead of ZkUtils

2020-08-03 Thread Viktor Somogyi-Vass (Jira)
Viktor Somogyi-Vass created ATLAS-3910:
--

 Summary: Use KafkaZkClient instead of ZkUtils
 Key: ATLAS-3910
 URL: https://issues.apache.org/jira/browse/ATLAS-3910
 Project: Atlas
  Issue Type: Improvement
Reporter: Viktor Somogyi-Vass
Assignee: Viktor Somogyi-Vass


Kafka removed ZkUtils from 2.5 as it's an internal class and replaced it with 
KafkaZkClient. To be compatible with newer Kafka versions Atlas shouldn't use 
ZkUtils.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible

2020-08-03 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3864:
---
Summary: Atlas should use AdminClient where possible  (was: Atlas should 
use AdminClient where possible and remove ZkUtils)

> Atlas should use AdminClient where possible
> ---
>
> Key: ATLAS-3864
> URL: https://issues.apache.org/jira/browse/ATLAS-3864
> Project: Atlas
>  Issue Type: Improvement
>    Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> Kafka removed ZkUtils as it's an internal class and replaced it with 
> KafkaZkClient. To be compatible with newer Kafka versions Atlas should use 
> AdminClient and migrate from ZkUtils to KafkaZkClient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible and remove ZkUtils

2020-08-03 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3864:
---
Summary: Atlas should use AdminClient where possible and remove ZkUtils  
(was: KafkaBridge should use AdminClient)

> Atlas should use AdminClient where possible and remove ZkUtils
> --
>
> Key: ATLAS-3864
> URL: https://issues.apache.org/jira/browse/ATLAS-3864
> Project: Atlas
>  Issue Type: Improvement
>    Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> Kafka removed ZkUtils as it's an internal class and replaced it with 
> KafkaZkClient. To be compatible with newer Kafka versions Atlas should use 
> AdminClient and migrate from ZkUtils to KafkaZkClient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3865) HookConsumer shouldn't use ShutdownThrea

2020-06-25 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3865:
---
Summary: HookConsumer shouldn't use ShutdownThrea  (was: HookConsumer 
shouldn't use ShutdownThread or depend on Scala code)

> HookConsumer shouldn't use ShutdownThrea
> 
>
> Key: ATLAS-3865
> URL: https://issues.apache.org/jira/browse/ATLAS-3865
> Project: Atlas
>  Issue Type: Improvement
>    Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> The internal ShutdownThread depends on Kafka core which isn't desired as it 
> isn't a public API and also it pulls in a Scala dependency that can change 
> across Kafka versions and isn't good to expose.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3865) HookConsumer shouldn't use ShutdownThread

2020-06-25 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3865:
---
Summary: HookConsumer shouldn't use ShutdownThread  (was: HookConsumer 
shouldn't use ShutdownThrea)

> HookConsumer shouldn't use ShutdownThread
> -
>
> Key: ATLAS-3865
> URL: https://issues.apache.org/jira/browse/ATLAS-3865
> Project: Atlas
>  Issue Type: Improvement
>    Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> The internal ShutdownThread depends on Kafka core which isn't desired as it 
> isn't a public API and also it pulls in a Scala dependency that can change 
> across Kafka versions and isn't good to expose.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3865) HookConsumer shouldn't use ShutdownThread or depend on Scala code

2020-06-25 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3865:
---
Summary: HookConsumer shouldn't use ShutdownThread or depend on Scala code  
(was: HookConsumer shouldn't use ShutdownThread)

> HookConsumer shouldn't use ShutdownThread or depend on Scala code
> -
>
> Key: ATLAS-3865
> URL: https://issues.apache.org/jira/browse/ATLAS-3865
> Project: Atlas
>  Issue Type: Improvement
>    Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> The internal ShutdownThread depends on Kafka core which isn't desired as it 
> isn't a public API and also it pulls in a Scala dependency that can change 
> across Kafka versions and isn't good to expose.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3865) HookConsumer shouldn't use ShutdownThread

2020-06-25 Thread Viktor Somogyi-Vass (Jira)
Viktor Somogyi-Vass created ATLAS-3865:
--

 Summary: HookConsumer shouldn't use ShutdownThread
 Key: ATLAS-3865
 URL: https://issues.apache.org/jira/browse/ATLAS-3865
 Project: Atlas
  Issue Type: Improvement
Reporter: Viktor Somogyi-Vass


The internal ShutdownThread depends on Kafka core which isn't desired as it 
isn't a public API and also it pulls in a Scala dependency that can change 
across Kafka versions and isn't good to expose.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Asking for JIRA permissions

2020-06-25 Thread Viktor Somogyi-Vass
Hi devs,

I created a JIRA [1] that I'd like to resolve (and have a PR already) but
I'd like to assign it to myself. Could you please grant me the
required permissions?
Also, do you accept PR's against the master branch or does it have to be
provided via JIRA in git patch format?

[1]
https://issues.apache.org/jira/browse/ATLAS-3864

Thanks,
Viktor


[jira] [Created] (ATLAS-3864) KafkaBridge should use AdminClient

2020-06-25 Thread Viktor Somogyi-Vass (Jira)
Viktor Somogyi-Vass created ATLAS-3864:
--

 Summary: KafkaBridge should use AdminClient
 Key: ATLAS-3864
 URL: https://issues.apache.org/jira/browse/ATLAS-3864
 Project: Atlas
  Issue Type: Improvement
Reporter: Viktor Somogyi-Vass


Kafka removed ZkUtils as it's an internal class and replaced it with 
KafkaZkClient. To be compatible with newer Kafka versions Atlas should use 
AdminClient and migrate from ZkUtils to KafkaZkClient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)