[jira] [Assigned] (ATLAS-3929) Kafka hook for Atlas to capture production/consumption

2020-08-31 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass reassigned ATLAS-3929:
--

Assignee: Viktor Somogyi-Vass

> Kafka hook for Atlas to capture production/consumption
> --
>
> Key: ATLAS-3929
> URL: https://issues.apache.org/jira/browse/ATLAS-3929
> Project: Atlas
>  Issue Type: Improvement
>  Components: kafka-integration
>Reporter: Dennis Jaheruddin
>Assignee: Viktor Somogyi-Vass
>Priority: Major
>
> Currently we already have hooks for many engines (such as Spark, Nifi, 
> Flink). We also have a hook into the classical storage (HDFS). However, with 
> streaming becoming more and more dominant, we should also get a hook into the 
> streaming storage: Kafka.
> Currently producers and consumers need to explicitly take atlas into account, 
> but for stronger governance and auditability, it should be possible to 
> capture every interaction with Kafka, regardless of how consumption or 
> production is done.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible

2020-08-03 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3864:
---
Description: 
Zookeeper access should be restricted wherever possible as going forward Kafka 
won't use Zookeeper (see 
[KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]).
Also the Zookeeper classes in Kafka aren't public apis so usage should be 
avoided wherever possible. As a tactical fix (see ATLAS-3910) though it is 
possible to continue using KafkaZkClient as that class likely won't change much 
until the completion of KIP-500.

  was:
Zookeeper access should be restricted wherever possible as going forward Kafka 
won't use Zookeeper (see 
[KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]).
Also the Zookeeper classes in Kafka aren't public apis so usage should be 
avoided wherever possible. As a tactical fix though it is possible to continue 


> Atlas should use AdminClient where possible
> ---
>
> Key: ATLAS-3864
> URL: https://issues.apache.org/jira/browse/ATLAS-3864
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> Zookeeper access should be restricted wherever possible as going forward 
> Kafka won't use Zookeeper (see 
> [KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]).
> Also the Zookeeper classes in Kafka aren't public apis so usage should be 
> avoided wherever possible. As a tactical fix (see ATLAS-3910) though it is 
> possible to continue using KafkaZkClient as that class likely won't change 
> much until the completion of KIP-500.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible

2020-08-03 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3864:
---
Description: 
Zookeeper access should be restricted wherever possible as going forward Kafka 
won't use Zookeeper (see 
[KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]).
Also the Zookeeper classes in Kafka aren't public apis so usage should be 
avoided wherever possible. As a tactical fix though it is possible to continue 

  was:Zookeeper access should be restricted wherever possible as going forward 
Kafka won't use Zookeeper (see KIP-500).


> Atlas should use AdminClient where possible
> ---
>
> Key: ATLAS-3864
> URL: https://issues.apache.org/jira/browse/ATLAS-3864
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> Zookeeper access should be restricted wherever possible as going forward 
> Kafka won't use Zookeeper (see 
> [KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]).
> Also the Zookeeper classes in Kafka aren't public apis so usage should be 
> avoided wherever possible. As a tactical fix though it is possible to 
> continue 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible

2020-08-03 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3864:
---
Description: Zookeeper access should be restricted wherever possible as 
going forward Kafka won't use Zookeeper (see KIP-500).  (was: Kafka removed 
ZkUtils as it's an internal class and replaced it with KafkaZkClient. To be 
compatible with newer Kafka versions Atlas should use AdminClient and migrate 
from ZkUtils to KafkaZkClient.)

> Atlas should use AdminClient where possible
> ---
>
> Key: ATLAS-3864
> URL: https://issues.apache.org/jira/browse/ATLAS-3864
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> Zookeeper access should be restricted wherever possible as going forward 
> Kafka won't use Zookeeper (see KIP-500).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3910) Use KafkaZkClient instead of ZkUtils

2020-08-03 Thread Viktor Somogyi-Vass (Jira)
Viktor Somogyi-Vass created ATLAS-3910:
--

 Summary: Use KafkaZkClient instead of ZkUtils
 Key: ATLAS-3910
 URL: https://issues.apache.org/jira/browse/ATLAS-3910
 Project: Atlas
  Issue Type: Improvement
Reporter: Viktor Somogyi-Vass
Assignee: Viktor Somogyi-Vass


Kafka removed ZkUtils from 2.5 as it's an internal class and replaced it with 
KafkaZkClient. To be compatible with newer Kafka versions Atlas shouldn't use 
ZkUtils.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible

2020-08-03 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3864:
---
Summary: Atlas should use AdminClient where possible  (was: Atlas should 
use AdminClient where possible and remove ZkUtils)

> Atlas should use AdminClient where possible
> ---
>
> Key: ATLAS-3864
> URL: https://issues.apache.org/jira/browse/ATLAS-3864
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> Kafka removed ZkUtils as it's an internal class and replaced it with 
> KafkaZkClient. To be compatible with newer Kafka versions Atlas should use 
> AdminClient and migrate from ZkUtils to KafkaZkClient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible and remove ZkUtils

2020-08-03 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3864:
---
Summary: Atlas should use AdminClient where possible and remove ZkUtils  
(was: KafkaBridge should use AdminClient)

> Atlas should use AdminClient where possible and remove ZkUtils
> --
>
> Key: ATLAS-3864
> URL: https://issues.apache.org/jira/browse/ATLAS-3864
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> Kafka removed ZkUtils as it's an internal class and replaced it with 
> KafkaZkClient. To be compatible with newer Kafka versions Atlas should use 
> AdminClient and migrate from ZkUtils to KafkaZkClient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3865) HookConsumer shouldn't use ShutdownThrea

2020-06-25 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3865:
---
Summary: HookConsumer shouldn't use ShutdownThrea  (was: HookConsumer 
shouldn't use ShutdownThread or depend on Scala code)

> HookConsumer shouldn't use ShutdownThrea
> 
>
> Key: ATLAS-3865
> URL: https://issues.apache.org/jira/browse/ATLAS-3865
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> The internal ShutdownThread depends on Kafka core which isn't desired as it 
> isn't a public API and also it pulls in a Scala dependency that can change 
> across Kafka versions and isn't good to expose.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3865) HookConsumer shouldn't use ShutdownThread

2020-06-25 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3865:
---
Summary: HookConsumer shouldn't use ShutdownThread  (was: HookConsumer 
shouldn't use ShutdownThrea)

> HookConsumer shouldn't use ShutdownThread
> -
>
> Key: ATLAS-3865
> URL: https://issues.apache.org/jira/browse/ATLAS-3865
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> The internal ShutdownThread depends on Kafka core which isn't desired as it 
> isn't a public API and also it pulls in a Scala dependency that can change 
> across Kafka versions and isn't good to expose.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3865) HookConsumer shouldn't use ShutdownThread or depend on Scala code

2020-06-25 Thread Viktor Somogyi-Vass (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viktor Somogyi-Vass updated ATLAS-3865:
---
Summary: HookConsumer shouldn't use ShutdownThread or depend on Scala code  
(was: HookConsumer shouldn't use ShutdownThread)

> HookConsumer shouldn't use ShutdownThread or depend on Scala code
> -
>
> Key: ATLAS-3865
> URL: https://issues.apache.org/jira/browse/ATLAS-3865
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> The internal ShutdownThread depends on Kafka core which isn't desired as it 
> isn't a public API and also it pulls in a Scala dependency that can change 
> across Kafka versions and isn't good to expose.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3865) HookConsumer shouldn't use ShutdownThread

2020-06-25 Thread Viktor Somogyi-Vass (Jira)
Viktor Somogyi-Vass created ATLAS-3865:
--

 Summary: HookConsumer shouldn't use ShutdownThread
 Key: ATLAS-3865
 URL: https://issues.apache.org/jira/browse/ATLAS-3865
 Project: Atlas
  Issue Type: Improvement
Reporter: Viktor Somogyi-Vass


The internal ShutdownThread depends on Kafka core which isn't desired as it 
isn't a public API and also it pulls in a Scala dependency that can change 
across Kafka versions and isn't good to expose.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3864) KafkaBridge should use AdminClient

2020-06-25 Thread Viktor Somogyi-Vass (Jira)
Viktor Somogyi-Vass created ATLAS-3864:
--

 Summary: KafkaBridge should use AdminClient
 Key: ATLAS-3864
 URL: https://issues.apache.org/jira/browse/ATLAS-3864
 Project: Atlas
  Issue Type: Improvement
Reporter: Viktor Somogyi-Vass


Kafka removed ZkUtils as it's an internal class and replaced it with 
KafkaZkClient. To be compatible with newer Kafka versions Atlas should use 
AdminClient and migrate from ZkUtils to KafkaZkClient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)