[jira] [Updated] (ATLAS-4440) Upgrade Atlas's Kafka dependency to 2.8
[ https://issues.apache.org/jira/browse/ATLAS-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass updated ATLAS-4440: --- Fix Version/s: 2.3.0 > Upgrade Atlas's Kafka dependency to 2.8 > --- > > Key: ATLAS-4440 > URL: https://issues.apache.org/jira/browse/ATLAS-4440 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Reporter: Viktor Somogyi-Vass >Assignee: Viktor Somogyi-Vass >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: 0001-ATLAS-4440-Update-Kafka-dependency-to-2.8.1.patch > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (ATLAS-4246) Make Kafka Interface aware of Kafka Schema Registry
[ https://issues.apache.org/jira/browse/ATLAS-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425044#comment-17425044 ] Viktor Somogyi-Vass commented on ATLAS-4246: +1, reviewed it, looks good. > Make Kafka Interface aware of Kafka Schema Registry > --- > > Key: ATLAS-4246 > URL: https://issues.apache.org/jira/browse/ATLAS-4246 > Project: Atlas > Issue Type: Improvement > Components: kafka-integration >Affects Versions: 2.1.0, 3.0.0 >Reporter: Aileen Toleikis >Assignee: Viktor Somogyi-Vass >Priority: Major > Labels: Kafka > Fix For: 3.0.0, 2.3.0 > > > Kafka Community is using Schema Registry more and more heavily but as Atlas > is currently unaware of this, this extension helps Atlas make use of the > Schemas. > > We have tested this extension and we have production environments where Atlas > will not be allowed without schema registry access. We have received feedback > that this extension would be sufficient to allow production use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4440) Upgrade Atlas's Kafka dependency to 2.8
[ https://issues.apache.org/jira/browse/ATLAS-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424342#comment-17424342 ] Viktor Somogyi-Vass commented on ATLAS-4440: [~sarath] yes, it is fully backward compatible, there are no issues between client-server communication. Although new protocols were added Kafka is able to handle it via its protocol handshake mechanism so there will be no new issues when older clients connect to newer brokers or vice-versa. This protocol compatibility still goes back to 0.10 for both clients and brokers. Also in this version the "zookeeperless" mode was introduced but for now that is only a beta feature and it won't be enabled by default. > Upgrade Atlas's Kafka dependency to 2.8 > --- > > Key: ATLAS-4440 > URL: https://issues.apache.org/jira/browse/ATLAS-4440 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Reporter: Viktor Somogyi-Vass >Assignee: Viktor Somogyi-Vass >Priority: Major > Attachments: 0001-ATLAS-4440-Update-Kafka-dependency-to-2.8.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4440) Upgrade Atlas's Kafka dependency to 2.8
Viktor Somogyi-Vass created ATLAS-4440: -- Summary: Upgrade Atlas's Kafka dependency to 2.8 Key: ATLAS-4440 URL: https://issues.apache.org/jira/browse/ATLAS-4440 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Viktor Somogyi-Vass Assignee: Viktor Somogyi-Vass -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3929) Kafka hook for Atlas to capture production/consumption
[ https://issues.apache.org/jira/browse/ATLAS-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass reassigned ATLAS-3929: -- Assignee: Viktor Somogyi-Vass > Kafka hook for Atlas to capture production/consumption > -- > > Key: ATLAS-3929 > URL: https://issues.apache.org/jira/browse/ATLAS-3929 > Project: Atlas > Issue Type: Improvement > Components: kafka-integration >Reporter: Dennis Jaheruddin >Assignee: Viktor Somogyi-Vass >Priority: Major > > Currently we already have hooks for many engines (such as Spark, Nifi, > Flink). We also have a hook into the classical storage (HDFS). However, with > streaming becoming more and more dominant, we should also get a hook into the > streaming storage: Kafka. > Currently producers and consumers need to explicitly take atlas into account, > but for stronger governance and auditability, it should be possible to > capture every interaction with Kafka, regardless of how consumption or > production is done. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible
[ https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass updated ATLAS-3864: --- Description: Zookeeper access should be restricted wherever possible as going forward Kafka won't use Zookeeper (see [KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]). Also the Zookeeper classes in Kafka aren't public apis so usage should be avoided wherever possible. As a tactical fix (see ATLAS-3910) though it is possible to continue using KafkaZkClient as that class likely won't change much until the completion of KIP-500. was: Zookeeper access should be restricted wherever possible as going forward Kafka won't use Zookeeper (see [KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]). Also the Zookeeper classes in Kafka aren't public apis so usage should be avoided wherever possible. As a tactical fix though it is possible to continue > Atlas should use AdminClient where possible > --- > > Key: ATLAS-3864 > URL: https://issues.apache.org/jira/browse/ATLAS-3864 > Project: Atlas > Issue Type: Improvement >Reporter: Viktor Somogyi-Vass >Priority: Major > > Zookeeper access should be restricted wherever possible as going forward > Kafka won't use Zookeeper (see > [KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]). > Also the Zookeeper classes in Kafka aren't public apis so usage should be > avoided wherever possible. As a tactical fix (see ATLAS-3910) though it is > possible to continue using KafkaZkClient as that class likely won't change > much until the completion of KIP-500. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible
[ https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass updated ATLAS-3864: --- Description: Zookeeper access should be restricted wherever possible as going forward Kafka won't use Zookeeper (see [KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]). Also the Zookeeper classes in Kafka aren't public apis so usage should be avoided wherever possible. As a tactical fix though it is possible to continue was:Zookeeper access should be restricted wherever possible as going forward Kafka won't use Zookeeper (see KIP-500). > Atlas should use AdminClient where possible > --- > > Key: ATLAS-3864 > URL: https://issues.apache.org/jira/browse/ATLAS-3864 > Project: Atlas > Issue Type: Improvement >Reporter: Viktor Somogyi-Vass >Priority: Major > > Zookeeper access should be restricted wherever possible as going forward > Kafka won't use Zookeeper (see > [KIP-500|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum]). > Also the Zookeeper classes in Kafka aren't public apis so usage should be > avoided wherever possible. As a tactical fix though it is possible to > continue -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible
[ https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass updated ATLAS-3864: --- Description: Zookeeper access should be restricted wherever possible as going forward Kafka won't use Zookeeper (see KIP-500). (was: Kafka removed ZkUtils as it's an internal class and replaced it with KafkaZkClient. To be compatible with newer Kafka versions Atlas should use AdminClient and migrate from ZkUtils to KafkaZkClient.) > Atlas should use AdminClient where possible > --- > > Key: ATLAS-3864 > URL: https://issues.apache.org/jira/browse/ATLAS-3864 > Project: Atlas > Issue Type: Improvement >Reporter: Viktor Somogyi-Vass >Priority: Major > > Zookeeper access should be restricted wherever possible as going forward > Kafka won't use Zookeeper (see KIP-500). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3910) Use KafkaZkClient instead of ZkUtils
Viktor Somogyi-Vass created ATLAS-3910: -- Summary: Use KafkaZkClient instead of ZkUtils Key: ATLAS-3910 URL: https://issues.apache.org/jira/browse/ATLAS-3910 Project: Atlas Issue Type: Improvement Reporter: Viktor Somogyi-Vass Assignee: Viktor Somogyi-Vass Kafka removed ZkUtils from 2.5 as it's an internal class and replaced it with KafkaZkClient. To be compatible with newer Kafka versions Atlas shouldn't use ZkUtils. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible
[ https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass updated ATLAS-3864: --- Summary: Atlas should use AdminClient where possible (was: Atlas should use AdminClient where possible and remove ZkUtils) > Atlas should use AdminClient where possible > --- > > Key: ATLAS-3864 > URL: https://issues.apache.org/jira/browse/ATLAS-3864 > Project: Atlas > Issue Type: Improvement >Reporter: Viktor Somogyi-Vass >Priority: Major > > Kafka removed ZkUtils as it's an internal class and replaced it with > KafkaZkClient. To be compatible with newer Kafka versions Atlas should use > AdminClient and migrate from ZkUtils to KafkaZkClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3864) Atlas should use AdminClient where possible and remove ZkUtils
[ https://issues.apache.org/jira/browse/ATLAS-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass updated ATLAS-3864: --- Summary: Atlas should use AdminClient where possible and remove ZkUtils (was: KafkaBridge should use AdminClient) > Atlas should use AdminClient where possible and remove ZkUtils > -- > > Key: ATLAS-3864 > URL: https://issues.apache.org/jira/browse/ATLAS-3864 > Project: Atlas > Issue Type: Improvement >Reporter: Viktor Somogyi-Vass >Priority: Major > > Kafka removed ZkUtils as it's an internal class and replaced it with > KafkaZkClient. To be compatible with newer Kafka versions Atlas should use > AdminClient and migrate from ZkUtils to KafkaZkClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3865) HookConsumer shouldn't use ShutdownThrea
[ https://issues.apache.org/jira/browse/ATLAS-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass updated ATLAS-3865: --- Summary: HookConsumer shouldn't use ShutdownThrea (was: HookConsumer shouldn't use ShutdownThread or depend on Scala code) > HookConsumer shouldn't use ShutdownThrea > > > Key: ATLAS-3865 > URL: https://issues.apache.org/jira/browse/ATLAS-3865 > Project: Atlas > Issue Type: Improvement >Reporter: Viktor Somogyi-Vass >Priority: Major > > The internal ShutdownThread depends on Kafka core which isn't desired as it > isn't a public API and also it pulls in a Scala dependency that can change > across Kafka versions and isn't good to expose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3865) HookConsumer shouldn't use ShutdownThread
[ https://issues.apache.org/jira/browse/ATLAS-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass updated ATLAS-3865: --- Summary: HookConsumer shouldn't use ShutdownThread (was: HookConsumer shouldn't use ShutdownThrea) > HookConsumer shouldn't use ShutdownThread > - > > Key: ATLAS-3865 > URL: https://issues.apache.org/jira/browse/ATLAS-3865 > Project: Atlas > Issue Type: Improvement >Reporter: Viktor Somogyi-Vass >Priority: Major > > The internal ShutdownThread depends on Kafka core which isn't desired as it > isn't a public API and also it pulls in a Scala dependency that can change > across Kafka versions and isn't good to expose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3865) HookConsumer shouldn't use ShutdownThread or depend on Scala code
[ https://issues.apache.org/jira/browse/ATLAS-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass updated ATLAS-3865: --- Summary: HookConsumer shouldn't use ShutdownThread or depend on Scala code (was: HookConsumer shouldn't use ShutdownThread) > HookConsumer shouldn't use ShutdownThread or depend on Scala code > - > > Key: ATLAS-3865 > URL: https://issues.apache.org/jira/browse/ATLAS-3865 > Project: Atlas > Issue Type: Improvement >Reporter: Viktor Somogyi-Vass >Priority: Major > > The internal ShutdownThread depends on Kafka core which isn't desired as it > isn't a public API and also it pulls in a Scala dependency that can change > across Kafka versions and isn't good to expose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3865) HookConsumer shouldn't use ShutdownThread
Viktor Somogyi-Vass created ATLAS-3865: -- Summary: HookConsumer shouldn't use ShutdownThread Key: ATLAS-3865 URL: https://issues.apache.org/jira/browse/ATLAS-3865 Project: Atlas Issue Type: Improvement Reporter: Viktor Somogyi-Vass The internal ShutdownThread depends on Kafka core which isn't desired as it isn't a public API and also it pulls in a Scala dependency that can change across Kafka versions and isn't good to expose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3864) KafkaBridge should use AdminClient
Viktor Somogyi-Vass created ATLAS-3864: -- Summary: KafkaBridge should use AdminClient Key: ATLAS-3864 URL: https://issues.apache.org/jira/browse/ATLAS-3864 Project: Atlas Issue Type: Improvement Reporter: Viktor Somogyi-Vass Kafka removed ZkUtils as it's an internal class and replaced it with KafkaZkClient. To be compatible with newer Kafka versions Atlas should use AdminClient and migrate from ZkUtils to KafkaZkClient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
