[
https://issues.apache.org/jira/browse/ATLAS-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Padashetty updated ATLAS-4043:
------------------------------------
Description:
Even though kafka-topics.sh --list, lists the __consumer_offsets topic as
below, this topic alone is not imported into Atlas when import-kafka.sh script
is run
{code:java}
ATLAS_ENTITIES
ATLAS_HOOK
ATLAS_SPARK_HOOK
__consumer_offsets
connect-configs
connect-offsets
connect-status {code}
I did not find anything unusual in atlas/application.log. This was working fine
before, seems like a regression.
As per official Kafka documentation, __consumer_offsets is used to store
information about committed offsets for each topic:partition per group of
consumers (groupID). It is compacted topic, so data will be periodically
compressed and only latest offsets information available.
Since __consumer_offsets has __ in the beginning, I have created a topic
starting with __ and ran the import-kafka script. And it works fine. So that
leaves us with having issue with __consumer_offsets alone.
was:
Even though kafka-topics.sh --list, lists the __consumer_offsets topic as
below, this topic alone is not imported into Atlas when import-kafka.sh script
is run
{code:java}
ATLAS_ENTITIES
ATLAS_HOOK
ATLAS_SPARK_HOOK
__consumer_offsets
connect-configs
connect-offsets
connect-status {code}
I did not find anything unusual in atlas/application.log
> Atlas import-kafka.sh does not import "__consumer_offsets" topic
> ----------------------------------------------------------------
>
> Key: ATLAS-4043
> URL: https://issues.apache.org/jira/browse/ATLAS-4043
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Affects Versions: 2.1.0
> Reporter: Umesh Padashetty
> Priority: Major
> Attachments: Screenshot 2020-11-20 at 11.23.47 PM.png
>
>
> Even though kafka-topics.sh --list, lists the __consumer_offsets topic as
> below, this topic alone is not imported into Atlas when import-kafka.sh
> script is run
> {code:java}
> ATLAS_ENTITIES
> ATLAS_HOOK
> ATLAS_SPARK_HOOK
> __consumer_offsets
> connect-configs
> connect-offsets
> connect-status {code}
> I did not find anything unusual in atlas/application.log. This was working
> fine before, seems like a regression.
> As per official Kafka documentation, __consumer_offsets is used to store
> information about committed offsets for each topic:partition per group of
> consumers (groupID). It is compacted topic, so data will be periodically
> compressed and only latest offsets information available.
> Since __consumer_offsets has __ in the beginning, I have created a topic
> starting with __ and ran the import-kafka script. And it works fine. So that
> leaves us with having issue with __consumer_offsets alone.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
