pjain1 edited a comment on issue #8605:
URL: https://github.com/apache/druid/issues/8605#issuecomment-931417828
One of the cause of this is when you resubmit the supervisor to consume from
a different topic without changing the supervisor name, in that case you see
following message in overlord logs
```
Not updating metadata, existing
state[KafkaDataSourceMetadata{SeekableStreamStartSequenceNumbers=SeekableStreamEndSequenceNumbers{stream='STREAM1',
partitionSequenceNumberMap={}] in metadata store doesn't match to the new
start
state[KafkaDataSourceMetadata{SeekableStreamStartSequenceNumbers=SeekableStreamStartSequenceNumbers{stream='STREAM2',
partitionSequenceNumberMap={}, exclusivePartitions=[]}}]
```
Druid maintains end offsets of a topic for last published segments in
`druid_dataSource` table and checks if it matches the start offsets of
currently publishing segments for consistency. It uses the `datasource` name
(which is same as supervisor name) as the key to store this metadata. So when
you change the topic name, obviously the current start offsets to the task will
not match with the stored end offsets hence task will fail.
Solution -
1. If there is no existing datasource having same name as new topic/stream
name, just terminate existing supervisor and submit a new supervisor having
same datasource and topic name.
2. If you want to keep the datasource name - (this is a hack) terminate
currently running supervisor, delete the row for the corresponding dataSource
in the `druid_dataSource` table and resubmit the supervisor. Word of caution -
Since we are directly editing the metadata store do it at your own risk as it
may cause data consistency issues for your druid datasource.
General advice is to keep the datasource name same as topic name and if you
change topic then create a new supervisor with the changed name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
