pjain1 commented on issue #8605:
URL: https://github.com/apache/druid/issues/8605#issuecomment-931417828
One of the cause of this is when you resubmit the supervisor to consume from
a different topic without changing the supervisor name, in that case you see
following message in overlord logs
```
Not updating metadata, existing
state[KafkaDataSourceMetadata{SeekableStreamStartSequenceNumbers=SeekableStreamEndSequenceNumbers{stream='STREAM1',
partitionSequenceNumberMap={}] in metadata store doesn't match to the new
start
state[KafkaDataSourceMetadata{SeekableStreamStartSequenceNumbers=SeekableStreamStartSequenceNumbers{stream='STREAM2',
partitionSequenceNumberMap={}, exclusivePartitions=[]}}]
```
Druid maintains end offsets of a topic for last published segments in
`druid_dataSource` table and checks if it matches the start offsets of
currently publishing segments for consistency. It uses the `datasource` name
(which is same as supervisor name) as the key to store this metadata. So when
you change the topic name, obviously the current start offsets to the task will
not match with the stored end offsets hence task will fail.
Solution (a hack) is to terminate currently running supervisor, delete the
row for the corresponding dataSource in the `druid_dataSource` table and
resubmit the supervisor. Word of caution - Since we are directly editing the
metadata store do it at your own risk.
General advice is to keep the datasource name same as topic name and if you
change topic then create a new supervisor with the changed name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
