sthetland commented on issue #9253: Web console: make supervisor reset really scary in the UI URL: https://github.com/apache/druid/pull/9253#issuecomment-578983511 I took a quick look at doc impact. The only existing doc I could find where this concept is touched upon is https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html#resetting-supervisors (and equivalent section in the Kinesis page). That section can throttle up the fear factor a bit, but it would be good to put the explanation in docs that are set in the context of UI doc (rather than this operation endpoint doc linked). This would be a larger effort though, I as don't see an existing logical spot for that doc. Something for me to work on. Meanwhile, taking a stab at how to reword the existing section to induce more fear, for one thing, by moving possible ill effects up : ``` The `POST /druid/indexer/v1/supervisor/<supervisorId>/reset` operation clears stored offsets, causing the supervisor to start reading offsets from either the earliest or latest offsets in Kafka (depending on the value of useEarliestOffset). After clearing stored offsets, the supervisor kills and recreates any active tasks, so that tasks begin reading from valid offsets. Use care when using this operation! Resetting the supervisor may cause Kafka messages to be skipped or read twice, resulting in missing or duplicate data. The reason for using this operation is to recover from a state in which the supervisor ceases operating due to missing offsets. The indexing service keeps track of the latest persisted Kafka offsets in order to provide exactly-once ingestion guarantees across tasks. Subsequent tasks must start reading from where the previous task completed in order for the generated segments to be accepted. If the messages at the expected starting offsets are no longer available in Kafka (typically because the message retention period has elapsed or the topic was removed and re-created) the supervisor will refuse to start and in flight tasks will fail. This operation enables you to recover from this condition. Note that the supervisor must be running for this endpoint to be available. ``` (Change applies to https://github.com/apache/druid/blob/master/docs/development/extensions-core/kafka-ingestion.md) Let me know if my rewrite is accurate (and addresses the goal). If so, I’ll file a doc ticket/PR (or add it to this one?)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
