Raven888888 opened a new issue #11757: URL: https://github.com/apache/druid/issues/11757
### Affected Version 0.21.1 ### Description In DataLoader GUI, I have set `skipOffsetGaps` to be `True`  Which has been reflected correctly in the spec before submission  However, after submission, the supervisor spec does not have the setting anymore  The reason I want to set `skipOffsetGaps` to be `True` is because the offset held by the supervisor does not return any data. For example, if I listen to the kafka topic `apache-kafka/bin/kafka-console-consumer.sh --topic test --offset 95719032467224513 --partition 0 --bootstrap-server 192.168.0.1:9092` and it stuck forever. This is causing the KIS to get `disconnectException: null` due to FETCH request timeout. (See below for logs) If I try a smaller 85719032467224513 or bigger 105719032467224513 offset , able to consume the messages. So I expect by setting `skipOffsetGaps` to be `True`, KIS will move on with the **next available offset by kafka**. Hard resetting the supervisor works, but it will only restart from earliest/latest offset, which may not be the next available offset. (Therefore `resetOffsetAutomatically` setting is not the ideal solution). Not sure if this is a bug or I am misusing `skipOffsetGaps`. Any pointer on how to resolve this issue is greatly appreciated. ### Logs `2021-09-28T08:03:00,669 DEBUG [task-runner-0-priority-0] org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-kafka-supervisor-macfmjnl-1, groupId=kafka-supervisor-macfmjnl] Sending FETCH request with header RequestHeader(apiKey=FETCH, apiVersion=8, clientId=consumer-kafka-supervisor-macfmjnl-1, correlationId=7) and timeout 30000 to node 221569297: {replica_id=-1,max_wait_ms=500,min_bytes=1,max_bytes=52428800,isolation_level=1,session_id=0,session_epoch=0,topics=[{topic=test,partitions=[{partition=0,fetch_offset=95719032467224513,log_start_offset=-1,partition_max_bytes=1048576}]}],forgotten_topics_data=[]}` `2021-09-28T08:03:30,686 DEBUG [task-runner-0-priority-0] org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-kafka-supervisor-macfmjnl-1, groupId=kafka-supervisor-macfmjnl] Disconnecting from node 221569297 due to request timeout.` `2021-09-28T08:03:30,686 DEBUG [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient - [Consumer clientId=consumer-kafka-supervisor-macfmjnl-1, groupId=kafka-supervisor-macfmjnl] Cancelled request with header RequestHeader(apiKey=FETCH, apiVersion=8, clientId=consumer-kafka-supervisor-macfmjnl-1, correlationId=7) due to node 221569297 being disconnected` `2021-09-28T08:03:30,686 INFO [task-runner-0-priority-0] org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-kafka-supervisor-macfmjnl-1, groupId=kafka-supervisor-macfmjnl] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 221569297: org.apache.kafka.common.errors.DisconnectException: null` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
