jon-wei commented on issue #9133: Kinesis Indexing throwing exception URL: https://github.com/apache/druid/issues/9133#issuecomment-571349266 How often is data written to that shard? You may be hitting the third issue fixed by this PR: https://github.com/apache/druid/pull/8671 > Adjusts checkSequenceNumberAvailability (renamed to checkOffsetAvailability), only the earliest available offset needs to be compared against the target offset. Because the "LATEST" Kinesis shard iterator only provides access to the seq no after the most recent record (and there is no mode to get the latest readable record), the "get latest offset" operation in Kinesis runs in a polling loop. When data is not being continuously written to a shard, this can result in a Kinesis supervisor making extremely slow progress as it waits for the polling loop to timeout for each shard that has no recent data. Removing the latest offset check restricts this problem situation to only when a Kinesis supervisor first discovers a shard that's empty (after data is written to that shard once, the earliest seq no check can return immediately)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
