applike-ss commented on issue #9011: URL: https://github.com/apache/druid/issues/9011#issuecomment-1193975219
> @applike-ss could you please share how many open segments are present for a given task by searching for occurrences of "Announcing segment"? > > Was this stream resharded within 24 hours of this task's run? Are there any WARN / ERROR logs or any errors related to AWS for this stream in the overlord's logs during this time? AFAIK the stream was not resharded anytime lately. In regards to errors we do not get any until we stop the supervisor to exchange middle-manager and coordinators to make the system work again. Warnings we get these: `[GET http://10.11.178.110:8100/druid-internal/v1/segments?counter=24&hash=1658707907926&timeout=240000] Channel disconnected before response complete` `skipInterval[2022-07-25T00:00:00.000Z/2022-07-26T00:00:00.000Z] is not contained in remainingInterval[2019-01-01T00:00:00.000Z/2022-07-25T00:00:00.000Z]` `Exception in supervisor run loop for dataSource [<datasource_name>]` `Exception while getting current/latest sequences` (with error.message `com.amazonaws.services.kinesis.model.ProvisionedThroughputExceededException: Rate exceeded for shard shardId-000000000056 in stream <stream_name> under account <account_id>. (Service: AmazonKinesis; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: <aws_request_id>; Proxy: null)`) @AmatyaAvadhanula which insight will point 1 give us? and which specific log do you refer to for point 2? @j4k4 what do you think about also ingesting into a temporary datasource with coardser segment granularity? Do you want to try it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
