kfaraz commented on code in PR #14356: URL: https://github.com/apache/druid/pull/14356#discussion_r1213441716
########## docs/development/extensions-core/kinesis-ingestion.md: ########## @@ -572,22 +574,23 @@ In this way, configuration changes can be applied without requiring any pause in #### On the Subject of Segments Each Kinesis Indexing Task puts events consumed from Kinesis Shards assigned to it in a single segment for each segment -granular interval until maxRowsPerSegment, maxTotalRows or intermediateHandoffPeriod limit is reached, at this point a new shard +granular interval until maxRowsPerSegment, maxTotalRows or intermediateHandoffPeriod limit is reached. At this point, a new shard for this segment granularity is created for further events. Kinesis Indexing Task also does incremental hand-offs which means that all the segments created by a task will not be held up till the task duration is over. As soon as maxRowsPerSegment, maxTotalRows or intermediateHandoffPeriod limit is hit, all the segments held by the task at that point in time will be handed-off and new set of segments will be created for further events. This means that the task can run for longer durations of time -without accumulating old segments locally on Middle Manager processes and it is encouraged to do so. +without accumulating old segments locally on Middle Manager processes, and it is encouraged to do so. -Kinesis Indexing Service may still produce some small segments. Lets say the task duration is 4 hours, segment granularity -is set to an HOUR and Supervisor was started at 9:10 then after 4 hours at 13:10, new set of tasks will be started and -events for the interval 13:00 - 14:00 may be split across previous and new set of tasks. If you see it becoming a problem then +Kinesis Indexing Service may still produce some small segments. Let's say the task duration is 4 hours, segment granularity +is set to an HOUR and Supervisor was started at 9:10 then after 4 hours at 13:10. The new set of tasks will be started and Review Comment: The period should be right before then. ```suggestion is set to an HOUR and Supervisor was started at 9:10. Then after 4 hours at 13:10, the new set of tasks will be started and ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
