abhishekagarwal87 commented on issue #15167: URL: https://github.com/apache/druid/issues/15167#issuecomment-1772277169
You can still keep historical cheaper by introducing a tier that just handles very recent data. This tier could be co-hosted with your indexer processes. So any additional query load is really just moving to a different process on same machine. On Fri, 20 Oct 2023 at 12:56 PM, Mohammad Ghazanfar < ***@***.***> wrote: > @abhishekagarwal87 <https://github.com/abhishekagarwal87> doesn't matter > if you use maxRowsPerSegment or maxTotalRows - at handoff time, all > segments on the indexer get handed-off. This will again result in increased > load on the historicals. > > Quoting from here > <https://druid.apache.org/docs/latest/development/extensions-core/kafka-supervisor-operations#deployment-notes-on-kafka-partitions-and-druid-segments> > , > > When the task reaches one of maxRowsPerSegment, maxTotalRows, or > intermediateHandoffPeriod, it hands off all the segments and creates a new > new set of segments will be created for further events > > ------------------------------ > > An increase in historical load is a problem because, I have to > over-provision my historicals to be able to handle the additional load > during the segment hand-off. If I can prevent this load on the historicals, > I can move my historicals to cheaper instances or reduce their number. > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/druid/issues/15167#issuecomment-1772221758>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AALIWUKPMP6I2K6IEAPSIR3YAIRUHAVCNFSM6AAAAAA6BUFOK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZSGIZDCNZVHA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
