jihoonson commented on issue #11231: URL: https://github.com/apache/druid/issues/11231#issuecomment-841414976
Hmm so, based on your comment, I assume you are thinking of pushing all current segments at once in the merge & push phase. In your proposal, you provided promising evidences for why we should not keep sinks and hydrants for batch ingestion. However, it does not provide enough details of the batch appenderator or the rationale for why the batch appenderator must behave in that way besides not holding sinks and hydrants. Because what you are proposing is not just not keeping sinks and hydrants for batch ingestion but _adding a new batch appenderator_, I think my questions for how it will work is still relevant to this proposal. I think our proposal is a place holder to discuss what's reasonable design and implementation in the long term. I'm not sure how you define the "basic" implementation, but if you want to replicate the current behavior of `AppenderatorImpl` in your initial implementation, I think it's OK because it doesn't make anything worse than now. But then everyone should understand that's your execution plan. Please add it in the proposal. > The first recommendation, `Sink` keeping track of how many rows have been added is something I must need to address and I will in the implementation. Our proposal is not only for overall architecture and design, but also for a certain level of implementation details. See https://github.com/apache/druid/issues/8728 and https://gist.github.com/gianm/39548daef74f0373b3c87056e3db4627 as a reference. To me, it seems a valid discussion to talk about how you will implement this, but if you want, we can move it to your pr when it's ready. Please keep in mind that the point of design review is planning out details upfront as much as possible so that we can avoid pitfalls. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
