jihoonson commented on issue #5869: Period load rules should include the future by default URL: https://github.com/apache/incubator-druid/issues/5869#issuecomment-425679362 > I think current design of the load/drop rules is flexible and able to deal with almost all scenarios. Yeah, I think it was the rationale of the current design, but it doesn't explain why `PeriodLoadRule` doesn't consider future data or `PeriodDropRule` is designed in such a way. For `PeriodLoadRule`, I think we didn't consider future data at that time. For `PeriodDropRule`, I have no idea. > I prefer the second way and want to modify current PeriodDropRule not add a new one because the current one is very impractical, IMO no people would like to use such drop rule. I would say it's better to add a new rule because it's much easier. Modifying an existing rule in an incompatible way would take a long time, because we need to discuss about it and check its use cases as many as possible. Please note that modifying `PeriodLoadRule` to include future data is compatible with the current behavior. If `PeriodDropRule` is really not useful, we can deprecate and remove it in the future. But, I know there is a production use case that `PeriodDropRule` _might_ be useful. They are ingesting data from a single Kafka stream into two different Druid dataSources. One stores the recent data of, let's say last month while another stores old data. In such cases, they can use `PeriodDropRule` for the old dataSource to drop last month data to avoid the potential duplicate data issue. So, I would say adding new rules is more welcomed and having the current `PeriodDropRule` would be fine if there's no real problem. What do you think?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
