caoyang10 opened a new issue #4846:
URL: https://github.com/apache/incubator-doris/issues/4846
When we have both routine load and broker load in one table, the broker load
data will cover the routine load data because routine load data is not credible
enough. At the same time, we don't hope to change the broker load data but some
DIRTY data(earlier data) in routine load will do it. So it's necessary to set
the receive window on routine load so that broker load data will not change by
routine load data.
Compare to Partition stmt, routine load window feature uses dynamic
partitions but Partition stmt is not.
For example, I only would like to receive near 6 hours data. I could set
properties like
PROPERTIES
(
"window_interval_sec" = "21600"
)
The receive window is [now - 6h, now], what we have to do is check the range
of each partition if it has intersaction with receive window. For example, now
= "2020-11-05 07:01:23", receive window is ["2020-11-05 01:01:23", "2020-11-05
07:01:23"], partition unit is DAY, partition ["2020-11-05", "2020-11-06"] is
hit while other partitions are not. Another example, now = "2020-01-05
05:01:23", receive window is ["2020-11-04 23:01:23", "2020-11-05 05:01:23"], 2
partitions match the rule: ["2020-11-04", "2020-11-05"], ["2020-11-05",
"2020-11-06"].
I have already finished the feature and please review the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
