jihaozh commented on a change in pull request #4803: [TE] Set non additive flag
for Pinot dataset
URL: https://github.com/apache/incubator-pinot/pull/4803#discussion_r344391755
##########
File path:
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/auto/onboard/ConfigGenerator.java
##########
@@ -92,6 +95,16 @@ private static TimeGranularity
getExpectedDelayFromTimeunit(TimeUnit timeUnit) {
return expectedDelay;
}
+ /**
+ * Check if the dataset is non-additive. If it is, set the additive flag to
false and set the pre-aggregated keyword.
+ * @param dataset the dataset DTO to check
+ */
+ static void checkNonAdditive(DatasetConfigDTO dataset) {
+ if (dataset.isAdditive() && dataset.getDataset().endsWith(NON_ADDITIVE)) {
Review comment:
Actually, it's not a one-time fix. The logic a bit tricky here.
When creating a `new DatasetConfigDTO()`, the additive is `true`. Then we
check if it's a non_additive dataset, we set the flag to be `false` and set the
keyword to be `*`.
When checking an existing `DatasetConfigDTO`, if additive = `true` for a
non_additive dataset, we set the flag to be `false` and set the keyword to be
`*`. But if additive = `false` for a non_additive dataset, we leave the flag
and the keyword unchanged. If remove the `dataset.isAdditive()` check, we might
override the `PreAggregatedKeyword`. For some old dataset, the keyword is set
to `ALL` instead of `*`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
