Jackie-Jiang commented on a change in pull request #7158:
URL: https://github.com/apache/incubator-pinot/pull/7158#discussion_r670037480
##########
File path:
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtime_to_offline_segments/RealtimeToOfflineSegmentsTaskGenerator.java
##########
@@ -207,29 +207,35 @@ public String getTaskType() {
configs.put(MinionConstants.DOWNLOAD_URL_KEY,
StringUtils.join(downloadURLs, MinionConstants.URL_SEPARATOR));
configs.put(MinionConstants.UPLOAD_URL_KEY,
_clusterInfoAccessor.getVipUrl() + "/segments");
- // Execution window
+ // Segment processor configs
configs.put(RealtimeToOfflineSegmentsTask.WINDOW_START_MS_KEY,
String.valueOf(windowStartMs));
configs.put(RealtimeToOfflineSegmentsTask.WINDOW_END_MS_KEY,
String.valueOf(windowEndMs));
-
- // Segment processor configs
- String timeColumnTransformationConfig =
-
taskConfigs.get(RealtimeToOfflineSegmentsTask.TIME_COLUMN_TRANSFORM_FUNCTION_KEY);
- if (timeColumnTransformationConfig != null) {
-
configs.put(RealtimeToOfflineSegmentsTask.TIME_COLUMN_TRANSFORM_FUNCTION_KEY,
timeColumnTransformationConfig);
+ String roundBucketTimePeriod =
taskConfigs.get(RealtimeToOfflineSegmentsTask.ROUND_BUCKET_TIME_PERIOD_KEY);
+ if (roundBucketTimePeriod != null) {
+
configs.put(RealtimeToOfflineSegmentsTask.ROUND_BUCKET_TIME_PERIOD_KEY,
roundBucketTimePeriod);
Review comment:
Good question. Partitioning on time value is needed for merge/rollup
task because it does not filter on the time values. For realtime-to-offline
task, it can be useful when the time window covers multiple partition buckets,
but we don't generate task this way currently. Removed it for now and we can
add it back if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
