siddharthteotia commented on a change in pull request #4368: 3891: Check for
validity of segment start/end time during segment generation
URL: https://github.com/apache/incubator-pinot/pull/4368#discussion_r303545109
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
##########
@@ -391,6 +399,134 @@ void writeMetadata()
properties.save();
}
+ /**
+ * Check for the validity of segment start and end time
+ * @param startTime segment start time
+ * @param endTime segment end time
+ * @param segmentName segment name
+ */
+ private void checkTime(final SegmentGeneratorConfig config, final Object
startTime, final Object endTime,
Review comment:
The method in SegmentValidator does the check using segment metadata and
compares the startMillis and endMillis against the valid range.
I have written a common utility method in TimeUtils that takes the start and
end time as millis from epoch and compared them against the valid range. This
ensures that if in future the validity comparison logic changes, we can do it
in single place and it will impact both scenarios -- check during generation
and check during segment upload in controller.
It didn't seem feasible to abstract the complete method into a common method
since new code (in segment generation), handles a lot more information -- data
types (INT, LONG, STRING) along with time units (DAYS, MILLIS, HOURs etc) and
whether we have EPOCH or SIMPLE_DATE_FORMAT. This also means that if the check
fails, I dump few additional information into logger and error message.
Let me know if this is fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
