jenniferdai commented on a change in pull request #4464: Delete extra refreshed
segments after segment push
URL: https://github.com/apache/incubator-pinot/pull/4464#discussion_r306997584
##########
File path:
pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentTarPushJob.java
##########
@@ -48,10 +54,45 @@ public void run()
throws Exception {
FileSystem fileSystem = FileSystem.get(_conf);
try (ControllerRestApi controllerRestApi = getControllerRestApi()) {
- controllerRestApi.pushSegments(fileSystem,
getDataFilePaths(_segmentPattern));
+ // TODO: Deal with invalid prefixes in the future
Review comment:
I'm not plannign on deleting segments pushed together. I am deleting extra
segments. For example,
If table_day1_0
table_day1_1
table_day1_2 are pushed today.
Table_day1_0
table_day1_1 are pushed tomorrow,
I will delete table_day1_2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
