npawar opened a new pull request, #8815: URL: https://github.com/apache/pinot/pull/8815
`METADATA` push didn't allow the option of `moveSegmentToFinalLocation`. This meant that if someone had generated segments in a location that was not the deep store, there was absolutely no way to move those segments into deep store without manual scripting. Chatting with @xiangfu0 , I understood that this was done on purpose, in order to keep METADATA push very light weight and not involve copying from outputDir to dataDir in the push. But this is a very valid scenario where users would want to generate segments in a location other than deep store, and still want to use METADATA push (reasons could be multiple, as listed in https://github.com/apache/pinot/issues/7328). While there have been more elaborate solutions proposed in the issue above (like periodic background tasks), this PR simply enables one to allow METADATA push to so `moveToFinalLocation`. And while this would increase the data push time, it is a better solution than asking users to switch to SegmentTarPush or SegmentUriPush if the issue was disparate permissions between outputDir ad dataDir (one of the reasons mentioned in the issue). Backward compatible behavior or this being false by default is maintained. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
