moszutij commented on issue #8116: URL: https://github.com/apache/pinot/issues/8116#issuecomment-1029378519
I think the root cause might be the differences between the implementation of LocalPinotFS.exists(...). and S3PinotFS.exists(...). LocalPinotFS.exists() for the following URI: file://var/pinot/controller/data/actions/ will return true and then attempt to move the file(s) to the Deleted_Segments folder whereas S3PinotFS.exists() for the following URI: s3://<bucket>/pinot/controller/data/actions/ will return false and then log a warning that it is unable to find the segment file. I know that the concept of folders does not really exist in object storage and files in a bucket contain slashes to emulate folders in a file system. That could be the reason but I didn't dig any further. My use case was to implement a segment retention policy where S3 is used as the backed segment store e.g. delete segments that are older than 1 year which wasn't working for me and was finally narrowed down to this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
