mcvsubbu commented on issue #8072: URL: https://github.com/apache/pinot/issues/8072#issuecomment-1022647715
I still don't understand why the latency is a problem. Is it that the background task takes too long to complete ? Or, is it that each I/O to S3 costs, and we want to minimize the number of I/Os (at the cost of not having a backup, of course)? Or, is it both? Assuming it is both, the best way to solve this problem as of now will be to - Set the retention days to 0 (instead of default 7) AND - Change the code so that deletion manager deletes it right away if the retention time is set to 0. This is a cluster-wide behavior. So, if you want to change this on a per table basis, we should have a table config (that overrides the cluster config) with a different retention time (which can be 0 to indicate immediate deletion). A change to the API is not the best way to solve this IMO. You could have clients make a mistake while deletion, causing some segments to be retained and others to be deleted. Better to enforce uniform behavior across all segments in the table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
