GWphua commented on code in PR #19187:
URL: https://github.com/apache/druid/pull/19187#discussion_r2983340423
##########
processing/src/main/java/org/apache/druid/segment/loading/DataSegmentKiller.java:
##########
@@ -98,4 +103,22 @@ default void killQuietly(DataSegment segment)
* is only implemented by local and HDFS.
*/
void killAll() throws IOException;
+
+ /**
+ * Best-effort removal of all deep-storage shuffle intermediates for a
native parallel index supervisor task.
+ * Native parallel indexing writes shuffle files only under {@code
shuffle-data/<supervisorTaskId>/} (see
+ * {@link #SHUFFLE_DATA_DIR_NAME} and {@code
org.apache.druid.indexing.worker.shuffle.DeepStorageIntermediaryDataManager});
+ * the default implementation is a no-op.
+ * <p>
+ * Object stores (S3, GCS, Azure Blob, etc.): there is usually no
recursive-delete primitive; implementors
+ * should list objects under the key prefix {@code
shuffle-data/<supervisorTaskId>/} (or the extension's equivalent
+ * layout under the configured bucket/prefix), delete in pages, and tolerate
missing keys (idempotent cleanup). Use
+ * batch delete APIs where available. Be careful with listing consistency:
eventual consistency and pagination
+ * boundaries may require retries or a second list pass. Never delete keys
outside that prefix (other supervisors
+ * share {@code shuffle-data/}). If the supervisor JVM dies before {@code
cleanUp}, operators can remove the same
+ * prefix manually.
+ */
+ default void killShuffleSupervisorPrefix(String supervisorTaskId) throws
SegmentLoadingException
Review Comment:
I followed how `kill` is being designed, and since it throws
`SegmentLoadingException` for any failures there, I also threw it here.
Maybe `IOException` will make more sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
