xiangfu0 commented on code in PR #20160:
URL: https://github.com/apache/pinot/pull/20160#discussion_r4482195427
##########
pinot-spi/src/main/java/org/apache/pinot/spi/ingestion/batch/spec/PushJobSpec.java:
##########
@@ -57,6 +57,14 @@ public class PushJobSpec implements Serializable {
/// recursively.
private String _pushFileNamePattern;
+ /// Applicable for URI push type.
+ /// If true, and `pushFileNamePattern` is configured, the files matched by
the pattern are pushed even without a
+ /// segment archive extension, e.g. segments stored in the deep store such
as committed REALTIME segments.
+ /// Directories, segment metadata files and temporary segment files
(`<segment>.tmp.<uuid>`) are always skipped.
+ /// When `outputDirURI` points to the deep store of another table, keep
`cleanUpOutputDir` false, otherwise the pushed
Review Comment:
Optional: since this flag is meant for pointing at another table's deep
store, consider failing fast (or at least warning) in the URI push runners when
both this flag and `cleanUpOutputDir` are true. Otherwise `sendSegmentUris`
deletes each source segment after pushing, which quietly removes the source
table's deep-store copy. A javadoc note alone is easy to miss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
