havefunatcode commented on PR #20160: URL: https://github.com/apache/pinot/pull/20160#issuecomment-6895630229
@Jackie-Jiang Thanks, that's a lot simpler. Changed it in e62baaa6cf. Files with no dot in the name are now selected along with the archive extensions, for all push types, and the flag, the URI-only overloads and the tmp file check are removed. Only the file name is checked, so dots in parent dirs or the host don't matter. For directories, all push runners now list the output dir through `SegmentPushUtils.listFiles()`, which drops them. If the `PinotFS` doesn't implement `listFilesWithMetadata()` it falls back to `listFiles()` plus `isDirectory()` for names without a dot. Names starting with `_` like `_SUCCESS` are skipped. `pushSegments()` and `getSegmentsToReplace()` use `getSegmentName()` now, so tar push and consistent push work with these names too. Metadata push already did, I added a test with gzip, LZ4 and zstd content in an extensionless file. @xiangfu0's fail fast is still there: with `cleanUpOutputDir`, tar and URI push throw before pushing anything if a segment has no archive extension. Note that a non-segment file without a dot under `outputDirURI` (e.g. `README`) is now picked up and fails the push. `pushFileNamePattern` can exclude it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
