elangelo commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-6165784494
Heads-up from the S3 side: this module copies the directory-marker design of the S3 repository, which turned out to break restores of copied backups. I'm fixing that for S3 in #5128. The problem: - `createDirectory()` (and `pushStream()` for missing parents) writes a zero-byte `hdi_isfolder=true` blob for every directory. - `pathExists()` only checks for that exact blob, so a directory without its marker "doesn't exist". The javadoc already mentions this for azcopy. - Core relies on `exists()` for directories during restore. Examples are the backup path in `BackupManager.forRestore()`, the configset dir in `BackupManager.uploadConfigDir()`, and the location check in `BackupAPIBase`. A backup that was copied to another container or account without its marker blobs therefore can't be restored. As far as I know, `azcopy` skips these directory stubs unless you pass `--include-directory-stub`. `isDirectory()` already falls back to a prefix listing; `pathExists()` doesn't. In #5128 I went further and dropped the markers entirely: - `createDirectory()` is a no-op. - `exists()` on a directory path and `isDirectory()` are a single prefix listing (max 1 result). That also finds old marker blobs, so existing backups keep working, and `deleteDirectory()` cleans them up. - Writes no longer check or create parent directories, which saves a request or more per uploaded file. - To run the shared contract tests without empty directories, I added a `supportsEmptyDirectories()` hook to `AbstractBackupRepositoryTest`. It may be worth doing the same here, or at least making `pathExists()` fall back to a prefix listing for directory paths. Hierarchical-namespace (ADLS Gen2) accounts have real directories and probably aren't affected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
