pratsgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-6958157913

   > Heads-up from the S3 side: this module copies the directory-marker design 
of the S3 repository, which turned out to break restores of copied backups. I'm 
fixing that for S3 in #5128.
   > 
   > The problem:
   > 
   > * `createDirectory()` (and `pushStream()` for missing parents) writes a 
zero-byte `hdi_isfolder=true` blob for every directory.
   > * `pathExists()` only checks for that exact blob, so a directory without 
its marker "doesn't exist". The javadoc already mentions this for azcopy.
   > * Core relies on `exists()` for directories during restore. Examples are 
the backup path in `BackupManager.forRestore()`, the configset dir in 
`BackupManager.uploadConfigDir()`, and the location check in `BackupAPIBase`. A 
backup that was copied to another container or account without its marker blobs 
therefore can't be restored. As far as I know, `azcopy` skips these directory 
stubs unless you pass `--include-directory-stub`.
   > 
   > `isDirectory()` already falls back to a prefix listing; `pathExists()` 
doesn't. In #5128 I went further and dropped the markers entirely:
   > 
   > * `createDirectory()` is a no-op.
   > * `exists()` on a directory path and `isDirectory()` are a single prefix 
listing (max 1 result). That also finds old marker blobs, so existing backups 
keep working, and `deleteDirectory()` cleans them up.
   > * Writes no longer check or create parent directories, which saves a 
request or more per uploaded file.
   > * To run the shared contract tests without empty directories, I added a 
`supportsEmptyDirectories()` hook to `AbstractBackupRepositoryTest`.
   > 
   > It may be worth doing the same here, or at least making `pathExists()` 
fall back to a prefix listing for directory paths. Hierarchical-namespace (ADLS 
Gen2) accounts have real directories and probably aren't affected.
   
   thanks for pointing that out, I will look into it and the update the pr. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to