elangelo opened a new pull request, #5128: URL: https://github.com/apache/solr/pull/5128
Supersedes #4182 (closed as stale, and it can't be reopened after the branch was rebased). See that PR for the earlier review discussion. # Description The S3 backup repository simulated directories with zero-byte "directory marker" objects (keys ending in `/`, content type `application/x-directory`), and relied on them when restoring. Tools that copy buckets, such as `aws s3 sync`, only copy the real files and drop those markers. A backup copied that way could no longer be restored: the configset directory, among others, was reported as missing. # Solution Following the review on #4182, the S3 repository no longer uses directory markers at all, rather than adding a fallback for when they are missing: - `S3StorageClient.createDirectory()` is a no-op. On S3 a directory is just a key prefix that has objects under it. - `exists()` on a directory path and `isDirectory()` use a single prefix listing (`maxKeys=1`). This finds markerless directories as well as marker objects left by older Solr versions, so existing backups can still be restored. `deleteDirectory()` also removes those old markers. - Writing a file no longer checks that its parent "directory" exists, and reading a file's length takes a single HEAD request again. Both mean fewer S3 requests. - Core backup/restore code is unchanged and keeps calling `createDirectory()`/`exists()`. That keeps `LocalFileSystemRepository`, which needs real parent directories, and the other repositories working as before. On S3 an empty directory can't exist any more. The shared `AbstractBackupRepositoryTest` contract assumed it could, so it gets a `supportsEmptyDirectories()` hook. S3 overrides it to run the shared tests instead of copying and overriding them. Copilot (#4182) and Claude Code were used to develop this change. # Tests All new tests run against S3Mock instead of Mockito mocks: - `S3IncrementalBackupTest#testRestoreBackupCopiedWithoutDirectoryMarkers` is the end-to-end regression test. It backs up a collection and checks the backup contains no marker objects. It then copies the backup file by file to another location, as `aws s3 sync` would, and restores it under a new configset name so the configset is uploaded from the backup. Finally it checks the configset and the document count. - `S3BackupRepositoryTest#testLegacyDirectoryMarkers` checks that a backup containing old-style markers can still be checked for existence, listed, read and fully deleted. - `S3PathsTest`, `S3ReadWriteTest` and `S3BackupRepositoryTest` are updated for markerless directories. Ran `:solr:modules:s3-repository:test -Ptests.nightly=true`; all pass except `S3InstallShardTest#testInstallSkipsWriteLockFile`, which fails the same way on current `main` and is unrelated to this change. I also ran `LocalFileSystemRepositoryTest` and `GCSBackupRepositoryTest` to cover the shared contract test change, and `check -x test` on the changed modules. # Checklist - [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [ ] I have created a Jira issue and added the issue ID to my pull request title. - [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation) - [x] I have developed this patch against the `main` branch. - [ ] I have run `./gradlew check`. - [x] I have added tests for my changes. - [x] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide) - [x] I have added a [changelog entry](https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc) for my change 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
