[
https://issues.apache.org/jira/browse/OAK-12241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rishabh Daim updated OAK-12241:
-------------------------------
Attachment: S3-GCP-EMULATOR-TESTS-PLAN.md
> run S3/GCP integration tests with S3Mock emulator in CI
> -------------------------------------------------------
>
> Key: OAK-12241
> URL: https://issues.apache.org/jira/browse/OAK-12241
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: blob-cloud, blob-cloud-gcp
> Reporter: Rishabh Daim
> Assignee: Rishabh Daim
> Priority: Major
> Attachments: S3-GCP-EMULATOR-TESTS-PLAN.md
>
>
> h2. Summary
> Integration tests in \{{oak-blob-cloud}} require real AWS or GCP credentials
> (\{{aws.properties}}) and are skipped in Apache CI because secrets cannot be
> committed. This issue adds a local S3 emulator (Adobe S3Mock via
> Testcontainers) so existing integration tests run in CI without cloud
> credentials, in both *S3* and *GCP* storage modes.
> h2. Background
> * Most \{{oak-blob-cloud}} integration tests gate on
> \{{S3DataStoreUtils.isS3Configured()}} and skip when
> \{{accessKey}}/\{{secretKey}} are empty.
> * Azure blob tests already use \{{AzuriteDockerRule}} + Testcontainers
> (\{{oak-blob-cloud-azure}}) — same pattern, no secrets in CI.
> * AWS does not provide an official S3 emulator (unlike Azurite for Azure).
> * Oak *GCP mode* uses the AWS S3 SDK against GCS S3 interoperability — not
> the native GCS client — so an S3-compatible emulator covers both modes with
> different config.
> * Detailed implementation plan:
> \{{oak-blob-cloud/docs/S3-GCP-EMULATOR-TESTS-PLAN.md}}
> h2. Proposed solution
> # Add Adobe S3Mock 5.x (\{{com.adobe.testing:s3mock-testcontainers}}) +
> Testcontainers to \{{oak-blob-cloud}}.
> # Introduce \{{S3MockRule}} and \{{S3EmulatorSupport}} (mirror
> \{{AzuriteDockerRule}}): lazy-start Docker container, skip tests if Docker
> unavailable.
> # Wire emulator config into \{{S3DataStoreUtils.getS3Config()}}:
> #* Real credentials in \{{aws.properties}} / \{{-Ds3.config}} take precedence
> (manual cloud testing unchanged).
> #* Otherwise use emulator properties when S3Mock is available.
> # Run existing test classes in *both* modes via two Maven Surefire executions
> (\{{-Ds3.test.mode=S3}} and \{{-Ds3.test.mode=GCP}}) — no duplicate test
> classes.
> # Small production change: opt-in \{{pathStyleAccess}} property (default
> \{{false}}) for S3Mock path-style URLs; GCP mode already enables path-style.
> # Test-only fix: support \{{http://}} presigned URLs in
> \{{S3DataStoreUtils.getHttpsConnection()}} for direct-access tests.
> h2. Test tiers (after change)
> || Tier || When || Purpose ||
> | Unit | Always | Mock/no-network tests (unchanged) |
> | Emulator | CI / local Docker, no creds | S3Mock — S3 and GCP mode |
> | Real cloud | Populated \{{aws.properties}} | Optional manual validation |
> h2. Scope
> *In scope:*
> * \{{TestS3Ds}} and subclasses (cache, small cache)
> * \{{TestS3DataStore}}
> * \{{S3DataRecordAccessProviderTest}} / \{{S3DataRecordAccessProviderIT}}
> * \{{S3DataStoreServiceTest}}
> * SSE subclasses: triage; KMS-with-key / SSE-C remain cloud-only via existing
> skip logic
> *Out of scope:*
> * \{{oak-it}}, \{{oak-jcr}}, \{{oak-upgrade}} S3 tests
> * Migrating \{{oak-segment-aws}} findify S3Mock to Adobe S3Mock
> * fake-gcs-server / native GCS client testing
> h2. Acceptance criteria
> * \{{mvn test -pl oak-blob-cloud}} passes with Docker and *no* cloud
> credentials.
> * Integration tests that currently skip run under the emulator (S3 mode).
> * Same tests run again under GCP mode (second Surefire execution).
> * Populated real \{{aws.properties}} still takes precedence.
> * No secrets in repo or CI.
> * Production behavior unchanged unless \{{pathStyleAccess=true}} is
> explicitly set.
> * Unit tests pass without Docker.
> h2. References
> * Azure precedent: \{{oak-blob-cloud-azure/.../AzuriteDockerRule.java}}
> * S3Mock: https://github.com/adobe/S3Mock
> * Plan doc: \{{oak-blob-cloud/docs/S3-GCP-EMULATOR-TESTS-PLAN.md}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
