[
https://issues.apache.org/jira/browse/HDDS-16156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16156:
----------------------------------
Labels: pull-request-available (was: )
> Flaky kubernetes check: s3g CrashLoopBackOff with "Access denied:
> ozone_http_tmp_base_dir" (HTTP base dir created under non-writable CWD)
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16156
> URL: https://issues.apache.org/jira/browse/HDDS-16156
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: kubernetes, s3gateway
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-16156.001.patch
>
>
> h3. Symptom
> In the {{kubernetes}} acceptance check, the {{s3g}} container
> CrashLoopBackOffs: it exits immediately at startup, before serving anything,
> so cluster bring-up fails ("Test execution of ozone is FAILED"). Every
> {{s3g}} pod across all k8s examples in the run (getting-started, ozone,
> ozone-dev, ozone-ha, minikube) fails identically; {{scm}}/{{om}}/{{recon}}
> start fine.
> {code}
> 2026-08-11 22:11:26 INFO TracingUtil:87 - Initialized tracing service:
> S3gateway (enabled=false, applicationAware=true)
> Error: Access denied: ozone_http_tmp_base_dir3661902540861472713
> {code}
> (from {{kubernetes/.../s3g-0}} pod log)
> h3. Root cause
> Only {{s3g}} calls the static {{HttpServer2.setHttpBaseDir()}} at startup, in
> {{Gateway.call()}} (before the HTTP servers are constructed):
> {code}
> // hadoop-ozone/s3gateway/.../s3/Gateway.java
> TracingUtil.initTracing("S3gateway", ...); // -> "Initialized tracing
> service: S3gateway" (last line before the error)
> ...
> setHttpBaseDir(OzoneConfigurationHolder.configuration()); // <-- throws here
> httpServer = new S3GatewayHttpServer(...); // never reached
> {code}
> When {{ozone.http.basedir}} is unset, that helper creates the temp dir under
> the process *current working directory*:
> {code}
> // hadoop-hdds/framework/.../server/http/HttpServer2.java
> File tmpMetaDir = Files.createTempDirectory(Paths.get(""),
> "ozone_http_tmp_base_dir").toFile();
> {code}
> The container CWD is the {{ozone-runner}} image working directory; the s3g
> statefulset sets no {{workingDir}} and no {{runAsUser}}. If that directory is
> not writable by the container's effective uid, {{Files.createTempDirectory}}
> throws {{AccessDeniedException}}, which {{GenericCli}} reports as {{"Access
> denied: " + message}} and the process exits.
> Why only {{s3g}} is affected: {{scm}}/{{om}}/{{recon}}/{{datanode}} never
> call the static helper. They use {{BaseHttpServer}}, which resolves an unset
> {{ozone.http.basedir}} to {{${ozone.metadata.dirs}/webserver}}
> (BaseHttpServer.java:189-190) on the mounted, writable {{/data}} volume, so
> they never touch the CWD. ({{freon}} also calls the static helper, but it is
> a client, not a long-running k8s pod.)
> Why only the {{kubernetes}} check: every compose {{docker-config}} sets
> {{ozone.http.basedir=/tmp/ozone_http}} explicitly, so the CWD path never runs
> under docker-compose. The k8s configmaps set no basedir, so s3g falls into it.
> Pass/fail is therefore determined by whether the runner image working
> directory is writable by the container uid, a property of the image and k8s
> runtime rather than Ozone logic or a timing race. That is why it looks
> intermittent while master is green on the same code and image. The
> CWD-relative temp dir was introduced in HDDS-9483 (see also HDDS-5239).
> h3. Fix
> Create the HTTP base dir under {{java.io.tmpdir}} instead of the process CWD,
> so startup no longer depends on the working directory's writability. Patch
> attached ({{HDDS-16156.001.patch}}): {{HttpServer2.setHttpBaseDir()}} uses
> {{Files.createTempDirectory("ozone_http_tmp_base_dir")}}
> ({{java.io.tmpdir}}), plus a unit test asserting the resolved base dir exists
> and lives under {{java.io.tmpdir}}. This also covers {{freon}}, which uses
> the same helper.
> h3. Notes
> Observed on a PR build (run below) but *not* caused by that PR: the failing
> code path predates it and runs during early s3g startup, before any SCM
> interaction, so it is unrelated to the change under test. PR-build
> logs/artifacts expire; the s3g pod log snippet above is the evidence.
> - https://github.com/apache/ozone/actions/runs/31539983527/job/93943601739
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
