[
https://issues.apache.org/jira/browse/HDDS-16156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16156:
------------------------------
Description:
h3. Symptom
In the {{kubernetes}} acceptance check, the {{s3g}} container
CrashLoopBackOffs: it exits immediately at startup, before serving anything, so
cluster bring-up fails ("Test execution of ozone is FAILED"). Every {{s3g}} pod
across all k8s examples in the run (getting-started, ozone, ozone-dev,
ozone-ha, minikube) fails identically; {{scm}}/{{om}}/{{recon}} start fine.
{code}
2026-08-11 22:11:26 INFO TracingUtil:87 - Initialized tracing service:
S3gateway (enabled=false, applicationAware=true)
Error: Access denied: ozone_http_tmp_base_dir3661902540861472713
{code}
(from {{kubernetes/.../s3g-0}} pod log)
h3. Root cause
Only {{s3g}} calls the static {{HttpServer2.setHttpBaseDir()}} at startup, in
{{Gateway.call()}} (before the HTTP servers are constructed):
{code}
// hadoop-ozone/s3gateway/.../s3/Gateway.java
TracingUtil.initTracing("S3gateway", ...); // -> "Initialized tracing
service: S3gateway" (last line before the error)
...
setHttpBaseDir(OzoneConfigurationHolder.configuration()); // <-- throws here
httpServer = new S3GatewayHttpServer(...); // never reached
{code}
When {{ozone.http.basedir}} is unset, that helper creates the temp dir under
the process *current working directory*:
{code}
// hadoop-hdds/framework/.../server/http/HttpServer2.java
File tmpMetaDir = Files.createTempDirectory(Paths.get(""),
"ozone_http_tmp_base_dir").toFile();
{code}
The container CWD is the {{ozone-runner}} image working directory; the s3g
statefulset sets no {{workingDir}} and no {{runAsUser}}. If that directory is
not writable by the container's effective uid, {{Files.createTempDirectory}}
throws {{AccessDeniedException}}, which {{GenericCli}} reports as {{"Access
denied: " + message}} and the process exits.
Why only {{s3g}} is affected: {{scm}}/{{om}}/{{recon}}/{{datanode}} never call
the static helper. They use {{BaseHttpServer}}, which resolves an unset
{{ozone.http.basedir}} to {{${ozone.metadata.dirs}/webserver}}
(BaseHttpServer.java:189-190) on the mounted, writable {{/data}} volume, so
they never touch the CWD. ({{freon}} also calls the static helper, but it is a
client, not a long-running k8s pod.)
Why only the {{kubernetes}} check: every compose {{docker-config}} sets
{{ozone.http.basedir=/tmp/ozone_http}} explicitly, so the CWD path never runs
under docker-compose. The k8s configmaps set no basedir, so s3g falls into it.
Pass/fail is therefore determined by whether the runner image working directory
is writable by the container uid, a property of the image and k8s runtime
rather than Ozone logic or a timing race. That is why it looks intermittent
while master is green on the same code and image. The CWD-relative temp dir was
introduced in HDDS-9483 (see also HDDS-5239).
h3. Fix
Create the HTTP base dir under {{java.io.tmpdir}} instead of the process CWD,
so startup no longer depends on the working directory's writability. Patch
attached ({{HDDS-16156.001.patch}}): {{HttpServer2.setHttpBaseDir()}} uses
{{Files.createTempDirectory("ozone_http_tmp_base_dir")}} ({{java.io.tmpdir}}),
plus a unit test asserting the resolved base dir exists and lives under
{{java.io.tmpdir}}. This also covers {{freon}}, which uses the same helper.
h3. Notes
Observed on a PR build (run below) but *not* caused by that PR: the failing
code path predates it and runs during early s3g startup, before any SCM
interaction, so it is unrelated to the change under test. PR-build
logs/artifacts expire; the s3g pod log snippet above is the evidence.
- https://github.com/apache/ozone/actions/runs/31539983527/job/93943601739
was:
h3. Symptom
In the {{kubernetes}} acceptance check, the {{s3g}} container
CrashLoopBackOffs: it exits immediately at startup, before serving anything, so
cluster bring-up for the {{ozone}} example (and others) fails ("Test execution
of ozone is FAILED"). Every {{s3g}} pod across all k8s examples in the run
(getting-started, ozone, ozone-dev, ozone-ha, minikube) fails identically;
{{scm}}/{{om}}/{{recon}}/{{httpfs}} start fine.
{code}
2026-08-11 22:11:26 INFO TracingUtil:87 - Initialized tracing service:
S3gateway (enabled=false, applicationAware=true)
Error: Access denied: ozone_http_tmp_base_dir3661902540861472713
{code}
(from {{kubernetes/.../s3g-0}} pod log)
h3. Root cause
{{HttpServer2.setHttpBaseDir()}} creates the HTTP temp base directory under the
process *current working directory* when {{ozone.http.basedir}} is unset:
{code}
// hadoop-hdds/framework/.../server/http/HttpServer2.java:1808-1815
File tmpMetaDir = Files.createTempDirectory(Paths.get(""),
"ozone_http_tmp_base_dir").toFile();
{code}
When the container's CWD is not writable, {{Files.createTempDirectory}} throws
{{AccessDeniedException}}, which {{GenericCli}} reports as {{"Access denied: "
+ message}} (GenericCli.java:157) and the process exits. Only {{s3g}} is
affected: it is stateless and has no metadata dir, so {{ozone.http.basedir}}
stays empty and it falls into this CWD path, whereas {{scm}}/{{om}}/{{recon}}
resolve {{ozone.http.basedir}} to their writable metadata dir and never enter
it (their logs never reference {{ozone_http_tmp_base_dir}}). The s3g
statefulset sets no {{workingDir}}, so the CWD is the image default. This makes
s3g startup fragile to the runtime environment's CWD writability, so the
{{kubernetes}} check fails intermittently (it is green on master for the same
code/image). The CWD-relative temp dir was introduced in HDDS-9483 (see also
HDDS-5239).
h3. Fix direction
Create the HTTP base dir under a guaranteed-writable location
({{java.io.tmpdir}}, or the service metadata dir) instead of the process CWD
({{Paths.get("")}}); alternatively set {{ozone.http.basedir}} for s3g. Also
consider surfacing the offending path in the error so this is not just "Access
denied".
h3. Notes
Observed on a PR build (run below) but *not* caused by that PR: the failing
code path predates it and runs during early s3g startup, before any SCM
interaction, so it is unrelated to the change under test. PR-build
logs/artifacts expire; the s3g pod log snippet above is the evidence.
- https://github.com/apache/ozone/actions/runs/31539983527/job/93943601739
> Flaky kubernetes check: s3g CrashLoopBackOff with "Access denied:
> ozone_http_tmp_base_dir" (HTTP base dir created under non-writable CWD)
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16156
> URL: https://issues.apache.org/jira/browse/HDDS-16156
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: kubernetes, s3gateway
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16156.001.patch
>
>
> h3. Symptom
> In the {{kubernetes}} acceptance check, the {{s3g}} container
> CrashLoopBackOffs: it exits immediately at startup, before serving anything,
> so cluster bring-up fails ("Test execution of ozone is FAILED"). Every
> {{s3g}} pod across all k8s examples in the run (getting-started, ozone,
> ozone-dev, ozone-ha, minikube) fails identically; {{scm}}/{{om}}/{{recon}}
> start fine.
> {code}
> 2026-08-11 22:11:26 INFO TracingUtil:87 - Initialized tracing service:
> S3gateway (enabled=false, applicationAware=true)
> Error: Access denied: ozone_http_tmp_base_dir3661902540861472713
> {code}
> (from {{kubernetes/.../s3g-0}} pod log)
> h3. Root cause
> Only {{s3g}} calls the static {{HttpServer2.setHttpBaseDir()}} at startup, in
> {{Gateway.call()}} (before the HTTP servers are constructed):
> {code}
> // hadoop-ozone/s3gateway/.../s3/Gateway.java
> TracingUtil.initTracing("S3gateway", ...); // -> "Initialized tracing
> service: S3gateway" (last line before the error)
> ...
> setHttpBaseDir(OzoneConfigurationHolder.configuration()); // <-- throws here
> httpServer = new S3GatewayHttpServer(...); // never reached
> {code}
> When {{ozone.http.basedir}} is unset, that helper creates the temp dir under
> the process *current working directory*:
> {code}
> // hadoop-hdds/framework/.../server/http/HttpServer2.java
> File tmpMetaDir = Files.createTempDirectory(Paths.get(""),
> "ozone_http_tmp_base_dir").toFile();
> {code}
> The container CWD is the {{ozone-runner}} image working directory; the s3g
> statefulset sets no {{workingDir}} and no {{runAsUser}}. If that directory is
> not writable by the container's effective uid, {{Files.createTempDirectory}}
> throws {{AccessDeniedException}}, which {{GenericCli}} reports as {{"Access
> denied: " + message}} and the process exits.
> Why only {{s3g}} is affected: {{scm}}/{{om}}/{{recon}}/{{datanode}} never
> call the static helper. They use {{BaseHttpServer}}, which resolves an unset
> {{ozone.http.basedir}} to {{${ozone.metadata.dirs}/webserver}}
> (BaseHttpServer.java:189-190) on the mounted, writable {{/data}} volume, so
> they never touch the CWD. ({{freon}} also calls the static helper, but it is
> a client, not a long-running k8s pod.)
> Why only the {{kubernetes}} check: every compose {{docker-config}} sets
> {{ozone.http.basedir=/tmp/ozone_http}} explicitly, so the CWD path never runs
> under docker-compose. The k8s configmaps set no basedir, so s3g falls into it.
> Pass/fail is therefore determined by whether the runner image working
> directory is writable by the container uid, a property of the image and k8s
> runtime rather than Ozone logic or a timing race. That is why it looks
> intermittent while master is green on the same code and image. The
> CWD-relative temp dir was introduced in HDDS-9483 (see also HDDS-5239).
> h3. Fix
> Create the HTTP base dir under {{java.io.tmpdir}} instead of the process CWD,
> so startup no longer depends on the working directory's writability. Patch
> attached ({{HDDS-16156.001.patch}}): {{HttpServer2.setHttpBaseDir()}} uses
> {{Files.createTempDirectory("ozone_http_tmp_base_dir")}}
> ({{java.io.tmpdir}}), plus a unit test asserting the resolved base dir exists
> and lives under {{java.io.tmpdir}}. This also covers {{freon}}, which uses
> the same helper.
> h3. Notes
> Observed on a PR build (run below) but *not* caused by that PR: the failing
> code path predates it and runs during early s3g startup, before any SCM
> interaction, so it is unrelated to the change under test. PR-build
> logs/artifacts expire; the s3g pod log snippet above is the evidence.
> - https://github.com/apache/ozone/actions/runs/31539983527/job/93943601739
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
