chihsuan opened a new pull request, #10209:
URL: https://github.com/apache/ozone/pull/10209
## What changes were proposed in this pull request?
`ozone fs -ls -e <path>` previously threw `UnsupportedOperationException:
FileSystem ofs://om does not support Erasure Coding` against any Ozone cluster
— affecting both `ofs://` and `o3fs://`. The error was misleading: Ozone
supports EC; the real cause is that Hadoop's upstream `Ls.processPathArgument`
reads `ContentSummary.getErasureCodingPolicy()` and throws when the value is
`null`, and Ozone's filesystem implementations were never setting the field.
The PR is split into three commits for review:
1. **`HDDS-14043. Plumb erasure coding policy through FileStatusAdapter`** —
adds a nullable `erasureCodingPolicy` field to `FileStatusAdapter`, populates
it from each key's `ReplicationConfig` (canonical EC scheme name when EC,
`"Replicated"` otherwise) in both adapter impls, and derives the policy from
the bucket's own `ReplicationConfig` for synthetic bucket / bucket-snapshot
adapters (which previously hardcoded `"Replicated"`, contradicting the existing
`isErasureCoded` flag for EC buckets). The 15-arg `FileStatusAdapter`
constructor is preserved as a back-compat overload that delegates with a `null`
policy.
2. **`HDDS-14043. Set erasure coding policy on ContentSummary in ofs/o3fs`**
— `BasicOzoneFileSystem` and `BasicRootedOzoneFileSystem` now set the policy on
the `ContentSummary.Builder` using the path's own `FileStatusAdapter`, matching
HDFS's *"policy of the nearest ancestor"* semantic rather than aggregating
descendants (verified in HDFS bytecode at
`FSDirStatAndListingOp.getContentSummaryInt` →
`FSDirErasureCodingOp.unprotectedGetErasureCodingPolicy`).
`BasicOzoneFileSystem` also gains a `getContentSummary` override so o3fs no
longer falls through to the `FileSystem` default, which left the field `null`.
`ContentSummary.Builder.erasureCodingPolicy(String)` does **not** exist
in Hadoop 2.10.2 (verified by inspecting `hadoop-common-2.10.2.jar`). To avoid
a `NoSuchMethodError` on Hadoop 2 clients (`ozonefs-hadoop2`), the new builder
call is isolated behind a protected `applyEcPolicy` hook in the common base
classes; the hook is overridden only in the Hadoop 3 subclasses (`ozonefs/` and
`ozonefs-hadoop3/`). `ozonefs-hadoop2` inherits the no-op default — and Hadoop
2.10.2's `Ls` has no `-e` flag anyway, so there is no functional regression
there.
3. **`HDDS-14043. Add ls -e regression tests for ofs/o3fs`** — adds
`testContentSummaryErasureCodingPolicy` and `testLsDashEDoesNotThrow` to
`AbstractOzoneFileSystemTest` and `AbstractRootedOzoneFileSystemTest`,
exercising the fix through `TestO3FS`, `TestO3FSWithFSO`, `TestOFS`, and
`TestOFSWithFSO`. The second test runs Hadoop `FsShell` directly with `-ls -e`
and asserts return code `0` — the literal regression guard for the original
exception.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-14043
## How was this patch tested?
- New integration tests pass on all four concrete fixtures (`TestO3FS`,
`TestO3FSWithFSO`, `TestOFS`, `TestOFSWithFSO`), each running 2 tests in ~18 s.
- `mvn -pl
hadoop-ozone/ozonefs,hadoop-ozone/ozonefs-common,hadoop-ozone/ozonefs-hadoop2,hadoop-ozone/ozonefs-hadoop3
-am clean install -DskipTests` succeeds (this is the load-bearing check that
the no-op default in `ozonefs-common` does not reference any Hadoop 3-only API;
`ozonefs-hadoop2` compiles `ozonefs-common` against the Hadoop 2.10.2 classpath
via dependency exclusion).
- Bytecode inspection: `BasicRootedOzoneFileSystem.class` in
`ozonefs-common` contains no reference to
`ContentSummary$Builder.erasureCodingPolicy`; `OzoneFileSystem.class` in
`ozonefs/` (the runtime artifact for the `ozone fs` CLI) does contain it.
- Checkstyle clean across all touched modules.
- End-to-end manual reproduction of the original bug confirmed against
`compose/ozone` before the fix; integration tests cover the post-fix behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
