[
https://issues.apache.org/jira/browse/HDDS-16760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-16760:
-----------------------------------
Fix Version/s: 2.3.0
Release Note: Rooted OFS (ofs://) no longer calls OM InfoBucket before
each key-path listStatus when OM is at LIST_STATUS_REJECTS_OBS or newer,
reducing OM RPCs on directory listing. After this change, rooted OFS listing
matches the S3 pattern (one list call, ListBucket-oriented Ranger policy, no
Head/Get-bucket-first) and uses the same OM path as shallow S3 ListObjects.
Ranger deployments that denied bucket read at Info but allowed ListBucket or
prefix list may see listings succeed where they previously failed; review
policies that relied on that double gate.
> Remove the per-call InfoBucket from rooted-ofs listStatus
> ---------------------------------------------------------
>
> Key: HDDS-16760
> URL: https://issues.apache.org/jira/browse/HDDS-16760
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Manager
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> `BasicRootedOzoneClientAdapterImpl.listStatus` issues a `getBucketDetails`
> (`InfoBucket`) RPC on every
> call, so a file system client doing N listings sends 2N OM requests. On
> metadata-heavy `ofs` workloads
> this makes `InfoBucket` one of the highest-volume RPCs OM serves — compare
> OM's `NumBucketInfos` counter
> against `NumListStatus`.
> The fetch does two jobs, and both have to be addressed before it can be
> dropped.
> *Layout check.* `OzoneFSUtils.validateBucketLayout` rejects `OBJECT_STORE`.
> Move it into OM, following
> HDDS-15925: `OmMetadataReader.listStatus` and `listStatusLight` lack the
> check today, and
> `ResolvedBucket` already carries the link-resolved layout, so it costs no
> extra lookup. `listStatus` has
> no object-store caller — nothing in `s3gateway` calls it, and
> `DeleteKeyHandler`'s call is gated on
> `isFileSystemOptimized()` — so the rejection can be unconditional, with no
> protobuf change.
> *Orphan-link pre-check.* `isSourcePathExist()` makes `listStatus` return an
> empty list when a link
> bucket's source has been deleted. OM already fails that case with
> `BUCKET_NOT_FOUND` via
> `resolveBucketLink(..., allowDanglingBuckets=false)`. Move the disambiguation
> to the error path: list
> optimistically with a locally built handle, and only on `BUCKET_NOT_FOUND` do
> the `getBucket()` plus link
> resolve that picks between the empty list and `FileNotFoundException`.
> The client then drops the fetch behind a new `OzoneManagerVersion` constant,
> retaining the current path
> as the fallback for an older OM.
>
> {{listStatus}} _does_ have object-store callers. {{OzoneBucket.listKeys(...,
> shallow=true)}} on OBS/LEGACY buckets uses ListStatusLight, and S3 Gateway
> uses it for {{ListObjects}} with {{{}delimiter=/{}}}. OM trash also lists OBS
> buckets. So the rejection is opt-in: {{ListStatusRequest}} gets {{{}optional
> bool requireFileSystemLayout = 7{}}}, set only by the ofs client. OM rejects
> OBJECT_STORE only when it is set. This is one new optional proto field, with
> no new result code or configuration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
