[
https://issues.apache.org/jira/browse/HDDS-16772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16772:
------------------------------
Attachment: CR-7-resolve-default-acls-in-preexecute.patch
> Default ACLs of new keys and directories come from the applying OM's local
> group mapping, so OMs can store different ACLs
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16772
> URL: https://issues.apache.org/jira/browse/HDDS-16772
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-7-resolve-default-acls-in-preexecute.patch,
> TestBugCR7HaFollowerReplayGroupMapping.java
>
>
> h3. Mechanism
> {{OMKeyRequest.getAclsForDir}} and {{getAclsForKey}} run inside
> {{validateAndUpdateCache}} and call
> {{OzoneAclUtil.getDefaultAclList(createUGIForApi())}}, which resolves the
> creator's primary group through the applying OM's own Hadoop {{Groups}}
> service. The Raft log entry carries only the user name ({{preExecute}} puts
> no ACLs or groups in the request).
> {{OMClientRequest}} warns against exactly this ("Do NOT bring external
> dependencies into this method, doing so could potentially cause divergence in
> OM DB states in HA"). The lookup came in with HDDS-11656.
> h3. Trigger
> A user creates a key, file, directory or multipart upload while the OMs
> resolve that user's primary group differently when they apply the entry. A
> transient LDAP or SSSD failure on one OM host is enough (a failed lookup
> silently drops the group ACL); a membership change while one OM is down or
> lagging is another way. The helpers are reached from the key, file and
> directory create requests and from multipart initiate and complete, so LEGACY
> and OBS buckets are affected as well as FSO.
> h3. Impact
> The same log entry stores different default group ACL rows in the directory /
> file tables of different OMs, permanently. {{getAcl}} output and, under the
> native authorizer, access decisions change depending on which OM serves the
> request. The lookup also runs on the apply thread under the bucket lock, so a
> slow group lookup stalls the state machine.
> h3. Reproduction
> PASS: [^TestBugCR7HaFollowerReplayGroupMapping.java]
> ({{hadoop-ozone/integration-test}}), 3 OM mini cluster, follower stopped and
> restarted, with the creator's primary group changed between applies. The test
> changes the group mapping inside one shared JVM, which stands in for hosts
> that resolve groups differently.
> h3. Scope
> This covers default ACLs derived from local group mapping only. ACL checks
> and bucket link resolution inside apply are already tracked by HDDS-13634 and
> HDDS-15460 and are not part of this report. This could be a subtask of
> HDDS-13634.
> h3. Suggested fix
> Resolve the default ACLs once in {{preExecute}} on the leader and carry the
> result in the request. {{OMVolumeCreateRequest}} ({{:165}}) and
> {{OMBucketCreateRequest}} ({{:341}}) call the same helper inside apply and
> have the same problem, confirmed from code; they need their own carrier field.
> h3. Patch
> [^CR-7-resolve-default-acls-in-preexecute.patch], against
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
> c99d291e863.
> {{KeyArgs}} gets {{repeated OzoneAclInfo defaultAcls = 25}}. A new helper,
> {{OMKeyRequest.setDefaultAcls}}, clears whatever the client sent, resolves
> the default ACLs for the requesting user on the leader and stores them in the
> request. It is called from the five {{preExecute}} methods that feed
> {{getAclsForKey}} / {{getAclsForDir}} ({{OMKeyCreateRequest}},
> {{OMFileCreateRequest}}, {{OMDirectoryCreateRequest}},
> {{S3InitiateMultipartUploadRequest}}, {{S3MultipartUploadCompleteRequest}};
> the FSO subclasses inherit them). Apply uses the list from the request and
> falls back to the old local lookup when it is empty, which is what an entry
> from a leader without this change looks like, so no layout feature is needed.
> Review point: apply now trusts {{defaultAcls}}, so any future request type
> that reuses these helpers must call {{setDefaultAcls}} in its {{preExecute}}.
> Volume and bucket create are not covered by this patch.
> Covered by {{testDefaultAclsResolvedInPreExecute}} in the existing
> {{TestOMDirectoryCreateRequestWithFSO}} and {{TestOMKeyCreateRequest}}
> (inherited by the FSO variant): the user maps to {{group1}} at {{preExecute}}
> and {{group2}} at apply, and a client supplied {{defaultAcls}} entry must not
> be stored. Without the change they fail with {{["user1", "group2"]}} where
> {{["user1", "group1"]}} is expected. All OM request unit suites (1307 tests)
> and {{TestOzoneNonHAWithNativeACL}} pass, checkstyle is clean. The proto
> backwards compatibility check was not run.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
