[
https://issues.apache.org/jira/browse/MESOS-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720831#comment-16720831
]
Chun-Hung Hsiao commented on MESOS-9474:
----------------------------------------
Review: https://reviews.apache.org/r/69563/
Will work on a unit test and post another review so marking this ticket as
in-progress.
> Master processes authorization inproperly for `LAUNCH_GROUP`, `CREATE_DISK`
> and `DESTROY_DISK`.
> -----------------------------------------------------------------------------------------------
>
> Key: MESOS-9474
> URL: https://issues.apache.org/jira/browse/MESOS-9474
> Project: Mesos
> Issue Type: Bug
> Components: master
> Affects Versions: 1.7.0
> Reporter: Chun-Hung Hsiao
> Assignee: Chun-Hung Hsiao
> Priority: Blocker
> Labels: mesosphere, storage
>
> On our internal cluster with a custom authorizer module we observed the
> following problem:
> {noformat}
> I1212 14:16:58.424782 12492 master.cpp:4489] Processing ACCEPT call for
> offers: [ 98c96586-1980-4007-9651-18dd837f529a-O16233 ] on agent
> 98c96586-1980-4007-9651-18dd837f529a-S1 at slave(1)@172.31.15.78:5051
> (172.31.15.78) for framework 98c96586-1980-4007-9651-18dd837f529a-0009
> (storage)
> I1212 14:16:58.424831 12492 master.cpp:3949] Authorizing principal
> 'storage-principal' to destroy disk 'disk(allocated:
> dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_d17d081f-c138-472d-9e0b-aa6b21e43c7c,profile-b)]:100'
> I1212 14:16:58.424913 12492 master.cpp:3949] Authorizing principal
> 'storage-principal' to destroy disk 'disk(allocated:
> dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_c62a4e89-f902-4968-96d8-69b9efa5f35a,profile-b)]:100'
> I1212 14:16:58.424979 12492 master.cpp:3949] Authorizing principal
> 'storage-principal' to destroy disk 'disk(allocated:
> dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_1f490a57-9e41-41a0-b7be-408147281113,profile-b)]:100'
> I1212 14:16:58.425055 12492 master.cpp:3653] Authorizing principal
> 'storage-principal' to reserve resources 'disk(allocated:
> dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_c60ec637-3f67-4d41-bbc8-448c022c05fb,profile-b)]:100'
> I1212 14:16:58.425118 12492 master.cpp:3949] Authorizing principal
> 'storage-principal' to destroy disk 'disk(allocated:
> dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_5ce79163-aef2-4fd4-8073-f310dec239bf,profile-b)]:100'
> I1212 14:16:58.425499 12488 authorizer.cpp:957] dstport=5050
> dstip=172.31.5.116 result=deny object="" action="DESTROY_MOUNT_DISK"
> uid="storage-principal" reason="" authorizer="mesos-master"
> timestamp=2018-12-12 14:16:58.425453056+00:00 type=audit
> I1212 14:16:58.426434 12483 master.cpp:5769] Processing DESTROY_DISK
> operation for volume disk(allocated: dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_d17d081f-c138-472d-9e0b-aa6b21e43c7c,profile-b)]:100
> from framework 98c96586-1980-4007-9651-18dd837f529a-0
> .78:5051 (172.31.15.78)
> I1212 14:16:58.426746 12483 master.cpp:5769] Processing DESTROY_DISK
> operation for volume disk(allocated: dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_c62a4e89-f902-4968-96d8-69b9efa5f35a,profile-b)]:100
> from framework 98c96586-1980-4007-9651-18dd837f529a-0
> .78:5051 (172.31.15.78)
> I1212 14:16:58.427112 12483 master.cpp:5769] Processing DESTROY_DISK
> operation for volume disk(allocated: dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_1f490a57-9e41-41a0-b7be-408147281113,profile-b)]:100
> from framework 98c96586-1980-4007-9651-18dd837f529a-0
> .78:5051 (172.31.15.78)
> W1212 14:16:58.427274 12483 master.cpp:2275] Dropping RESERVE operation from
> framework 98c96586-1980-4007-9651-18dd837f529a-0009 (storage): Not authorized
> to reserve resources as 'storage-principal'
> I1212 14:16:58.427366 12483 master.cpp:5769] Processing DESTROY_DISK
> operation for volume disk(allocated: dcos-storage)(reservations:
> [(DYNAMIC,dcos-storage,storage-principal)])[MOUNT(g4FmoSCHyiqTZWw1nkOd0v9_5ce79163-aef2-4fd4-8073-f310dec239bf,profile-b)]:100
> from framework 98c96586-1980-4007-9651-18dd837f529a-0009 (storage) to agent
> 98c96586-1980-4007-9651-18dd837f529a-S1 at slave(1)@172.31.15.78:5051
> (172.31.15.78){noformat}
> The authorizer module caches authorization results, and that's why there was
> only one logged authorization requset. The problem is that, the logged
> request was for {{CREATE_MOUNT_DISK}}, and the result was {{deny}}, but
> despite the denial, all {{CREATE_DISK}} operations were processed, however
> another {{RESERVE}} operation was dropped because of this denial.
> The bug is that the master pushed a authorization future in the {{futures}}
> vector in {{Master::accept}} for each {{DESTROY_DISK}}:
>
> [https://github.com/apache/mesos/blob/18356bf3f4ac730b4a798261aad042555c4a4834/src/master/master.cpp#L4599-L4601]
> However, the master never popped and checked the future in
> {{Master::_accept}}, but go ahead to process the operation:
>
> [https://github.com/apache/mesos/blob/18356bf3f4ac730b4a798261aad042555c4a4834/src/master/master.cpp#L5706]
> The future ended up mismatched with the {{RESERVE}} operation, causing it to
> be dropped.
> Updated: a similar bug would happen if we have a validation error for
> {{LAUNCH_GROUP}}, or an authorization error for a task except for the last
> one: the remaining authorization futures for this operation will be examined
> when processing subsequent operations.
> https://github.com/apache/mesos/blob/3ade731d0c1772206c4afdf56318cfab6356acee/src/master/master.cpp#L5487-L5521
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
