[
https://issues.apache.org/jira/browse/MESOS-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575179#comment-16575179
]
Meng Zhu edited comment on MESOS-9144 at 8/9/18 6:03 PM:
---------------------------------------------------------
Log `master_auth_log.tar.gz` attached. In about 2mins log covered time,
`Queuing up authentication request from...because authentication is still in
progress` got printed 53K times.
Spotting a single agent `slave(1)@172.31.21.47:5051`, the above log printed
1909 times.
was (Author: mzhu):
Log attached pertaining the described auth logic.
> Master handles authentication requests with strong assumption thus can easily
> be overwhelmed.
> ---------------------------------------------------------------------------------------------
>
> Key: MESOS-9144
> URL: https://issues.apache.org/jira/browse/MESOS-9144
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 1.5.1, 1.6.1
> Reporter: Meng Zhu
> Priority: Blocker
> Attachments: master_auth_log.tar.gz
>
>
> In `Master::authenticate(const UPID& from, const UPID& pid)`, we have:
> https://github.com/apache/mesos/blob/master/src/master/master.cpp#L9594
> {code:c++}
> if (authenticating.contains(pid)) {
> LOG(INFO) << "Queuing up authentication request from " << pid
> << " because authentication is still in progress";
> // Try to cancel the in progress authentication by discarding the
> // future.
> authenticating[pid].discard();
> // Retry after the current authenticator session finishes.
> authenticating[pid]
> .onAny(defer(self(), &Self::authenticate, from, pid));
> return;
> }
> {code}
> Let's say the master is processing authentication request R1 (whose
> associating future is held in `authenticating`). Now it receives another
> request R2 from the same client (due to e.g. re-try), according to the above
> code logic, we will (1) discard R1; (2) enqueue R2 as a callback which will
> be triggered when R1 is discarded, and we will redo `:: authenticate` with R2.
> Here the master assumes that R2 is the most current request. This is true in
> the above example. However, this assumption could easily break when auth
> requests come faster than they are discarded. If we have 3 requests (R1, R2,
> R3) in the event queue, then we could trigger `::authenticate` SIX times in
> total, once for R1, twice for R2 and three times for R3. This grows in
> quadratic to the number of enqueued requests and the master will be
> overwhelmed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
