Dario,

It's not immediately clear to me where the bottleneck might be. I filed
MESOS-6405 <https://issues.apache.org/jira/browse/MESOS-6405> to write a
benchmark that tries to mimic your test setup and then go about fixing the
issues.

-anand

On Sun, Oct 16, 2016 at 6:20 PM, Dario Rexin <dre...@apple.com> wrote:

> Hi Anand,
>
> I tested with and without pipelining and it doesn’t make a difference.
> First of all because unlimited pipelining is not a good idea, because we
> still have to handle the responses and need to be able to relate the
> request and response upon return, i.e. store the context of the request
> until we receive the response. Also, we want to know as soon as possible
> when an error occurs, so early returns are very desirable. I agree that it
> shouldn't make a difference in how fast events can be processed if they are
> queued on the master vs. client, but this observation made it very apparent
> that throughput is a problem on the master. I did not make any requests
> that would potentially block for a long time, so it’s even weirder to me,
> that the throughput is so low. One thing I don’t understand for example, is
> why all messages go through the master process. The parsing for example
> could be done in a completely separate process and if every connected
> framework would be backed by its own process, the check if a framework is
> connected could also be done there (not to mention that this requirement
> exists only because we need to use multiple connections). Requiring all
> messages to go through a single process that can indefinitely block is
> obviously a huge bottleneck. I understand that this problem is not limited
> to the HTTP API, but I think it has to be fixed.
>
> —
> Dario
>
> On Oct 16, 2016, at 5:52 PM, Anand Mazumdar <mazumdar.an...@gmail.com>
> wrote:
>
> Dario,
>
> Regarding:
>
> >This is especially concerning, as it means that accepting calls will
> completely stall when a long running call (e.g. retrieving state.json) is
> running.
>
> How does it help a client when it gets an early accepted response versus
> when accepting of calls is stalled i.e., queued up on the master actor? The
> client does not need to wait for a response before pipelining its next
> request to the master anyway. In your tests, do you send the next REVIVE
> call only upon receiving the response to the current call? That might
> explain the behavior you are seeing.
>
> -anand
>
> On Sun, Oct 16, 2016 at 11:58 AM, tommy xiao <xia...@gmail.com> wrote:
>
>> interesting this topic.
>>
>> 2016-10-17 2:51 GMT+08:00 Dario Rexin <dre...@apple.com>:
>>
>>> Hi Anand,
>>>
>>> I tested with current HEAD. After I saw low throughput on our own HTTP
>>> API client, I wrote a small server that sends out fake events and accepts
>>> calls and our client was able to send a lot more calls to that server. I
>>> also wrote a small tool that simply sends as many calls to Mesos as
>>> possible without handling any events and get similar results there.I also
>>> observe extremely high CPU usage. While my sending tool is using ~10% CPU,
>>> Mesos runs on ~185%. The calls I send for testing are all REVIVE and I
>>> don’t have any agents connected, so there should be essentially nothing
>>> happening. One reason I could think of for the reduced throughput is that
>>> all calls are processed in the master process, before it sends back an
>>> ACCEPTED, leading to effectively single threaded processing of HTTP calls,
>>> interleaved with all other calls that are sent to the master process.
>>> Libprocess however just forwards the messages to the master process and
>>> then immediately  returns ACCEPTED. It also handles all connections in
>>> separate processes, whereas HTTP calls are effectively all handled by the
>>> master process.This is especially concerning, as it means that accepting
>>> calls will completely stall when a long running call (e.g. retrieving
>>> state.json) is running.
>>>
>>> Thanks,
>>> Dario
>>>
>>> On Oct 16, 2016, at 11:01 AM, Anand Mazumdar <an...@apache.org> wrote:
>>>
>>> Dario,
>>>
>>> Thanks for reporting this. Did you test this with 1.0 or the recent
>>> HEAD? We had done performance testing prior to 1.0rc1 and had not found any
>>> substantial discrepancy on the call ingestion path. Hence, we had focussed
>>> on fixing the performance issues around writing events on the stream in
>>> MESOS-5222 <https://issues.apache.org/jira/browse/MESOS-5222> and
>>> MESOS-5457 <https://issues.apache.org/jira/browse/MESOS-5457>.
>>>
>>> The numbers in the benchmark test pointed by Haosdent (v0 vs v1) differ
>>> due to the slowness of the client (scheduler library) in processing the
>>> status update events. We should add another benchmark that measures just
>>> the time taken by the master to write the events. I would file an issue
>>> shortly to address this.
>>>
>>> Do you mind filing an issue with more details on your test setup?
>>>
>>> -anand
>>>
>>> On Sun, Oct 16, 2016 at 12:05 AM, Dario Rexin <dre...@apple.com> wrote:
>>>
>>>> Hi haosdent,
>>>>
>>>> thanks for the pointer! Your results show exactly what I’m
>>>> experiencing. I think especially for bigger clusters this could be very
>>>> problematic. It would be great to get some input from the folks working on
>>>> the HTTP API, especially Anand.
>>>>
>>>> Thanks,
>>>> Dario
>>>>
>>>> On Oct 16, 2016, at 12:01 AM, haosdent <haosd...@gmail.com> wrote:
>>>>
>>>> Hmm, this is an interesting topic. @anandmazumdar create a benchmark
>>>> test case to compare v1 and v0 APIs before. You could run it via
>>>>
>>>> ```
>>>> ./bin/mesos-tests.sh --benchmark --gtest_filter="*SchedulerReco
>>>> ncileTasks_BENCHMARK_Test*"
>>>> ```
>>>>
>>>> Here is the result that run it in my machine.
>>>>
>>>> ```
>>>> [ RUN      ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerLibrary/0
>>>> Reconciling 1000 tasks took 386.451108ms using the scheduler library
>>>> [       OK ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerLibrary/0 (479 ms)
>>>> [ RUN      ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerLibrary/1
>>>> Reconciling 10000 tasks took 3.389258444secs using the scheduler library
>>>> [       OK ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerLibrary/1 (3435 ms)
>>>> [ RUN      ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerLibrary/2
>>>> Reconciling 50000 tasks took 16.624603964secs using the scheduler
>>>> library
>>>> [       OK ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerLibrary/2 (16737 ms)
>>>> [ RUN      ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerLibrary/3
>>>> Reconciling 100000 tasks took 33.134018718secs using the scheduler
>>>> library
>>>> [       OK ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerLibrary/3 (33333 ms)
>>>> [ RUN      ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerDriver/0
>>>> Reconciling 1000 tasks took 24.212092ms using the scheduler driver
>>>> [       OK ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerDriver/0 (89 ms)
>>>> [ RUN      ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerDriver/1
>>>> Reconciling 10000 tasks took 316.115078ms using the scheduler driver
>>>> [       OK ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerDriver/1 (385 ms)
>>>> [ RUN      ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerDriver/2
>>>> Reconciling 50000 tasks took 1.239050154secs using the scheduler driver
>>>> [       OK ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerDriver/2 (1379 ms)
>>>> [ RUN      ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerDriver/3
>>>> Reconciling 100000 tasks took 2.38445672secs using the scheduler driver
>>>> [       OK ] Tasks/SchedulerReconcileTasks_
>>>> BENCHMARK_Test.SchedulerDriver/3 (2711 ms)
>>>> ```
>>>>
>>>> *SchedulerLibrary* is the HTTP API, *SchedulerDriver* is the old way
>>>> based on libmesos.so.
>>>>
>>>> On Sun, Oct 16, 2016 at 2:41 PM, Dario Rexin <dre...@apple.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I recently did some performance testing on the v1 scheduler API and
>>>>> found that throughput is around 10x lower than for the v0 API. Using 1
>>>>> connection, I don’t get a lot more than 1,500 calls per second, where the
>>>>> v0 API can do ~15,000. If I use multiple connections, throughput maxes out
>>>>> at 3 connections and ~2,500 calls / s. If I add any more connections, the
>>>>> throughput per connection drops and the total throughput stays around
>>>>> ~2,500 calls / s. Has anyone done performance testing on the v1 API 
>>>>> before?
>>>>> It seems a little strange to me, that it’s so much slower, given that the
>>>>> v0 API also uses HTTP (well, more or less). I would be thankful for any
>>>>> comments and experience reports of other users.
>>>>>
>>>>> Thanks,
>>>>> Dario
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Haosdent Huang
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> Deshi Xiao
>> Twitter: xds2000
>> E-mail: xiaods(AT)gmail.com
>>
>
>
>
> --
> Anand Mazumdar
>
>
>

Reply via email to