Re: [Bro-Dev] Broker raw throughput

2016-03-09 Thread Matthias Vallentin
> The optimization I meant is to not wrap each integer in its own > message object, but rather make one message which then contains X > integers that are transparently interpreted by the receiver as X > messages. But this requires some form of "output queue" or lookahead > mechanism. I can see

Re: [Bro-Dev] Broker raw throughput

2016-03-09 Thread Dominik Charousset
>> You could additionally try to tweak >> caf#middleman.max_consecutive_reads, which configures how many >> new_data_msg messages a broker receives from the backend in a single >> shot. It makes sense to have the two separated, because one configures >> fairness in the scheduling and the other

Re: [Bro-Dev] Broker raw throughput

2016-03-08 Thread Matthias Vallentin
> You could additionally try to tweak > caf#middleman.max_consecutive_reads, which configures how many > new_data_msg messages a broker receives from the backend in a single > shot. It makes sense to have the two separated, because one configures > fairness in the scheduling and the other fairness

Re: [Bro-Dev] Broker raw throughput

2016-03-08 Thread Dominik Charousset
> the benchmark no longer > terminates and the server quickly stops getting data, and I would like > to know why. I'll have a look at it. > I've tried various parameters for the scheduler throughput, but they do > not seem to make a difference. Would you mind taking a look at what's > going on

Re: [Bro-Dev] Broker raw throughput

2016-03-07 Thread Matthias Vallentin
> I have created a ticket for further progress tracking / discussion [1] > as this is clearly not a Bro/Broker problem. Thank you all for > reporting this and all the input you have provided. It's good to see the new commit improves performance. But I want to take again the perspective of Broker,

Re: [Bro-Dev] Broker raw throughput

2016-03-02 Thread Matthias Vallentin
> @Matthias: FYI, I have used a new feature in CAF that allows senders > to get feedback from the I/O layer for not overloading it. This allows > the sender to adapt to the send rate of the network. Great, it sounds like this would fix the stall/hang issues. I expect to port Broker to the

Re: [Bro-Dev] Broker raw throughput

2016-03-02 Thread Dominik Charousset
With most noise like serialization etc. out of the way, this is what I measured on Linux:native sender -> native receiver567520085 Bytes/sCAF sender -> native receiver511333973 Bytes/snative sender -> CAF receiver229689173 Bytes/sCAF sender -> CAF receiver222102755 Bytes/sSend performance is OK,

Re: [Bro-Dev] Broker raw throughput

2016-03-01 Thread Dominik Charousset
Thanks for providing build scripts and sharing results. Just a quick heads-up from me: I have implemented a simple sender/receiver pair using C sockets as well as CAF brokers (attached, but works only with the current actor-system topic branch). Both sending and receiving are slower with CAF

Re: [Bro-Dev] Broker raw throughput

2016-02-25 Thread Matthias Vallentin
For better reproducibility, here's the Makefile that I used to drive the experiments: CC = cc CXX = c++ FLAGS = -O3 -g -std=c++11 -stdlib=libc++ LIBS = -lcaf_core -lcaf_io -ltcmalloc -lprofiler caf-client: caf-client.cpp $(CXX) $(FLAGS) $< -o $@ $(LIBS)

[Bro-Dev] Broker raw throughput

2016-02-24 Thread Matthias Vallentin
I'm taking a discussion about Broker performance public which we previously had on our internal mailing list. Below, I'm pasting in Dominik's answer to my email. Dominik, we've white-listed your email address on bro-dev, but free to subscribe there to get future responses. Apologies for the