I already did a standalone check of the critical MaxentTagger component. And for 11382 tuples it took 20865 ms, which is approximately 1.83 ms per tuple.
Still if it needs 5 ms per tuple, why do I see 60 ms execute latency in the corresponding bolt [1]? I will try a single bolt topology and see if the execute latency will decrease. [1] https://github.com/millecker/storm-apps/blob/master/commons/src/at/illecker/storm/commons/bolt/POSTaggerBolt.java#L74-93 2015-01-30 15:13 GMT+01:00 Nathan Leung <[email protected]>: > The number of tasks depends. We run the default 4 workers per supervisor, > also on fairly large machines. However, we have more executors than cpu > cores because a lot of our bolts do communications on the network. > > You should serialize with kryo if at all possible. That said I find it > highly doubtful that java serialization is your problem, unless you are > sending a lot of data. And even then if it stays in process it won't get > serialized at all. I would run a standalone check of the performance of the > MaxExt library (which I gather you are using from your other thread). > On Jan 30, 2015 9:07 AM, "Martin Illecker" <[email protected]> wrote: > >> Do you think Java serialization might cause such a huge overhead? >> I definitely have to optimize my software. >> >> By the way does it make sense to increase the number of tasks? >> I don't think so, because they are executed serially. >> >> Are the following assumptions correct? >> >> A good topology config would be one worker per node and one executor for >> each node and each core. >> e.g., two 16-core nodes = 2 worker and max 30 executors (1 acker per >> worker) >> These 30 executors have to be shared between all bolts and spouts. >> >> I think there would be a performance benefit if every worker runs all >> bolts. >> Because if a worker runs only a part of bolts the tuples have to be >> transferred to another worker. >> >> Thanks! >> >> 2015-01-30 13:35 GMT+01:00 Nathan Leung <[email protected]>: >> >>> Assuming you are truly cpu bound and not waiting on io, and 100ms / >>> tuple, each core can do 10 tuples / s. Each node can do 160, and 19 nodes >>> can do 3040 / s. So then you have to optimize your software or add more >>> nodes; it's not a storm issue. >>> >>> If you are doing io and not just purely cpu bound you can add more >>> threads to hide latency and would be able to get higher throughout. >>> On Jan 30, 2015 4:28 AM, "wlqpku" <[email protected]> wrote: >>> >>>> i met the same issue >>>> >>>> Sent from X1 7.0 >>>> >>>> Martin Illecker <[email protected]>编写： >>>> >>>> Hello, >>>> >>>> I'm observing a huge performance problem with my topology. >>>> The topology consists of 5 bolts and two of them are really slow about >>>> 60 ms and 30 ms of execute latency. (please see attached UI screenshot) >>>> >>>> My topology configuration consists of one worker per node and 19 >>>> 16-core nodes, which is a total of 19 workers. >>>> Every worker runs 3 executors, one for each of the three fast bolts, 5 >>>> executors for the 60 ms bolt, 3 executors for the 30ms bolt and 3 hidden >>>> threads (acker, worker buffer receive, worker buffer transfer). >>>> This leads to a total number of 14 threads per worker, which is the >>>> upper limit for a 16-core nodes. In my case the CPU is the limiting factor. >>>> >>>> With this configuration I could only measure 2000 to 3000 tuples per >>>> second at the end of this pipeline. I have tried multiple MaxSpoutPending >>>> settings but 2000 to 3000 tuples per second seem to be the maximum. >>>> I think, this might be the upper limit for a total execution latency of >>>> 100ms? >>>> >>>> How can I achieve x0,000 tuples for my topology? Or is this impossible >>>> with these two slow bolts? >>>> Is there anything I could try because horizontal scaling will not solve >>>> my problem. >>>> >>>> Thanks! >>>> >>>> Best regards >>>> Martin >>>> >>>> >>>> >>>> >>>> >>
