I highly recommend using a profiler to see what's using up the CPU in your workers. I've used YourKit to great success in profiling Storm.
On Fri, Jan 30, 2015 at 9:38 AM, Martin Illecker <[email protected]> wrote: > I already did a standalone check of the critical MaxentTagger component. > And for 11382 tuples it took 20865 ms, which is approximately 1.83 ms per > tuple. > > Still if it needs 5 ms per tuple, why do I see 60 ms execute latency in > the corresponding bolt [1]? > > I will try a single bolt topology and see if the execute latency will > decrease. > > [1] > https://github.com/millecker/storm-apps/blob/master/commons/src/at/illecker/storm/commons/bolt/POSTaggerBolt.java#L74-93 > > 2015-01-30 15:13 GMT+01:00 Nathan Leung <[email protected]>: > >> The number of tasks depends. We run the default 4 workers per supervisor, >> also on fairly large machines. However, we have more executors than cpu >> cores because a lot of our bolts do communications on the network. >> >> You should serialize with kryo if at all possible. That said I find it >> highly doubtful that java serialization is your problem, unless you are >> sending a lot of data. And even then if it stays in process it won't get >> serialized at all. I would run a standalone check of the performance of the >> MaxExt library (which I gather you are using from your other thread). >> On Jan 30, 2015 9:07 AM, "Martin Illecker" <[email protected]> wrote: >> >>> Do you think Java serialization might cause such a huge overhead? >>> I definitely have to optimize my software. >>> >>> By the way does it make sense to increase the number of tasks? >>> I don't think so, because they are executed serially. >>> >>> Are the following assumptions correct? >>> >>> A good topology config would be one worker per node and one executor for >>> each node and each core. >>> e.g., two 16-core nodes = 2 worker and max 30 executors (1 acker per >>> worker) >>> These 30 executors have to be shared between all bolts and spouts. >>> >>> I think there would be a performance benefit if every worker runs all >>> bolts. >>> Because if a worker runs only a part of bolts the tuples have to be >>> transferred to another worker. >>> >>> Thanks! >>> >>> 2015-01-30 13:35 GMT+01:00 Nathan Leung <[email protected]>: >>> >>>> Assuming you are truly cpu bound and not waiting on io, and 100ms / >>>> tuple, each core can do 10 tuples / s. Each node can do 160, and 19 nodes >>>> can do 3040 / s. So then you have to optimize your software or add more >>>> nodes; it's not a storm issue. >>>> >>>> If you are doing io and not just purely cpu bound you can add more >>>> threads to hide latency and would be able to get higher throughout. >>>> On Jan 30, 2015 4:28 AM, "wlqpku" <[email protected]> wrote: >>>> >>>>> i met the same issue >>>>> >>>>> Sent from X1 7.0 >>>>> >>>>> Martin Illecker <[email protected]>编写： >>>>> >>>>> Hello, >>>>> >>>>> I'm observing a huge performance problem with my topology. >>>>> The topology consists of 5 bolts and two of them are really slow about >>>>> 60 ms and 30 ms of execute latency. (please see attached UI screenshot) >>>>> >>>>> My topology configuration consists of one worker per node and 19 >>>>> 16-core nodes, which is a total of 19 workers. >>>>> Every worker runs 3 executors, one for each of the three fast bolts, 5 >>>>> executors for the 60 ms bolt, 3 executors for the 30ms bolt and 3 hidden >>>>> threads (acker, worker buffer receive, worker buffer transfer). >>>>> This leads to a total number of 14 threads per worker, which is the >>>>> upper limit for a 16-core nodes. In my case the CPU is the limiting >>>>> factor. >>>>> >>>>> With this configuration I could only measure 2000 to 3000 tuples per >>>>> second at the end of this pipeline. I have tried multiple MaxSpoutPending >>>>> settings but 2000 to 3000 tuples per second seem to be the maximum. >>>>> I think, this might be the upper limit for a total execution latency >>>>> of 100ms? >>>>> >>>>> How can I achieve x0,000 tuples for my topology? Or is this impossible >>>>> with these two slow bolts? >>>>> Is there anything I could try because horizontal scaling will not >>>>> solve my problem. >>>>> >>>>> Thanks! >>>>> >>>>> Best regards >>>>> Martin >>>>> >>>>> >>>>> >>>>> >>>>> >>> > -- Twitter: @nathanmarz http://nathanmarz.com
