| | | Bela, I'm pretty sure these tests use UDP. I'd be really surprised if | we could improve TCP performance by lowering max_credits.
True, they do. | | We do have a JIRA to change the state transfer behaviour to request | state from only a few nodes at a time (perhaps only 1): | https://issues.jboss.org/browse/ISPN-2580 . Adrian is working on it | ATM, and once it's integrated it would make UUPerf performance | largely irrelevant. I don't think so, I expect that e.g. 3 nodes to make ST from is perfectly reasonable scenario and as these tests are ran with 4 nodes, this is the case. | | Even if Adrian's fix doesn't make it into Final, I think a | max_credits of only 20k would impact performance in the "stable | state" (i.e. what UPerf is testing). So maybe we can find a | workaround, like lowering Infinispan's stateTransfer.chunkSize. Yeah, I have used 10MB messages for testing, I should do that for smaller ones as well. | | I wonder if we could automate UPerf and UUPerf, like RadarGun does | (or maybe make them RadarGun test scenarios?), so we can gather more | data points. At the moment there's a lot of manual work involved in | running the tests with all the possible configurations | (TCP/UNICAST2, TCP/UNICAST2/UFC, UDP/UNICAST, UDP/UNICAST/UFC, | UDP/UNICAST2/UFC, UDP/UNICAST2/UFC/RSVP, each protocol with several | tweak-able attributes) and figuring out which configuration is | "best". This sounds good, using JGroups cachewrapper I could just do GET on one slave in a loop, right? The only modification required is that the JGroupsWrapper.get should do dispatcher.callRemoteMethods(...) with all members instead of just single invocation. And maybe the I think I could grab some time for this next week. Radim | | | | | On Thu, Jan 3, 2013 at 12:42 PM, Bela Ban < [email protected] > wrote: | | | Let's make sure though that we have a meaningful default that's not | optimized for an edge case. Also, if we use TCP, we can remove UFC | from | the config, as TCP already performs point-to-point flow control. | | | | On 1/3/13 11:29 AM, Radim Vansa wrote: | > 20k credits seems to be the best choice for this test: | > | > 10k: bad performance | > 20k: Average of 2.79 requests / sec (27.87MB / sec), 358.81 ms | > /request (prot=UNICAST2) | > 30k: Average of 2.52 requests / sec (25.18MB / sec), 397.15 ms | > /request (prot=UNICAST2) | > 50k: Average of 2.35 requests / sec (23.47MB / sec), 426.10 ms | > /request (prot=UNICAST2) | > 80k: Average of 1.29 requests / sec (12.94MB / sec), 772.78 ms | > /request (prot=UNICAST2) | > 200k: bad performance | > | > (for remembrance: 4 nodes in hyperion, for these results I've set | > up 8k frag size) | > | > I have held dot key for the duration of the test so you can see how | > long each apply state took as the dots were inserted into console | > in constant rate (lame ascii chart). See attachements. | > | > Radim | > | > ----- Original Message ----- | > | From: "Dan Berindei" < [email protected] > | > | To: "infinispan -Dev List" < [email protected] > | > | Sent: Monday, December 24, 2012 8:01:26 AM | > | Subject: Re: [infinispan-dev] MFC/UFC credits in default config | > | | > | | > | | > | | > | This is weird, I would have expected problems with the last | > | message, | > | but not in the middle of the sequence (that's why I suggested | > | sending only 1 message). Maybe we need an an even lower | > | max_credits... | > | | > | Merry Christmas to you, too! | > | | > | Dan | > | On 21 Dec 2012 16:41, "Radim Vansa" < [email protected] > wrote: | > | | > | | > | Hi Dan, | > | | > | I have ran the test on 4 nodes in hyperion (just for the start to | > | see | > | how it will behave) but with 100 messages (1 message is nothing | > | for | > | a statistician) each 10MB and I see a weird behaviour - there are | > | about 5-10 messages received in a fast succession and then the | > | nothing is received for several seconds. I experience this | > | behaviour | > | for both 200k and 500k credits. Is this really how it should | > | perform? | > | | > | Merry Christmas and tons of snow :) | > | | > | Radim | > | | > | <h1>☃</h1> | > | | > | ----- Original Message ----- | > | | From: "Dan Berindei" < [email protected] > | > | | To: "infinispan -Dev List" < [email protected] > | > | | Sent: Tuesday, December 18, 2012 8:57:08 AM | > | | Subject: Re: [infinispan-dev] MFC/UFC credits in default config | > | | | > | | | > | | Hi Radim | > | | | > | | If you run the test with only 2 nodes and FC disabled, it's | > | | going | > | | to | > | | perform even better. But then as you increase the number of | > | | nodes, | > | | the speed with no FC will drop dramatically (when we didn't | > | | have | > | | RSVP enabled, with only 3 nodes, it didn't manage to send 1 x | > | | 10MB | > | | message in 10 minutes). | > | | | > | | Please run the tests with as many nodes as possible and just 1 | > | | message x 10MB. If 500k still performs better, create a JIRA to | > | | change the default. | > | | | > | | Cheers | > | | Dan | > | | | > | | | > | | | > | | | > | | | > | | On Mon, Dec 17, 2012 at 4:55 PM, Radim Vansa < | > | | [email protected] > | > | | wrote: | > | | | > | | | > | | Sorry I haven't specified the amount, I am a stupido... my | > | | tests | > | | are | > | | working with 500k credits. | > | | | > | | UUPerf (JGroups 3.2.4.Final-redhat-1) from one computer in | > | | perflab | > | | to | > | | another, 2 threads (default), 1000x sends 10MB message (default | > | | chunkSize = 10000 * our entry size is usually 1kB) executed 3x | > | | | > | | 200k: Average of 6.02 requests / sec (60.19MB / sec), 166.13 ms | > | | /request (prot=UNICAST2) | > | | Average of 5.61 requests / sec (56.09MB / sec), 178.30 ms | > | | /request | > | | (prot=UNICAST2) | > | | Average of 5.49 requests / sec (54.94MB / sec), 182.03 ms | > | | /request | > | | (prot=UNICAST2) | > | | | > | | 500k: Average of 7.93 requests / sec (79.34MB / sec), 126.04 ms | > | | /request (prot=UNICAST2) | > | | Average of 8.18 requests / sec (81.82MB / sec), 122.23 ms | > | | /request | > | | (prot=UNICAST2) | > | | Average of 8.41 requests / sec (84.09MB / sec), 118.92 ms | > | | /request | > | | (prot=UNICAST2) | > | | | > | | Can you also reproduce such results? I think that suggests that | > | | 500k | > | | behaves really better. | > | | | > | | Radun | > | | | > | | | > | | | > | | | > | | ----- Original Message ----- | > | | | From: "Dan Berindei" < [email protected] > | > | | | To: "infinispan -Dev List" < [email protected] > | > | | | Sent: Monday, December 17, 2012 12:43:37 PM | > | | | Subject: Re: [infinispan-dev] MFC/UFC credits in default | > | | | config | > | | | | > | | | | > | | | | > | | | | > | | | | > | | | On Mon, Dec 17, 2012 at 1:28 PM, Bela Ban < [email protected] > | > | | | wrote: | > | | | | > | | | | > | | | Dan reduced those values to 200K, IIRC it was for UUPerfwhich | > | | | behaved | > | | | best with 200K. Idon't know if this is still needed. Dan ? | > | | | | > | | | | > | | | | > | | | | > | | | I haven't run UUPerf in a while... | > | | | | > | | | | > | | | | > | | | | > | | | On 12/17/12 12:19 PM, Radim Vansa wrote: | > | | | > Hi, | > | | | > | > | | | > recently I have synchronized our jgroups configuration with | > | | | > the | > | | | > default one shipped with Infinispan | > | | | > (core/src/main/resources/jgroups-(tcp|udp).xml) and it has | > | | | > shown | > | | | > that 200k credits in UFC/MFC (I keep the two values in | > | | | > sync) is | > | | | > not enough even for our smallest resilience test (killing | > | | | > one | > | | | > of | > | | | > four nodes). The state transfer was often blocked when | > | | | > requesting | > | | | > for more credits which resulted in not completing it within | > | | | > the | > | | | > time limit. | > | | | > Therefore, I'd like to suggest to increase the amount of | > | | | > credits | > | | | > in | > | | | > default configuration as well, because we simply cannot use | > | | | > the | > | | | > lower setting and it's preferable to have the | > | | | > configurations as | > | | | > close as possible. The only settings we need to keep | > | | | > different | > | | | > are | > | | | > thread pool sizes and addresses and ports. | > | | | > | > | | | | > | | | | > | | | What value would you like to use instead? | > | | | | > | | | Can you try UUPerf with 200k and your proposed configuration | > | | | and | > | | | compare the results? | > | | | | > | | | Cheers | > | | | Dan | > | | | | > | | | | > | | | > | | | > | | | _______________________________________________ | > | | | infinispan-dev mailing list | > | | | [email protected] | > | | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | | _______________________________________________ | > | | infinispan-dev mailing list | > | | [email protected] | > | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | | | > | | | > | | _______________________________________________ | > | | infinispan-dev mailing list | > | | [email protected] | > | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | | > | _______________________________________________ | > | infinispan-dev mailing list | > | [email protected] | > | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | _______________________________________________ | > | infinispan-dev mailing list | > | [email protected] | > | https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | > | > _______________________________________________ | > infinispan-dev mailing list | > [email protected] | > https://lists.jboss.org/mailman/listinfo/infinispan-dev | | | -- | Bela Ban, JGroups lead ( http://www.jgroups.org ) | | | | _______________________________________________ | infinispan-dev mailing list | [email protected] | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | _______________________________________________ | infinispan-dev mailing list | [email protected] | https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list [email protected] https://lists.jboss.org/mailman/listinfo/infinispan-dev
