Re: [infinispan-dev] The need for a 5.1.1

2012-01-27 Thread Manik Surtani
ISPN-1786 is related to a potential bug in the config parser that didn't pick up numVirtualNodes settings. As well as a proposal for a different default value for numVirtualNodes - which really should be a separate JIRA. On 27 Jan 2012, at 07:03, Bela Ban wrote: Regarding ISPN-1786, I'd like

Re: [infinispan-dev] default value for virtualNodes

2012-01-27 Thread Manik Surtani
Good stuff! Thanks for this. Yes, I'm ok with numVirtualNodes=48 as a default. Galder, your thoughts from a Hot Rod perspective? On 27 Jan 2012, at 08:41, Dan Berindei wrote: Hi guys I've been working on a test to search for an optimal default value here:

Re: [infinispan-dev] default value for virtualNodes

2012-01-27 Thread Bela Ban
I assume the number of vnodes cannot be changed at runtime, dynamically adapting to a changing environment ? I understand everybody has to have the exact same number of vnodes for reads and writes to hit the correct node, right ? On 1/27/12 9:41 AM, Dan Berindei wrote: Hi guys I've been

Re: [infinispan-dev] default value for virtualNodes

2012-01-27 Thread Manik Surtani
On 27 Jan 2012, at 10:52, Bela Ban wrote: I assume the number of vnodes cannot be changed at runtime, dynamically adapting to a changing environment ? I understand everybody has to have the exact same number of vnodes for reads and writes to hit the correct node, right ? Yes. -- Manik

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Mircea Markus
On 26 Jan 2012, at 23:04, Sanne Grinovero wrote: Very nice! All my previous tests also confirm that there is a correlation between PUT and GET performance, when one increases the other goes down. These PUT operations are doing a GET as well, correct? I'd love to see such graphs using

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Manik Surtani
On 27 Jan 2012, at 11:07, Mircea Markus wrote: These PUT operations are doing a GET as well, correct? I'd love to see such graphs using SKIP_REMOTE_LOOKUP. it is configured with unsafe return values. With safe return, the values might get even better… Eh? Why would safe return values

Re: [infinispan-dev] DIST.retrieveFromRemoteSource

2012-01-27 Thread Manik Surtani
On 25 Jan 2012, at 08:51, Dan Berindei wrote: Slightly related, I wonder if Manik's comment is still true: if at all possible, try not to use JGroups' ANYCAST for now. Multiple (parallel) UNICASTs are much faster.) Intuitively it shouldn't be true, unicasts+FutureCollator do basically

Re: [infinispan-dev] DIST.retrieveFromRemoteSource

2012-01-27 Thread Manik Surtani
On 25 Jan 2012, at 09:42, Bela Ban wrote: No, parallel unicasts will be faster, as an anycast to A,B,C sends the unicasts sequentially Is this still the case in JG 3.x? -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org

Re: [infinispan-dev] DIST.retrieveFromRemoteSource

2012-01-27 Thread Manik Surtani
On 25 Jan 2012, at 17:09, Dan Berindei wrote: Keep in mind that we also want to introduce eventual consistency - I think that's going to eliminate our optimization opportunity here because we'll need to get the values from a majority of owners (if not all the owners). Also keep in mind

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Sanne Grinovero
On 27 January 2012 11:07, Mircea Markus mircea.mar...@jboss.com wrote: On 26 Jan 2012, at 23:04, Sanne Grinovero wrote: Very nice! All my previous tests also confirm that there is a correlation between PUT and GET performance, when one increases the other goes down. These PUT operations

Re: [infinispan-dev] DIST.retrieveFromRemoteSource

2012-01-27 Thread Manik Surtani
On 25 Jan 2012, at 17:09, Dan Berindei wrote: I think we already have a JIRA to make PutKeyValueCommands return the previous value, that would eliminate lots of GetKeyValueCommands and it would actually improve the performance of puts - we should probably make this a priority. Yes, this is

Re: [infinispan-dev] DIST.retrieveFromRemoteSource

2012-01-27 Thread Bela Ban
yes. On 1/27/12 12:13 PM, Manik Surtani wrote: On 25 Jan 2012, at 09:42, Bela Ban wrote: No, parallel unicasts will be faster, as an anycast to A,B,C sends the unicasts sequentially Is this still the case in JG 3.x? -- Bela Ban Lead JGroups (http://www.jgroups.org) JBoss / Red Hat

Re: [infinispan-dev] DIST.retrieveFromRemoteSource

2012-01-27 Thread Manik Surtani
On 25 Jan 2012, at 14:22, Mircea Markus wrote: Agreed that having a configurable remote get policy makes sense. We already have a JIRA for this[1], I'll start working on it as the performance results are hunting me. I'd like to have Dan's input on this as well first, as he has worked with

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Bela Ban
10 is an awfully small value; in my experience (I had a default of 100 for my JGroups perf tests), this made the tests longer than with the default (which is 1 IIRC) ! On 1/27/12 12:20 PM, Sanne Grinovero wrote: On 27 January 2012 11:07, Mircea Markusmircea.mar...@jboss.com wrote: On

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Mircea Markus
Are both reads and writes marked as OOB ? Then they share the same OOB thread pool ! We do mark reads as OOB (DistributionManagerImpl.retrieveFromRemoteSource). So reads and writes share the same OOB pool. I was looking an non-trnasactional puts, and these are not OOB. This benchmark

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Mircea Markus
On 27 Jan 2012, at 10:59, Manik Surtani wrote: On 26 Jan 2012, at 23:04, Sanne Grinovero wrote: How long are you warming up the VM? As mentioned in the other thread, I've discovered that even under high load it will take more than 15 minutes before all of Infinispan's code is running

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Sanne Grinovero
On 27 January 2012 11:37, Mircea Markus mircea.mar...@jboss.com wrote: On 27 Jan 2012, at 10:59, Manik Surtani wrote: On 26 Jan 2012, at 23:04, Sanne Grinovero wrote: How long are you warming up the VM? As mentioned in the other thread, I've discovered that even under high load it will

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Mircea Markus
On 27 Jan 2012, at 11:20, Sanne Grinovero wrote: On 27 January 2012 11:07, Mircea Markus mircea.mar...@jboss.com wrote: On 26 Jan 2012, at 23:04, Sanne Grinovero wrote: Very nice! All my previous tests also confirm that there is a correlation between PUT and GET performance, when one

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Mircea Markus
On 27 Jan 2012, at 11:36, Bela Ban wrote: 10 is an awfully small value; in my experience (I had a default of 100 for my JGroups perf tests), this made the tests longer than with the default (which is 1 IIRC) ! default is 1500:

Re: [infinispan-dev] Looking into OProfile logs

2012-01-27 Thread Manik Surtani
Also, lets please move this discussion to infinispan-dev… On 25 Jan 2012, at 17:57, Dan Berindei wrote: On Wed, Jan 25, 2012 at 6:32 PM, Sanne Grinovero sa...@infinispan.org wrote: On 25 January 2012 15:56, Dan Berindei dan.berin...@gmail.com wrote: On Wed, Jan 25, 2012 at 3:16 PM, Sanne

Re: [infinispan-dev] default value for virtualNodes

2012-01-27 Thread Mircea Markus
I've created a JIRA to track this: https://issues.jboss.org/browse/ISPN-1801 I understand everybody has to have the exact same number of vnodes for reads and writes to hit the correct node, right ? Yes. That's true, but it is not a good thing: numVirtNodes should be proportional with the

Re: [infinispan-dev] default value for virtualNodes

2012-01-27 Thread Dan Berindei
On Fri, Jan 27, 2012 at 2:35 PM, Mircea Markus mircea.mar...@jboss.com wrote: I've created a JIRA to track this: https://issues.jboss.org/browse/ISPN-1801 I understand everybody has to have the exact same number of vnodes for reads and writes to hit the correct node, right ? Yes. That's

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Sanne Grinovero
My experiments where using the default JVM settings regarding compile settings, with these others: -Xmx2G -Xms2G -XX:MaxPermSize=128M -XX:+HeapDumpOnOutOfMemoryError -Xss512k -XX:HeapDumpPath=/tmp/java_heap -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Mircea Markus
On 27 Jan 2012, at 13:31, Sanne Grinovero wrote: My experiments where using the default JVM settings regarding compile settings, with these others: -Xmx2G -Xms2G -XX:MaxPermSize=128M -XX:+HeapDumpOnOutOfMemoryError -Xss512k -XX:HeapDumpPath=/tmp/java_heap -Djava.net.preferIPv4Stack=true

Re: [infinispan-dev] The need for a 5.1.1

2012-01-27 Thread Mircea Markus
On 26 Jan 2012, at 22:42, Manik Surtani wrote: I really didn't want to do this, but it looks like a 5.1.1 will be necessary. The biggest (critical, IMO, for 5.1.1) issues I see are: 1. https://issues.jboss.org/browse/ISPN-1786 - I presume this has to do with a bug Mircea spotted that

Re: [infinispan-dev] default value for virtualNodes

2012-01-27 Thread Dan Berindei
On Fri, Jan 27, 2012 at 2:53 PM, Mircea Markus mircea.mar...@jboss.com wrote: That's true, but it is not a good thing: numVirtNodes should be proportional with the node's capacity, i.e. more powerful machines in the cluster should have assigned more virtual nodes. This way we can better

Re: [infinispan-dev] The need for a 5.1.1

2012-01-27 Thread Bela Ban
The branch is JGRP-1417 and the config for UNICAST2 is: UNICAST2 max_bytes=20M xmit_table_num_rows=20 xmit_table_msgs_per_row=1 xmit_table_max_compaction_time=1 max_msg_batch_size=100/ I've attached the config I've

Re: [infinispan-dev] The need for a 5.1.1

2012-01-27 Thread Manik Surtani
On 27 Jan 2012, at 14:09, Mircea Markus wrote: On 26 Jan 2012, at 22:42, Manik Surtani wrote: I really didn't want to do this, but it looks like a 5.1.1 will be necessary. The biggest (critical, IMO, for 5.1.1) issues I see are: 1. https://issues.jboss.org/browse/ISPN-1786 - I presume

Re: [infinispan-dev] The need for a 5.1.1

2012-01-27 Thread Mircea Markus
There's an initialisation error indicating a dependency on a test class[1]. Seems like jgroups is not jar-less anymore :-) [1] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:236) at

Re: [infinispan-dev] The need for a 5.1.1

2012-01-27 Thread Mircea Markus
On 27 Jan 2012, at 15:08, Bela Ban wrote: Build the JGroups JAR with ./build.sh jar, *not* via maven ! I attached the JAR for you. Thanks! JGroups *is* and *will remain* JAR less ! :-) Sorry for loosing the faith :) Might make sense to have the mvn install work as well though, I think

Re: [infinispan-dev] The need for a 5.1.1

2012-01-27 Thread Bela Ban
On 1/27/12 4:26 PM, Mircea Markus wrote: On 27 Jan 2012, at 15:08, Bela Ban wrote: Build the JGroups JAR with ./build.sh jar, *not* via maven ! I attached the JAR for you. Thanks! JGroups *is* and *will remain* JAR less ! :-) Sorry for loosing the faith :) Might make sense to have the

Re: [infinispan-dev] very interesting performance results

2012-01-27 Thread Dan Berindei
On Fri, Jan 27, 2012 at 3:43 PM, Mircea Markus mircea.mar...@jboss.com wrote: On 27 Jan 2012, at 13:31, Sanne Grinovero wrote: My experiments where using the default JVM settings regarding compile settings, with these others: -Xmx2G -Xms2G -XX:MaxPermSize=128M -XX:+HeapDumpOnOutOfMemoryError

Re: [infinispan-dev] The need for a 5.1.1

2012-01-27 Thread Manik Surtani
Branch created. https://github.com/infinispan/infinispan/tree/5.1.x Of the JIRAs I mentioned below, if they have been committed to master I'll cherry pick them onto 5.1.x as well. If they haven't been completed, I'll change their target accordingly, please make sure you create pull reqs for

Re: [infinispan-dev] DIST.retrieveFromRemoteSource

2012-01-27 Thread Dan Berindei
Manik, Bela, I think we send the requests sequentially as well. In ReplicationTask.call: for (Address a : targets) { NotifyingFutureObject f = sendMessageWithFuture(constructMessage(buf, a), opts); futureCollator.watchFuture(f, a);

Re: [infinispan-dev] DIST.retrieveFromRemoteSource

2012-01-27 Thread Manik Surtani
Doesn't setBlockForResults(false) mean that we're not waiting on a response, and can proceed to the next message to the next recipient? On 27 Jan 2012, at 16:34, Dan Berindei wrote: Manik, Bela, I think we send the requests sequentially as well. In ReplicationTask.call: for

Re: [infinispan-dev] Write Skew issue (versioning)

2012-01-27 Thread Mircea Markus
Looks like a bug, mind creating a JIRA for it? On 24 Jan 2012, at 21:45, Pedro Ruivo wrote: Hi, yes I have the versioning enabled. Like you said, I've posted in the forum too [1]. btw, the ISPN config is here [2] [1] -- https://community.jboss.org/thread/177846 [2] --

Re: [infinispan-dev] The need for a 5.1.1

2012-01-27 Thread Dan Berindei
On Fri, Jan 27, 2012 at 5:31 PM, Bela Ban b...@redhat.com wrote: On 1/27/12 4:26 PM, Mircea Markus wrote: On 27 Jan 2012, at 15:08, Bela Ban wrote: Build the JGroups JAR with ./build.sh jar, *not* via maven ! I attached the JAR for you. Thanks! JGroups *is* and *will remain* JAR less !

Re: [infinispan-dev] Write Skew issue (versioning)

2012-01-27 Thread Manik Surtani
I'm taking a look - started a discussion on the forums. :) On 27 Jan 2012, at 16:44, Mircea Markus wrote: Looks like a bug, mind creating a JIRA for it? On 24 Jan 2012, at 21:45, Pedro Ruivo wrote: Hi, yes I have the versioning enabled. Like you said, I've posted in the forum too [1].