Re: [infinispan-dev] Infinispan xsite perf test demo
On 15 Dec 2012, at 09:54, Bela Ban b...@redhat.com wrote: When pointing out the different IP addresses you say something about needing them to be different, but I didn't understand if you needed them different because they are in different places, or to separate otherwise local machines to have them appear as in different places. The reason to separate mcast_addr/mcast_port was to simulate 3 sites on the same box. Had I not used different addresses/ports for the 3 sites, all nodes of the 3 sites would have found each other and formed a cluster of 9. Maybe for more impact next time, use AWS and set up servers in different availability zones? Or better still, one cluster in our Boston lab, one on AWS, and one on Rackspace. This would definitely have impact, as you can start a demo by showing the nodes in the AWS and Rackspace consoles. :) -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Infinispan xsite perf test demo
I don't do AWS anymore, killed my account last year On 12/19/12 1:37 PM, Manik Surtani wrote: On 15 Dec 2012, at 09:54, Bela Ban b...@redhat.com wrote: When pointing out the different IP addresses you say something about needing them to be different, but I didn't understand if you needed them different because they are in different places, or to separate otherwise local machines to have them appear as in different places. The reason to separate mcast_addr/mcast_port was to simulate 3 sites on the same box. Had I not used different addresses/ports for the 3 sites, all nodes of the 3 sites would have found each other and formed a cluster of 9. Maybe for more impact next time, use AWS and set up servers in different availability zones? Or better still, one cluster in our Boston lab, one on AWS, and one on Rackspace. This would definitely have impact, as you can start a demo by showing the nodes in the AWS and Rackspace consoles. :) -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Bela Ban, JGroups lead (http://www.jgroups.org) ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Infinispan xsite perf test demo
| | 3# Since get operations are always local (site), they are as you say | not meaningful for the benchmark; now since put operations are also | not meaningful as it's async .. what is the benchmark measuring? | That's an assumption and the test is there also to confirm it. For example in my XS perf test with async XS replication the reads are slowed down for some reasons as well (investigating). And writes have definitely some overhead for XS too, at least for the marshalling etc. Radim ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Infinispan xsite perf test demo
Thanks Ray, yes, each site has the other 2as backup sites. The current assumption though is for xsite replication to only be unidirectional, e.g. LON -- SFO/NYC, or if it is bidirectional, for the data that's concurrently accessed to be disjoint. This setup allows LON to be the active site and SFO/NYC to be the backup sites. One could imagine a follow-the-sun implementation where LON becomes inactive and NYC becomes active when it is EOB in LON. Later, SFO becomes active, then LON again and so on. In IRAC [1], we'll handle the problem of concurrent updates from different sites to the same data set. [1] https://community.jboss.org/wiki/RAC On 12/15/12 4:41 AM, Ray Tsang wrote: Great stuff Bela! Btw, is xsite replication bidirectional? LON has SFO as backup, and SFO also has LON as backup. On Fri, Dec 14, 2012 at 8:01 PM, Sanne Grinovero sa...@infinispan.org mailto:sa...@infinispan.org wrote: That was very very nice to see. Assuming you also asked for feedback to improve this as a talk: 1# you stress several times that reads are going to be local, so very fast. I think you meant local to the site ? as some ~33% of entries will need to be fetched from peers on the same site. 2# you aren't actually running this on multiple sites are you? When pointing out the different IP addresses you say something about needing them to be different, but I didn't understand if you needed them different because they are in different places, or to separate otherwise local machines to have them appear as in different places. 3# Since get operations are always local (site), they are as you say not meaningful for the benchmark; now since put operations are also not meaningful as it's async .. what is the benchmark measuring? 4# There seems to be some degree of redundancy when explaining LON/SFO/NYC setting as the local site vs the backup sites. Wouldn't it make more sense to be able to configure all backup sites the same and have it automatically ignore the self element as a backup site? So your script would only need to specify what the local site is. If that makes any sense it would even be nice to extend this to the IP addresses being defined in the zones area, so that they are applied both to the JGroups configuration for the local cluster and to the bridge configuration. 5# I was initially surprised to see x-site configuration as part of a cache configuration; I understand the reasons for options like strategy which one might want to specify differently on each cache, but what about take offline ? that sounds more something which should be globally managed at the channel level - not sure if in JGroups directly but if it's to be handled in Infinispan I would expect to have all caches use the same policy, consistent with FD. Also it doesn't looks like you have much of a choice in to which sites you want to replicate, as relay is setup at the jgroups level so affecting all caches: is relay going to be ignored by caches having no x-site enabled? And is it going to be relayed only to one site if the Infinispan configuration lists a single site? Not sure if this makes any sense, I just found it contrasting with my naive expectations of how such a configuration would look like. thanks a lot, I hope this is proof enough that your video was pretty catchy :) Cheers, Sanne On 14 December 2012 11:47, Radoslav Husar rhu...@redhat.com mailto:rhu...@redhat.com wrote: Thanks Bela, I enjoyed the demo! I suggest adding some sort of visualization (maybe just a few diagrams within the demo) of what is actually happening with the data would help understanding for users who are just starting with xsite. Rado On 14/12/12 12:09, Bela Ban wrote: FYI, I've uploaded a 10-minute video [1] to YouTube showing how I setup and run a perf test for Infinispan xsite replication [2]. The test (including all configuration) is available at [3]. This was run with a snapshot of Infinispan (roughly 5.2.0.Beta6) and JGroups 3.3.0.Alpha1. Enjoy ! [1] http://www.youtube.com/watch?v=owOs430vLZo [2] https://docs.jboss.org/author/display/ISPN/Cross+site+replication [3] https://github.com/belaban/IspnPerfTest ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org mailto:infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org mailto:infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Infinispan xsite perf test demo
Hi Sanne, thanks ! My first YouTube video, so next time I might venture into editing the video... :-) The video was initially meant to show the perf when adding 2 backup sites, but then (because I had to include config info) 'degenerated' into a tutorial on xsite. I know, bad job on that, but we should come up with a tutorial that does this in a much more comprehensive way... Comments inline On 12/15/12 2:01 AM, Sanne Grinovero wrote: That was very very nice to see. Assuming you also asked for feedback to improve this as a talk: 1# you stress several times that reads are going to be local, so very fast. I think you meant local to the site ? as some ~33% of entries will need to be fetched from peers on the same site. Yes, 'local' always refers to the local site (cluster). The scenario we're looking at is dist-sync within the site and async xsite repl between sites. I should also have mentioned that latency within a site is very small (e.g. 0.05ms) whereas we might have up to 60 *ms* between sites. 2# you aren't actually running this on multiple sites are you? Correct. This was all within our Boston lab, every node was running on a different box though. The ultimate goal is to inject latency into the system, e.g. using the DELAY protocol in the global cluster. But as a first step, I wanted to get the base performance for xsite repl. When pointing out the different IP addresses you say something about needing them to be different, but I didn't understand if you needed them different because they are in different places, or to separate otherwise local machines to have them appear as in different places. The reason to separate mcast_addr/mcast_port was to simulate 3 sites on the same box. Had I not used different addresses/ports for the 3 sites, all nodes of the 3 sites would have found each other and formed a cluster of 9. 3# Since get operations are always local (site), they are as you say not meaningful for the benchmark; now since put operations are also not meaningful as it's async .. what is the benchmark measuring? Well, in the meantime I modified the test and now we do support reads; you can define a read/write ratio. This scenario mimics how a large prospective xsite customer will use xsite repl: dist-sync for intra-site and xsite async repl for inter-site communication. One thing we ran into was that there was a 20% perf degradation if we added async xsite repl *per site* even if that site was down ! The root cause was that async xsite repl does not mark a site as offline in Infinispan even if it is down in JGroups. This will get fixed in Infinispan and should increase async xsite repl for down sites, see [1] for details. Note that the test can also measure sync xsite replication between sites; this is just a matter of configuring the cache differently. But as the scenario this will be used initially is async xseit repl, that's what we're focusing on for now. 4# There seems to be some degree of redundancy when explaining LON/SFO/NYC setting as the local site vs the backup sites. Wouldn't it make more sense to be able to configure all backup sites the same and have it automatically ignore the self element as a backup site? So your script would only need to specify what the local site is. If that makes any sense it would even be nice to extend this to the IP addresses being defined in the zones area, so that they are applied both to the JGroups configuration for the local cluster and to the bridge configuration. Regarding the mcast_addr/mcast_port settings: yes, I could have used only 1 config file (local.xml) and set these properties as variables. I've already changed and committed this. Regarding the setup of the self and backup sites: yes, this could have been done. Again, this is just a matter of setup and lazyness on my behalf :-) 5# I was initially surprised to see x-site configuration as part of a cache configuration; I understand the reasons for options like strategy which one might want to specify differently on each cache, but what about take offline ? Take offline/online is currently available via a JMX operations. Taking a site offline is also done automatically, but currently only when xsite repl is *sync*. There's a JIRA that'll fix this for async xsite repl. that sounds more something which should be globally managed at the channel level - not sure if in JGroups directly but if it's to be handled in Infinispan I would expect to have all caches use the same policy, consistent with FD. Actually this doesn't use JGroups failure detection as we can't use it across sites. This is where the takeOffline ... element comes in (as I mentioned, currently only for async) Also it doesn't looks like you have much of a choice in to which sites you want to replicate, as relay is setup at the jgroups level so affecting all caches: is relay going to be ignored by caches having no x-site enabled?
Re: [infinispan-dev] Infinispan xsite perf test demo
Thanks Bela, I enjoyed the demo! I suggest adding some sort of visualization (maybe just a few diagrams within the demo) of what is actually happening with the data would help understanding for users who are just starting with xsite. Rado On 14/12/12 12:09, Bela Ban wrote: FYI, I've uploaded a 10-minute video [1] to YouTube showing how I setup and run a perf test for Infinispan xsite replication [2]. The test (including all configuration) is available at [3]. This was run with a snapshot of Infinispan (roughly 5.2.0.Beta6) and JGroups 3.3.0.Alpha1. Enjoy ! [1] http://www.youtube.com/watch?v=owOs430vLZo [2] https://docs.jboss.org/author/display/ISPN/Cross+site+replication [3] https://github.com/belaban/IspnPerfTest ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Infinispan xsite perf test demo
That was very very nice to see. Assuming you also asked for feedback to improve this as a talk: 1# you stress several times that reads are going to be local, so very fast. I think you meant local to the site ? as some ~33% of entries will need to be fetched from peers on the same site. 2# you aren't actually running this on multiple sites are you? When pointing out the different IP addresses you say something about needing them to be different, but I didn't understand if you needed them different because they are in different places, or to separate otherwise local machines to have them appear as in different places. 3# Since get operations are always local (site), they are as you say not meaningful for the benchmark; now since put operations are also not meaningful as it's async .. what is the benchmark measuring? 4# There seems to be some degree of redundancy when explaining LON/SFO/NYC setting as the local site vs the backup sites. Wouldn't it make more sense to be able to configure all backup sites the same and have it automatically ignore the self element as a backup site? So your script would only need to specify what the local site is. If that makes any sense it would even be nice to extend this to the IP addresses being defined in the zones area, so that they are applied both to the JGroups configuration for the local cluster and to the bridge configuration. 5# I was initially surprised to see x-site configuration as part of a cache configuration; I understand the reasons for options like strategy which one might want to specify differently on each cache, but what about take offline ? that sounds more something which should be globally managed at the channel level - not sure if in JGroups directly but if it's to be handled in Infinispan I would expect to have all caches use the same policy, consistent with FD. Also it doesn't looks like you have much of a choice in to which sites you want to replicate, as relay is setup at the jgroups level so affecting all caches: is relay going to be ignored by caches having no x-site enabled? And is it going to be relayed only to one site if the Infinispan configuration lists a single site? Not sure if this makes any sense, I just found it contrasting with my naive expectations of how such a configuration would look like. thanks a lot, I hope this is proof enough that your video was pretty catchy :) Cheers, Sanne On 14 December 2012 11:47, Radoslav Husar rhu...@redhat.com wrote: Thanks Bela, I enjoyed the demo! I suggest adding some sort of visualization (maybe just a few diagrams within the demo) of what is actually happening with the data would help understanding for users who are just starting with xsite. Rado On 14/12/12 12:09, Bela Ban wrote: FYI, I've uploaded a 10-minute video [1] to YouTube showing how I setup and run a perf test for Infinispan xsite replication [2]. The test (including all configuration) is available at [3]. This was run with a snapshot of Infinispan (roughly 5.2.0.Beta6) and JGroups 3.3.0.Alpha1. Enjoy ! [1] http://www.youtube.com/watch?v=owOs430vLZo [2] https://docs.jboss.org/author/display/ISPN/Cross+site+replication [3] https://github.com/belaban/IspnPerfTest ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Infinispan xsite perf test demo
Great stuff Bela! Btw, is xsite replication bidirectional? LON has SFO as backup, and SFO also has LON as backup. On Fri, Dec 14, 2012 at 8:01 PM, Sanne Grinovero sa...@infinispan.orgwrote: That was very very nice to see. Assuming you also asked for feedback to improve this as a talk: 1# you stress several times that reads are going to be local, so very fast. I think you meant local to the site ? as some ~33% of entries will need to be fetched from peers on the same site. 2# you aren't actually running this on multiple sites are you? When pointing out the different IP addresses you say something about needing them to be different, but I didn't understand if you needed them different because they are in different places, or to separate otherwise local machines to have them appear as in different places. 3# Since get operations are always local (site), they are as you say not meaningful for the benchmark; now since put operations are also not meaningful as it's async .. what is the benchmark measuring? 4# There seems to be some degree of redundancy when explaining LON/SFO/NYC setting as the local site vs the backup sites. Wouldn't it make more sense to be able to configure all backup sites the same and have it automatically ignore the self element as a backup site? So your script would only need to specify what the local site is. If that makes any sense it would even be nice to extend this to the IP addresses being defined in the zones area, so that they are applied both to the JGroups configuration for the local cluster and to the bridge configuration. 5# I was initially surprised to see x-site configuration as part of a cache configuration; I understand the reasons for options like strategy which one might want to specify differently on each cache, but what about take offline ? that sounds more something which should be globally managed at the channel level - not sure if in JGroups directly but if it's to be handled in Infinispan I would expect to have all caches use the same policy, consistent with FD. Also it doesn't looks like you have much of a choice in to which sites you want to replicate, as relay is setup at the jgroups level so affecting all caches: is relay going to be ignored by caches having no x-site enabled? And is it going to be relayed only to one site if the Infinispan configuration lists a single site? Not sure if this makes any sense, I just found it contrasting with my naive expectations of how such a configuration would look like. thanks a lot, I hope this is proof enough that your video was pretty catchy :) Cheers, Sanne On 14 December 2012 11:47, Radoslav Husar rhu...@redhat.com wrote: Thanks Bela, I enjoyed the demo! I suggest adding some sort of visualization (maybe just a few diagrams within the demo) of what is actually happening with the data would help understanding for users who are just starting with xsite. Rado On 14/12/12 12:09, Bela Ban wrote: FYI, I've uploaded a 10-minute video [1] to YouTube showing how I setup and run a perf test for Infinispan xsite replication [2]. The test (including all configuration) is available at [3]. This was run with a snapshot of Infinispan (roughly 5.2.0.Beta6) and JGroups 3.3.0.Alpha1. Enjoy ! [1] http://www.youtube.com/watch?v=owOs430vLZo [2] https://docs.jboss.org/author/display/ISPN/Cross+site+replication [3] https://github.com/belaban/IspnPerfTest ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
