Re: [Openstack] [SWIFT] Rack-awareness

2012-11-02 Thread Oleg Gelbukh
John,

Could you please share a link to the materials of the discussion you've
mentioned, or may be place where the discussion takes place (if it's
online)? We're currently working on RD project that involves prototype
implementation of regions, and I'm eager to see suggestions for replication
optimizations and possibly get some feedback about our approach.

--
Best regards,
Oleg Gelbukh
Mirantis, Inc.

On Thu, Nov 1, 2012 at 7:55 PM, John Dickinson m...@not.mn wrote:

 This is already supported in Swift with the concept of availability zones.
 Swift will place each replica in different availability zones, if possible.
 If you only have one zone, Swift will place the replicas on different
 machines. If you only have one machine, Swift will place the replicas on
 different drives.

 There are active discussions right now about how Swift can support a tier
 above these availability zones: regions. A region would be defined by a
 hogher latency link and can provide additional data durability, and,
 depending on your deployment details, better availability.
 http://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/has
  more info on the ideas we're talking about.

 --John




 On Nov 1, 2012, at 8:45 AM, Leandro Reox leandro.r...@gmail.com wrote:

  Hi guys,
 
  Any plans to implement something like hadoop rack-awareness where we can
 define rack spaces to guarantee that a copy of an object is stored for
 example on another datacenter, on another coast. Or should this be managed
 by container sync to the other datacenter
 
  I think that this can be a nice-to-have feature, i dont know if its on
 the dev roadmap
 
  Best
  Lean
  ___
  Mailing list: https://launchpad.net/~openstack
  Post to : openstack@lists.launchpad.net
  Unsubscribe : https://launchpad.net/~openstack
  More help   : https://help.launchpad.net/ListHelp


 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] [SWIFT] Rack-awareness

2012-11-01 Thread John Dickinson
This is already supported in Swift with the concept of availability zones. 
Swift will place each replica in different availability zones, if possible. If 
you only have one zone, Swift will place the replicas on different machines. If 
you only have one machine, Swift will place the replicas on different drives.

There are active discussions right now about how Swift can support a tier above 
these availability zones: regions. A region would be defined by a hogher 
latency link and can provide additional data durability, and, depending on your 
deployment details, better availability. 
http://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/
 has more info on the ideas we're talking about.

--John




On Nov 1, 2012, at 8:45 AM, Leandro Reox leandro.r...@gmail.com wrote:

 Hi guys, 
 
 Any plans to implement something like hadoop rack-awareness where we can 
 define rack spaces to guarantee that a copy of an object is stored for 
 example on another datacenter, on another coast. Or should this be managed by 
 container sync to the other datacenter
 
 I think that this can be a nice-to-have feature, i dont know if its on the 
 dev roadmap
 
 Best
 Lean
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp



smime.p7s
Description: S/MIME cryptographic signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] [SWIFT] Rack-awareness

2012-11-01 Thread Alejandro Comisario
John, what i think would be terrific ( i hope is not implemented, if not im
gonna feel a dunce ) if, for latency matters, suppose you have 4 zones, 2
on each datacenter, and on each datacenter, you have 2 proxies for example.

De idea would be that there were some kind of mechanism to tell the ring,
what nodes are under what proxies, to give some kind of prefferece
regarding latency, and for example not having a proxy on DC1 to cross the
country to get an object from a datanode in DC2.

Have you thought anything like it?
If not, im wondering how, for example rackspace handles this kind of issues
( ignoring all the CDN thing )

Cheers.

*---*
*Alejandrito*

On Thu, Nov 1, 2012 at 12:55 PM, John Dickinson m...@not.mn wrote:

 This is already supported in Swift with the concept of availability zones.
 Swift will place each replica in different availability zones, if possible.
 If you only have one zone, Swift will place the replicas on different
 machines. If you only have one machine, Swift will place the replicas on
 different drives.

 There are active discussions right now about how Swift can support a tier
 above these availability zones: regions. A region would be defined by a
 hogher latency link and can provide additional data durability, and,
 depending on your deployment details, better availability.
 http://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/has
  more info on the ideas we're talking about.

 --John




 On Nov 1, 2012, at 8:45 AM, Leandro Reox leandro.r...@gmail.com wrote:

  Hi guys,
 
  Any plans to implement something like hadoop rack-awareness where we can
 define rack spaces to guarantee that a copy of an object is stored for
 example on another datacenter, on another coast. Or should this be managed
 by container sync to the other datacenter
 
  I think that this can be a nice-to-have feature, i dont know if its on
 the dev roadmap
 
  Best
  Lean
  ___
  Mailing list: https://launchpad.net/~openstack
  Post to : openstack@lists.launchpad.net
  Unsubscribe : https://launchpad.net/~openstack
  More help   : https://help.launchpad.net/ListHelp


 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] [SWIFT] Rack-awareness

2012-11-01 Thread John Dickinson
That's absolutely what is planned. The idea for global replicas involves three 
basic pieces:

1) A separate replication network
2) Exposing a new tier of uniqueness (the region) in the ring management tools
3) Implementing proxy affinity so that a proxy server chooses closer replicas 
rather than more distant ones

The first two are simple and already have much of the infrastructure in place. 
The third is a little more tricky.

We will probably also need the ability to increase or decrease the cluster's 
replica count to make this useful to more deployers.

--John




On Nov 1, 2012, at 9:49 AM, Alejandro Comisario 
alejandro.comisa...@mercadolibre.com wrote:

 John, what i think would be terrific ( i hope is not implemented, if not im 
 gonna feel a dunce ) if, for latency matters, suppose you have 4 zones, 2 on 
 each datacenter, and on each datacenter, you have 2 proxies for example.
 
 De idea would be that there were some kind of mechanism to tell the ring, 
 what nodes are under what proxies, to give some kind of prefferece regarding 
 latency, and for example not having a proxy on DC1 to cross the country to 
 get an object from a datanode in DC2.
 
 Have you thought anything like it?
 If not, im wondering how, for example rackspace handles this kind of issues ( 
 ignoring all the CDN thing )
 
 Cheers.
 
 ---
 Alejandrito
 
 On Thu, Nov 1, 2012 at 12:55 PM, John Dickinson m...@not.mn wrote:
 This is already supported in Swift with the concept of availability zones. 
 Swift will place each replica in different availability zones, if possible. 
 If you only have one zone, Swift will place the replicas on different 
 machines. If you only have one machine, Swift will place the replicas on 
 different drives.
 
 There are active discussions right now about how Swift can support a tier 
 above these availability zones: regions. A region would be defined by a 
 hogher latency link and can provide additional data durability, and, 
 depending on your deployment details, better availability. 
 http://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/
  has more info on the ideas we're talking about.
 
 --John
 
 
 
 
 On Nov 1, 2012, at 8:45 AM, Leandro Reox leandro.r...@gmail.com wrote:
 
  Hi guys,
 
  Any plans to implement something like hadoop rack-awareness where we can 
  define rack spaces to guarantee that a copy of an object is stored for 
  example on another datacenter, on another coast. Or should this be managed 
  by container sync to the other datacenter
 
  I think that this can be a nice-to-have feature, i dont know if its on the 
  dev roadmap
 
  Best
  Lean
  ___
  Mailing list: https://launchpad.net/~openstack
  Post to : openstack@lists.launchpad.net
  Unsubscribe : https://launchpad.net/~openstack
  More help   : https://help.launchpad.net/ListHelp
 
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp
 
 



smime.p7s
Description: S/MIME cryptographic signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] [SWIFT] Rack-awareness

2012-11-01 Thread Leandro Reox
Ale,

Guess that what
http://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/
link
latency feature is meant to be ... but John can confirm that :)

Best
Lean


On Thu, Nov 1, 2012 at 1:49 PM, Alejandro Comisario 
alejandro.comisa...@mercadolibre.com wrote:

 John, what i think would be terrific ( i hope is not implemented, if not
 im gonna feel a dunce ) if, for latency matters, suppose you have 4 zones,
 2 on each datacenter, and on each datacenter, you have 2 proxies for
 example.

 De idea would be that there were some kind of mechanism to tell the ring,
 what nodes are under what proxies, to give some kind of prefferece
 regarding latency, and for example not having a proxy on DC1 to cross the
 country to get an object from a datanode in DC2.

 Have you thought anything like it?
 If not, im wondering how, for example rackspace handles this kind of
 issues ( ignoring all the CDN thing )

 Cheers.

 *---*
 *Alejandrito*

 On Thu, Nov 1, 2012 at 12:55 PM, John Dickinson m...@not.mn wrote:

 This is already supported in Swift with the concept of availability
 zones. Swift will place each replica in different availability zones, if
 possible. If you only have one zone, Swift will place the replicas on
 different machines. If you only have one machine, Swift will place the
 replicas on different drives.

 There are active discussions right now about how Swift can support a tier
 above these availability zones: regions. A region would be defined by a
 hogher latency link and can provide additional data durability, and,
 depending on your deployment details, better availability.
 http://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/has
  more info on the ideas we're talking about.

 --John




 On Nov 1, 2012, at 8:45 AM, Leandro Reox leandro.r...@gmail.com wrote:

  Hi guys,
 
  Any plans to implement something like hadoop rack-awareness where we
 can define rack spaces to guarantee that a copy of an object is stored
 for example on another datacenter, on another coast. Or should this be
 managed by container sync to the other datacenter
 
  I think that this can be a nice-to-have feature, i dont know if its on
 the dev roadmap
 
  Best
  Lean
  ___
  Mailing list: https://launchpad.net/~openstack
  Post to : openstack@lists.launchpad.net
  Unsubscribe : https://launchpad.net/~openstack
  More help   : https://help.launchpad.net/ListHelp


 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] [SWIFT] Rack-awareness

2012-11-01 Thread Alejandro Comisario
Nic, a good reason to wait to Grizzly (?) with our arms wide open!

Thanks for confirming that John / Lean !

*
*
*
*
*Alejandro Comisario
#melicloud CloudBuilders*
Arias 3751, Piso 7 (C1430CRG)
Ciudad de Buenos Aires - Argentina
Cel: +549(11) 15-3770-1857
Tel : +54(11) 4640-8443


On Thu, Nov 1, 2012 at 2:07 PM, Leandro Reox leandro.r...@gmail.com wrote:

 Ale,

 Guess that what
 http://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/
  link
 latency feature is meant to be ... but John can confirm that :)

 Best
 Lean


 On Thu, Nov 1, 2012 at 1:49 PM, Alejandro Comisario 
 alejandro.comisa...@mercadolibre.com wrote:

 John, what i think would be terrific ( i hope is not implemented, if not
 im gonna feel a dunce ) if, for latency matters, suppose you have 4 zones,
 2 on each datacenter, and on each datacenter, you have 2 proxies for
 example.

 De idea would be that there were some kind of mechanism to tell the ring,
 what nodes are under what proxies, to give some kind of prefferece
 regarding latency, and for example not having a proxy on DC1 to cross the
 country to get an object from a datanode in DC2.

 Have you thought anything like it?
 If not, im wondering how, for example rackspace handles this kind of
 issues ( ignoring all the CDN thing )

 Cheers.

 *---*
 *Alejandrito*

 On Thu, Nov 1, 2012 at 12:55 PM, John Dickinson m...@not.mn wrote:

 This is already supported in Swift with the concept of availability
 zones. Swift will place each replica in different availability zones, if
 possible. If you only have one zone, Swift will place the replicas on
 different machines. If you only have one machine, Swift will place the
 replicas on different drives.

 There are active discussions right now about how Swift can support a
 tier above these availability zones: regions. A region would be defined
 by a hogher latency link and can provide additional data durability, and,
 depending on your deployment details, better availability.
 http://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/has
  more info on the ideas we're talking about.

 --John




 On Nov 1, 2012, at 8:45 AM, Leandro Reox leandro.r...@gmail.com wrote:

  Hi guys,
 
  Any plans to implement something like hadoop rack-awareness where we
 can define rack spaces to guarantee that a copy of an object is stored
 for example on another datacenter, on another coast. Or should this be
 managed by container sync to the other datacenter
 
  I think that this can be a nice-to-have feature, i dont know if its on
 the dev roadmap
 
  Best
  Lean
  ___
  Mailing list: https://launchpad.net/~openstack
  Post to : openstack@lists.launchpad.net
  Unsubscribe : https://launchpad.net/~openstack
  More help   : https://help.launchpad.net/ListHelp


 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp




___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp