I think John pointed you to some info on how to achieve the the hierarchical structure requirement The other requirement was around syncing remote clusters: Swift Diablo (1.4) can probably be suited for the scenario you're describing: See the spec for Multi cluster sync for swift [1] and the excellent comments in the code [2] implementing it. (I haven't yet played with it, but planning to soon)
[1] http://etherpad.openstack.org/QAoBrOHZxd [2] https://github.com/openstack/swift/blob/master/swift/container/sync.py#L72 hth, a. On Mon, Aug 15, 2011 at 8:50 AM, John Dickinson <[email protected]> wrote: > See > http://programmerthoughts.com/programming/nested-folders-in-cloud-files/for > info on how to use a nested directory structure in swift. > > --John > > > On Aug 14, 2011, at 11:07 PM, Thiago Moraes wrote: > > > I took a look at some distributed file systems and went a little deeper > in Hadoop and his HDFS, for instance. I don't really need full POSIX > compliance, but having a nested structure is important, but as far as I know > there are way to simulate this on Switf, is that correct? > > > > The problem I see in using something like hadoop is the single point of > failure, not because I need almost 100% availability, but because the people > who will access the data does not belong to the same organization. They will > be researchers from different institutions that may want to deploy a local > server with a subset of the data to improve their productivity, but the data > set's size makes impractical to just copy everything. > > > > The plan would be that the interface to the system would show which files > are stored locally and which are not, so that everyone gets access to > everything, almost like a peer to peer system where they download from the > closest source and then store for their own use. > > > > At first, I though of implementing something by hand, but using an > already mature solution makes a lot more sense. > > > > So, is this plausible or am I trying to use the wrong tools? > > > > thanks, again > > > > Thiago Moraes - EnC 07 - UFSCar > > > > > > 2011/8/14 Todd Deshane <[email protected]> > > On Sun, Aug 14, 2011 at 4:10 AM, Thiago Moraes > > <[email protected]> wrote: > > > Hey guys, > > > > > > I'm new on the list and I'm currently considering Openstack to solve a > data > > > distribution problem. Right now, there's a server which contains very > large > > > files (usual files have 30GB or even more). This server is accessed by > LAN > > > and over the internet but, of course, it's difficult to do this without > > > local connection. > > > > > > My idea to solve this problem is to deploy new servers on the places > which > > > access data more often in an such a way that they get a local copy of > the > > > most accessed part of data by then. In my head, I consider that there > will > > > be N different clouds, one at my location and the others spread on > another > > > networks. Then, these new clouds would download and store parts of the > data > > > (entire files) so that they can be accessed through their own LAN. > > > > > > > It sounds like you are looking for the functionality that Zones (aim > > to?) provide. > > > > Take a look at: > > > > http://wiki.openstack.org/MultiClusterZones > > > > > > > Is Openstack suitable in this environment? Anyone would recommend > another > > > solution? > > > > > > > Have you also looked at SheepDog, Hadoop or HC2? All of these seem to > > have some OpenStack integration points as well. > > > > Some links to look into: > > http://wiki.openstack.org/SheepdogSupport > > http://doubleclix.wordpress.com/2011/03/17/hadoop-2-0-openstack-pbj/ > > > http://www.quora.com/What-features-differentiate-HDFS-and-OpenStack-Object-Storage > > > > > > Hope that helps. > > > > Thanks, > > Todd > > > > > PS: I know the file size limitations of 5GB. I just need that all parts > of a > > > file to be in the same local area network so that a blazingly fast > Internet > > > connection is not required all the time. > > > > > > thanks, > > > > > > > > > Thiago Moraes - EnC 07 - UFSCar > > > > > > _______________________________________________ > > > Mailing list: https://launchpad.net/~openstack > > > Post to : [email protected] > > > Unsubscribe : https://launchpad.net/~openstack > > > More help : https://help.launchpad.net/ListHelp > > > > > > > > > > > > > > -- > > Todd Deshane > > http://www.linkedin.com/in/deshantm > > http://www.xen.org/products/cloudxen.html > > http://runningxen.com/ > > > > _______________________________________________ > > Mailing list: https://launchpad.net/~openstack > > Post to : [email protected] > > Unsubscribe : https://launchpad.net/~openstack > > More help : https://help.launchpad.net/ListHelp > > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : [email protected] > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > >
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
