Hi, As long as you don't share more details on how your files are accessed and what makes each URL unique, I can't help. As I said, splitting your files by directory path or by Host header may be good.
Concerning example in haproxy, having the following in your frontend will do the stuff: acl dir1 path_beg /dir1/ usebackend bk_dir1 if dir1 acl dir1 path_beg /dir2/ usebackend bk_dir2 if dir2 ... then create the backends: bk_dir1 balance roundrobin server srv1 server srv2 bk_dir2 balance roundrobin server srv3 server srv4 ... Hope this helps On Mon, Nov 21, 2011 at 3:24 PM, Rerngvit Yanggratoke <rerng...@kth.se> wrote: > Dear Baptiste, > Could you please exemplify a criterion that would reduce the > number of files per backends? And, if possible, how to implement that with > HAProxy? > > On Sat, Nov 19, 2011 at 8:29 PM, Baptiste <bed...@gmail.com> wrote: >> >> On Fri, Nov 18, 2011 at 5:48 PM, Rerngvit Yanggratoke <rerng...@kth.se> >> wrote: >> > Hello All, >> > First of all, pardon me if I'm not communicating very well. >> > English >> > is not my native language. We are running a static file distribution >> > cluster. The cluster consists of many web servers serving static files >> > over >> > HTTP. We have very large number of files such that a single server >> > simply >> > can not keep all files (don't have enough disk space). In particular, a >> > file >> > can be served only from a subset of servers. Each file is uniquely >> > identified by a file's URI. I would refer to this URI later as a key. >> > I am investigating deploying HAProxy as a front end to this >> > cluster. >> > We want HAProxy to provide load balancing and automatic fail over. In >> > other >> > words, a request comes first to HAProxy and HAProxy should forward the >> > request to appropriate backend server. More precisely, for a particular >> > key, >> > there should be at least two servers being forwarded to from HAProxy for >> > the >> > sake of load balancing. My question is what load >> > balancing strategy should I >> > use? >> > I could use hashing(based on key) or consistent hashing. >> > However, >> > each file would end up being served by a single server on a particular >> > moment. That means I wouldn't have load balancing and fail over for a >> > particular key. >> > Is there something like a combination of hashing and >> > roundrobin strategy? In particular, for a particular key, there would be >> > multiple servers serving the requests and HAProxy selects one of them >> > according to roundrobin policy. If there isn't such a stretegy, any >> > suggestions on how to implement this into HAProxy? Any other comments >> > are >> > welcome as well. >> > >> > -- >> > Best Regards, >> > Rerngvit Yanggratoke >> > >> >> Hi, >> >> You could create several backends and redirect requests based on an >> arbitrary criteria to reduce the number of files per backends. Using >> URL path prefix might be a good idea. >> Then inside a backend, you can use the hash url load-balancing algorithm. >> >> cheers > > > > -- > Best Regards, > Rerngvit Yanggratoke >