Re: [CODE4LIB] Autoscaling and streaming apps on EC2
> > The variability in traffic and the support for streaming in CloudFront > does make this look like a good use of AWS. Just out of curiosity what > is the user and staff client software that you are talking about? http://variations.sourceforge.net/ The demo movies at http://www.dlib.indiana.edu/projects/variations3/demos.html give a good introduction to what it does > If you store your content in S3, and clients stream it directly from > there (through CloudFront) you may not even need to autoscale your > app. But I guess I'm still not clear on what the app is. Is it some > kind of catalog of audio files. Is it distinct from the client/staff > software you mentioned? > The core use case we're interested in (which appears to be the same one this software was designed for) is to support streaming music reserves. In other words, staff catalog and upload music and there's an easy interface that allows people to listen to music for specific classes that's access controlled as appropriate. It also lets you do stuff like diagram the structure of the music or read scanned scores. There is a web interface, but the functionality is not the same as the Windows and MacOS client/staff setup at this time kyle -- -- Kyle Banerjee Digital Services Program Manager Orbis Cascade Alliance baner...@uoregon.edu / 503.999.9787
Re: [CODE4LIB] Autoscaling and streaming apps on EC2
On Thu, Mar 1, 2012 at 11:56 PM, Kyle Banerjee wrote: > There's no philosophical problem with CloudFront, but there might be > practical ones. While I should theoretically be able to use s3fs to allow > the software to seamlessly interact with S3, the software also assumes > you're using it to stream media files rather than an external service. > Maybe this change will be easy to implement, maybe it won't -- I won't > know until I try. If it isn't, I need to come up with a Plan B. The variability in traffic and the support for streaming in CloudFront does make this look like a good use of AWS. Just out of curiosity what is the user and staff client software that you are talking about? If you store your content in S3, and clients stream it directly from there (through CloudFront) you may not even need to autoscale your app. But I guess I'm still not clear on what the app is. Is it some kind of catalog of audio files. Is it distinct from the client/staff software you mentioned? //Ed
Re: [CODE4LIB] Autoscaling and streaming apps on EC2
> I might be missing something, but it seems to me that you are > comparing using CloudFront to trying to build your own CloudFront. > Building your own does not seem like it would be very easy or cost > effective. > > Essentially, S3 is an NFS, innit? We use it that way. > > What is the issue with CloudFront? There's no philosophical problem with CloudFront, but there might be practical ones. While I should theoretically be able to use s3fs to allow the software to seamlessly interact with S3, the software also assumes you're using it to stream media files rather than an external service. Maybe this change will be easy to implement, maybe it won't -- I won't know until I try. If it isn't, I need to come up with a Plan B. At this point in time, I'm just trying to make sure I understand my major options for setting up the service. kyle
Re: [CODE4LIB] Autoscaling and streaming apps on EC2
I might be missing something, but it seems to me that you are comparing using CloudFront to trying to build your own CloudFront. Building your own does not seem like it would be very easy or cost effective. Essentially, S3 is an NFS, innit? We use it that way. What is the issue with CloudFront? Cary On Thu, Mar 1, 2012 at 2:36 PM, Kyle Banerjee wrote: > Howdy all, > > I have no experience with autoscaling or streaming, so I'm looking for > thoughts that help me wrap my mind around how to implement it in a > production setting. > > I have been asked to examine the possibility of providing a consortia level > music reserves system using Variations (which I also have no experience > with). The software would be maintained centrally, but each institution > will manage its own collections, users, etc. Load is expected to vary > considerably, varying from practically nothing to possibly hundreds of > simultaneous streams at peak time. This strikes me as an excellent elastic > application. > > This strikes me a good EC2 app and as far as I can tell, there are two > basic ways to achieve the elasticity I'm looking for in that environment. > The first is to store the files in S3 buckets and serve them via > CloudFront. This strikes me as the preferred solution, but I don't yet know > if I'll be able to get the user and staff clients to play well with this > configuration. The second is to have a script monitor the service and spin > up more instances when certain triggers are met and destroy them when > demand drops. But if I do that, all instances need to be able to access the > same live data. For DB data, that's a no brainer since I can just run a DB > server. But how do you synchronize live files across instances since EBS > volumes can only be accessed by one instance? Somehow, NFS strikes me an > ugly way to deal with the problem. Actually even if EBS volumes could be > attached to multiple instances, that solution would still suck as you could > have multiple apps trying to access files at the same time. > > Obviously, I'm having trouble getting pointed in the right direction. I > could punt and just order capacity to handle heavy use cases. But that's a > copout, and figuring out autoscaling bandwidth and computing capacity just > feels like one of those tools that's really handy to have in your bag of > tricks. Any pointers would be appreciated. > > kyle > > -- > -- > Kyle Banerjee > Digital Services Program Manager > Orbis Cascade Alliance > baner...@uoregon.edu / 503.999.9787 -- Cary Gordon The Cherry Hill Company http://chillco.com
[CODE4LIB] Autoscaling and streaming apps on EC2
Howdy all, I have no experience with autoscaling or streaming, so I'm looking for thoughts that help me wrap my mind around how to implement it in a production setting. I have been asked to examine the possibility of providing a consortia level music reserves system using Variations (which I also have no experience with). The software would be maintained centrally, but each institution will manage its own collections, users, etc. Load is expected to vary considerably, varying from practically nothing to possibly hundreds of simultaneous streams at peak time. This strikes me as an excellent elastic application. This strikes me a good EC2 app and as far as I can tell, there are two basic ways to achieve the elasticity I'm looking for in that environment. The first is to store the files in S3 buckets and serve them via CloudFront. This strikes me as the preferred solution, but I don't yet know if I'll be able to get the user and staff clients to play well with this configuration. The second is to have a script monitor the service and spin up more instances when certain triggers are met and destroy them when demand drops. But if I do that, all instances need to be able to access the same live data. For DB data, that's a no brainer since I can just run a DB server. But how do you synchronize live files across instances since EBS volumes can only be accessed by one instance? Somehow, NFS strikes me an ugly way to deal with the problem. Actually even if EBS volumes could be attached to multiple instances, that solution would still suck as you could have multiple apps trying to access files at the same time. Obviously, I'm having trouble getting pointed in the right direction. I could punt and just order capacity to handle heavy use cases. But that's a copout, and figuring out autoscaling bandwidth and computing capacity just feels like one of those tools that's really handy to have in your bag of tricks. Any pointers would be appreciated. kyle -- -- Kyle Banerjee Digital Services Program Manager Orbis Cascade Alliance baner...@uoregon.edu / 503.999.9787
