Somewhat related, I found this interesting: Admitting that functional programming can be awkward: http://prog21.dadgum.com/3.html
On Nov 10, 2007 10:08 AM, mojo.talantikite <[EMAIL PROTECTED]> wrote: > > Thanks for all the answers, I really appreciate them. Sometime later > today or tomorrow I'd like to write up some questions that pertain to > more of what I'm trying to do, but for now that gives me some pretty > good ideas as to how I could use Erlang/ErlyWeb. > > Overall, what do you feel are Erlang/ErlyWeb's particular strengths > when it comes to web development? Of course you can pretty much get > any language/framework to do what you need it to do, but how does this > particular combination yield itself to solving (or simplifying) > particular web development problems? > > Thanks again, the answers were very helpful. > > best, > Mojo > > On Nov 9, 6:25 am, Al <[EMAIL PROTECTED]> wrote: > > On Nov 9, 9:08 am, "Yariv Sadan" <[EMAIL PROTECTED]> wrote: > > > > > > > > > On Nov 8, 2007 12:23 PM, mojo.talantikite <[EMAIL PROTECTED]> wrote: > > > > > > I'm in the very early stages of founding a startup, and currently > > > > we're in talks about what type of technology to be using. As there > > > > aren't too many people on board now, and none besides myself that are > > > > technical, there is a pretty blank slate for development as of now. > > > > After playing around with Erlang for the past couple of days I'm > > > > starting to think about the pros/cons of using Erlang. > > > > > > Anyways, I have questions pertaining to how Erlang/ErlyWeb addresses > > > > certain needs and issues of scaling. I thought it might be more useful > > > > for a greater number of people if instead of speaking solely about our > > > > needs for our web app, we use an example people might be familiar with > > > > and get more from: YouTube. Here are some notes on a Google Tech Talk > > > > that was given, which also contains a link to the tech talk video: > > > >http://kylecordes.com/2007/07/12/youtube-scalability/ > > > > > > Here are some questions that come to mind when thinking about what > > > > they faced: > > > > > > 1. Their main reason for choosing Python as their language is because > > > > development speed and productivity was (and is) very important for > > > > various reasons (one important one was probably showing prototypes and > > > > their progresses to potential investors). Erlang seems to be a very > > > > productive environment for some applications, but how does it stand up > > > > for web app development? Using Python, PHP, or Ruby has the added > > > > benefit of having lots of developers to draw from their experiences -- > > > > how would Erlang compare? > > > > > Erlang definitely has fewer developers using it for webapps that PHP, > > > Python or Ruby. My subjective take is that, having done PHP > > > programming and played with RoR, ErlyWeb can be just as productive if > > > not more and its code is very concise and expressive. I rarely feel > > > that I'm writing repetitive code that doesn't directly solve the > > > problem domain. Plus, I prefer Erlang's functional style to imperative > > > one. I think FP makes your code more readable and debuggable. Erlang > > > has fewer libraries than more popular languages, which may or may not > > > affect you depending on your app's needs, but I think that the fewer > > > libraries isn't as big an issue as some people say. Erlang has a > > > variety of ways for talking to other lanugages and I'd rather write > > > most of my code in Erlang in occasionally outsource some functionality > > > to another language than lock my whole app to a language I don't like > > > as much just because of the existence of a couple of libraries for it. > > > Btw, Erlang also has some tools that other languages don't have -- > > > most notably, elegant concurrency and Mnesia. > > > > > > 2. Serving content was of course a big issue. They moved to lighttpd > > > > from Apache for their videos and got a lot of performance increases > > > > (they use CDNs for their most popular content). Yaws seems great for > > > > dynamic content, but is there anything better for the static content? > > > > > Use lighttpd for static content and Yaws (or MochiWeb) for dynamic > > > content. > > > > > > 3. Another major issue YouTube ran into was serving their > > > > thumbnails. Each video can have about 4 thumbnails and each page, > > > > while having only one video playing, could have 40-60 thumbnails per > > > > page. Thumbnails, unlike video, aren't distributed across hundreds of > > > > machines (due to their size). So, you have lots of requests per > > > > second and tons of disk seeks. Apache didn't really work out, so they > > > > then went to a modified version of lighttpd that put the disk reads > > > > into worker threads -- but they ran into issues with that as well. > > > > They finally went to Google's BTFE. How would Erlang look at this > > > > problem -- would Yaws help at all? > > > > > I'm not too familiar with this problem, but if it's static data, it's > > > generally better to serve if from lighttpd than Yaws. > > > > Agreed LightHttpd or apache will outperform Yaws with static content. > > If its lots of small files/images look at using a cache like server > > with lots and lots of RAM also consider using solid state disks (SSDs) > > in Raid 0 configs, these are much faster than traditional disks for > > random small files, they blow away performance of even 15K scsi arrays > > for this job, as seeks on SSD flash are near zero in comparison. The > > limit here is capacity. You can get 64GB SSDs with Sata interfaces > > (Sata II soon from Samsung) raid these up with Sata Raid Boards on > > fast PCIe interfaces + built in Sata ports in a mirror config You will > > be surprised at how kick ass fast these are for the job. You also need > > good server Gigabit ports that can be aggregated. > > > > > > > > > > > > > > 4. They use MySql to store metadata, but as the site got huge, they > > > > ran into some issues with it. They went from having one database to > > > > doing db replication. That caused issues eventually, one of them > > > > being that spreading read load (being asynchronous) caused the > > > > database to serve outdated data from time to time. They then went to > > > > partitioning the database into shards. Overall, though, they're > > > > running into issues due to the fact that they're trying to do more and > > > > more stuff with the data. For example, recommendation systems and > > > > data mining gets really hard on 100s of millions of views per day, so > > > > they are needing some solutions that can do parallel queries and > > > > handle distributed computation. Would Mnesia be able to help handle > > > > these issues, or anything else from the Erlang world that could deal > > > > with these issues? > > > > > Mnesia is more suited for storing live session data than large amounts > > > of persistent data. And Mnesia isn't really suited for data mining -- > > > just for simple queries. However, Erlang helps in that it allows you > > > to connect to multiple databases using its great connection pooling > > > capabilities (a process is spawned for each database connection, and a > > > dispatcher process is responsible for routing queries to connection > > > processes). If I were building YouTube in Erlang, I would consider > > > doing all replication and partitioning work in the application code > > > rather than relying on the database engine. > > > > Use flat files for the metadata and cache them where possible across > > machines, consider an appending meta file storage controller (write it > > yourself) and file formats that enable appending rather than editing. > > The trick heres is to only append to files where possible, editing is > > expense (Well percolating the changes is).Break the files up like you > > would database tables and crate summarized view files for common > > queries. > > > > If there is a lot of data + analysis consider something like a map > > reduce pattern to create views on the backend, erlang is excellent for > > this sort of stuff, blows the socks of most others. > > > > regards > > Al > > > > Regards > > Al > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "erlyweb" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/erlyweb?hl=en -~----------~----~----~----~------~----~------~--~---
