Re: Netflix's New Peering Appliance Uses FreeBSD
On 06.06.12 03:16, Scott Long wrote: [...] Each disk has its own UFS+J filesystem, except for the SSDs that are mirrored together with gmirror. The SSDs hold the OS image and cache some of the busiest content. The other disks hold nothing but the audio and video files for our content streams. Could you please explain the rationale of using UFS+J for this large storage. Your published documentation states that you have reasonable redundancy in case of multiple disk failure and I wonder how you handle this with plain UFS. Things like avoiding hangs and panics when an disk is going to die. Daniel ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Netflix's New Peering Appliance Uses FreeBSD
On Jun 7, 2012, at 3:09 AM, Daniel Kalchev wrote: On 06.06.12 03:16, Scott Long wrote: [...] Each disk has its own UFS+J filesystem, except for the SSDs that are mirrored together with gmirror. The SSDs hold the OS image and cache some of the busiest content. The other disks hold nothing but the audio and video files for our content streams. Could you please explain the rationale of using UFS+J for this large storage. Your published documentation states that you have reasonable redundancy in case of multiple disk failure and I wonder how you handle this with plain UFS. Things like avoiding hangs and panics when an disk is going to die. Redundancy happens by allowing the streaming clients to choose multiple other sources for their stream, and buffer enough of the stream to make a switchover appear seamless. That other source might be a peer node on the same network, or might be a node that is upstream or on a different network. The point of the caches is to hold as much content as possible, and we've found that it's more effective to maximize capacity but allow drives to fail in place than to significantly reduce capacity with hardware or software RAID. When a disk starts having problems that affect its ability to deliver data on time, any clients affected by it simply switch to a different source. When the disk does finally die, it is removed from the available pool and content is reshuffled on the other drives during the next daily content update. Once enough disks fail that the cache is no longer effective, it gets replaced. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Netflix's New Peering Appliance Uses FreeBSD
On Jun 5, 2012, at 6:16 PM, Scott Long wrote: Yes, we are indeed using FreeBSD at Netflix! For those who are interested, I recently moved from Yahoo to Netflix to help support FreeBSD for them, and I'm definitely impressed with what is going on there. Other than a few small changes, we're using stock FreeBSD 9, tracking the 9-stable branch on a regular basis. Our chassis is a semi-custom 4U 19 form factor with thirty six 3TB SATA disks and 2 SSDs. Each disk has its own UFS+J filesystem, except for the SSDs that are mirrored together with gmirror. The SSDs hold the OS image and cache some of the busiest content. The other disks hold nothing but the audio and video files for our content streams. We connect to the outside world via a twin-port Intel 10GBe optical NIC (only one port is active at the moment), and we use LSI MPT2 controllers for 32 of the 36 disks. The other 4 disks connect to the onboard AHCI SATA controller. All of the disks are direct-attach with no SAS backplanes or expanders. Out-of-band management happens via IPMI on an on-board 1Gb NIC. The entire system consumes around 500W of power, making it a very efficient appliance for its functionality. Netflix is also at the front of the internet pack with IPv6 roll-out, and FreeBSD plays an essential part of that. We've been working hard on stabilizing the FreeBSD IPv6 stack for production-level traffic, and I recommend that all users of IPv6 update to the latest patches in 9-stable and 8-stable. Contact me directly if you have questions about this. That said, we're excited about World IPv6 Day, and we're ready with DNS records and content service from both Amazon and the traditional CDNs as well as our OpenConnect network. From an advocacy standpoint, Netflix represents 30% of all North American internet traffic during peak hours, and FreeBSD is becoming an integral part of that metric as we shift traffic off of the traditional CDNs. We're expanding quickly, which means that FreeBSD is once again a core part of the internet infrastructure. As we find and fix stability and performance issues, we're aggressively pushing those changes into FreeBSD so that everyone can benefit from them, just as we benefit from the contributions of the rest of the FreeBSD ecosystem. We're proud to be a part of the community, and look forward to a long-term relationship with FreeBSD. If you have any questions, let me know or follow the information links on the OpenConnect web site. I wanted to follow up on this briefly. I jumped the gun a little bit in talking about this publicly, since the Openconnect website wasn't fully globally online at the time. It is now, so anyone who previously had trouble getting to it should try again at https://signup.netflix.com/openconnect. Also, I mistakenly claimed that our regular CDN partners were serving streaming content over IPv6. This isn't the case, only OpenConnect is, and I apologize for any confusion (hey, I've only just started at Netflix, and I couldn't even spell IPv6 two weeks ago =-) Finally, I wanted to thank the NginX developers, they've done an amazing job supporting us. The community enthusiasm and interest has been outstanding so far, so please feel free to continue to ask questions on the mailing list and to make formal inquires to Netflix. Thanks, Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Netflix's New Peering Appliance Uses FreeBSD
Hi Scott, [...] I wanted to follow up on this briefly. I jumped the gun a little bit in talking about this publicly, since the Openconnect website wasn't fully globally online at the time. It is now, so anyone who previously had trouble getting to it should try again at https://signup.netflix.com/openconnect. Also, I mistakenly claimed that our regular CDN partners were serving streaming content over IPv6. This isn't the case, only OpenConnect is, and I apologize for any confusion (hey, I've only just started at Netflix, and I couldn't even spell IPv6 two weeks ago =-) Finally, I wanted to thank the NginX developers, they've done an amazing job supporting us. On behalf of nginx team I'd like to thank you and all Netflix team for the chance to work on such an interesting project and your help and cooperation. Thanks much! -- Maxim Konovalov ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
RE: Netflix's New Peering Appliance Uses FreeBSD
From: owner-freebsd-sta...@freebsd.org [mailto:owner-freebsd-sta...@freebsd.org] On Behalf Of Ian Smith On Tue, 5 Jun 2012, Kurt Jaeger wrote: I didn't see a link to this information in the e-mail below. I found this info detailed here: https://signup.netflix.com/openconnect/software If you come from an IP range outside of netflix' footprint, that page is not available. Indeed, I found it a tad strange that URL redirecting to https://signup.netflix.com/global which sayeth: Sorry, Netflix is not available in your country... yet Enter your name and email address below and we'll email you when Netflix is available. But have a look at that PDF, comes from their webpage: http://opsec.eu/backup/OpenConnectDeploymentGuide-v2.4a.pdf Interesting box alright. Hope it wasn't Top Secret in my country, and that I can ask my (Debian based) ISP when they'll be getting some? :) Good to see Scott's found something to keep him off the streets too .. Now if only Netflix figured out a way to allow FreeBSD (and Linux) users to watch their streams... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Netflix's New Peering Appliance Uses FreeBSD
I didn't see a link to this information in the e-mail below. I found this info detailed here: https://signup.netflix.com/openconnect/software From: Benjamin Francom bfran...@gmail.com To: freebsd-stable@freebsd.org Sent: Tue, June 5, 2012 11:00:01 AM Subject: Netflix's New Peering Appliance Uses FreeBSD I just saw this, and thought I'd share: Open Connect Appliance Software Netflix delivers streaming content using a combination of intelligent clients, a central control system, and a network of Open Connect appliances. When designing the Open Connect Appliance Software, we focused on these fundamental design goals: - Use of Open Source software - Ability to efficiently read from disk and write to network sockets - High-performance HTTP delivery - Ability to gather routing information via BGP Operating System For the operating system, we use FreeBSD http://www.freebsd.org/ version 9.0. This was selected for its balance of stability and features, a strong development community and staff expertise. We will contribute changes we make as part of our project to the community through the FreeBSD committers on our team. Web server We use the nginx http://www.nginx.org/ web server for its proven scalability and performance. Netflix audio and video is served via HTTP. Routing intelligence proxy We use the BIRD Internet routing daemon http://bird.network.cz/ to enable the transfer of network topology from ISP networks to the Netflix control system that directs clients to sources of content. Acknowledgements We would would like to express our thanks to the FreeBSD community, the nginx community, and Ondrej and the BIRD team for providing excellent open source software. We also work directly with Igor, Maxim, Andrew, Sergey, Ruslan and the rest of the team at nginx.com http://www.nginx.com/, who provide superb development support for our project. -- Benjamin Francom Information Technology Professional http://www.benfrancom.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Netflix's New Peering Appliance Uses FreeBSD
Hi! I didn't see a link to this information in the e-mail below. I found this info detailed here: https://signup.netflix.com/openconnect/software If you come from an IP range outside of netflix' footprint, that page is not available. But have a look at that PDF, comes from their webpage: http://opsec.eu/backup/OpenConnectDeploymentGuide-v2.4a.pdf -- p...@opsec.eu+49 171 3101372 8 years to go ! ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Netflix's New Peering Appliance Uses FreeBSD
On Jun 5, 2012, at 9:56 AM, Benjamin Francom wrote: I just saw this, and thought I'd share: Open Connect Appliance Software Netflix delivers streaming content using a combination of intelligent clients, a central control system, and a network of Open Connect appliances. When designing the Open Connect Appliance Software, we focused on these fundamental design goals: - Use of Open Source software - Ability to efficiently read from disk and write to network sockets - High-performance HTTP delivery - Ability to gather routing information via BGP Operating System For the operating system, we use FreeBSD http://www.freebsd.org/ version 9.0. This was selected for its balance of stability and features, a strong development community and staff expertise. We will contribute changes we make as part of our project to the community through the FreeBSD committers on our team. Web server Yes, we are indeed using FreeBSD at Netflix! For those who are interested, I recently moved from Yahoo to Netflix to help support FreeBSD for them, and I'm definitely impressed with what is going on there. Other than a few small changes, we're using stock FreeBSD 9, tracking the 9-stable branch on a regular basis. Our chassis is a semi-custom 4U 19 form factor with thirty six 3TB SATA disks and 2 SSDs. Each disk has its own UFS+J filesystem, except for the SSDs that are mirrored together with gmirror. The SSDs hold the OS image and cache some of the busiest content. The other disks hold nothing but the audio and video files for our content streams. We connect to the outside world via a twin-port Intel 10GBe optical NIC (only one port is active at the moment), and we use LSI MPT2 controllers for 32 of the 36 disks. The other 4 disks connect to the onboard AHCI SATA controller. All of the disks are direct-attach with no SAS backplanes or expanders. Out-of-band management happens via IPMI on an on-board 1Gb NIC. The entire system consumes around 500W of power, making it a very efficient appliance for its functionality. Netflix is also at the front of the internet pack with IPv6 roll-out, and FreeBSD plays an essential part of that. We've been working hard on stabilizing the FreeBSD IPv6 stack for production-level traffic, and I recommend that all users of IPv6 update to the latest patches in 9-stable and 8-stable. Contact me directly if you have questions about this. That said, we're excited about World IPv6 Day, and we're ready with DNS records and content service from both Amazon and the traditional CDNs as well as our OpenConnect network. From an advocacy standpoint, Netflix represents 30% of all North American internet traffic during peak hours, and FreeBSD is becoming an integral part of that metric as we shift traffic off of the traditional CDNs. We're expanding quickly, which means that FreeBSD is once again a core part of the internet infrastructure. As we find and fix stability and performance issues, we're aggressively pushing those changes into FreeBSD so that everyone can benefit from them, just as we benefit from the contributions of the rest of the FreeBSD ecosystem. We're proud to be a part of the community, and look forward to a long-term relationship with FreeBSD. If you have any questions, let me know or follow the information links on the OpenConnect web site. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Netflix's New Peering Appliance Uses FreeBSD
On Jun 5, 2012, at 20:16, Scott Long wrote: If you have any questions, let me know or follow the information links on the OpenConnect web site. Out of curiosity, given that Linux seems popular in so many other places (Google, Facebook), is there any particular reason why FreeBSD was chosen for this? I'm sure Linux is used in many other places (much of Netflix's IT infrastructure is on Amazon IIRC), so I'm kind of surprised that they went with FreeBSD when they probably already have so much knowledge with Linux. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Netflix's New Peering Appliance Uses FreeBSD
Maybe their knowledge of Linux drove them to use FreeBSD. Sorry, couldn't resist ;) On 06/05/2012 19:42, David Magda wrote: On Jun 5, 2012, at 20:16, Scott Long wrote: If you have any questions, let me know or follow the information links on the OpenConnect web site. Out of curiosity, given that Linux seems popular in so many other places (Google, Facebook), is there any particular reason why FreeBSD was chosen for this? I'm sure Linux is used in many other places (much of Netflix's IT infrastructure is on Amazon IIRC), so I'm kind of surprised that they went with FreeBSD when they probably already have so much knowledge with Linux. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Netflix's New Peering Appliance Uses FreeBSD
On Jun 5, 2012, at 6:42 PM, David Magda wrote: On Jun 5, 2012, at 20:16, Scott Long wrote: If you have any questions, let me know or follow the information links on the OpenConnect web site. Out of curiosity, given that Linux seems popular in so many other places (Google, Facebook), is there any particular reason why FreeBSD was chosen for this? I'm sure Linux is used in many other places (much of Netflix's IT infrastructure is on Amazon IIRC), so I'm kind of surprised that they went with FreeBSD when they probably already have so much knowledge with Linux. Linux works wonderfully on EC2 for our CC and computational tasks, FreeBSD is proving to work well on deployed hardware for serving bits. It highly maintainable, and there's an excellent community supporting it. From the website: For the operating system, we use FreeBSD http://www.freebsd.org/ version 9.0. This was selected for its balance of stability and features, a strong development community and staff expertise. We will contribute changes we make as part of our project to the community through the FreeBSD committers on our team. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Netflix's New Peering Appliance Uses FreeBSD
On Tue, 5 Jun 2012, Kurt Jaeger wrote: I didn't see a link to this information in the e-mail below. I found this info detailed here: https://signup.netflix.com/openconnect/software If you come from an IP range outside of netflix' footprint, that page is not available. Indeed, I found it a tad strange that URL redirecting to https://signup.netflix.com/global which sayeth: Sorry, Netflix is not available in your country... yet Enter your name and email address below and we'll email you when Netflix is available. But have a look at that PDF, comes from their webpage: http://opsec.eu/backup/OpenConnectDeploymentGuide-v2.4a.pdf Interesting box alright. Hope it wasn't Top Secret in my country, and that I can ask my (Debian based) ISP when they'll be getting some? :) Good to see Scott's found something to keep him off the streets too .. cheers, Ian ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
