Re: [Wikitech-l] All non-api traffic is now served by HHVM
On 09/12/14 23:10, Brian Wolff wrote: Awesome. Any chance the video scalars could be put near the top of the list for servers to upgrade Ubuntu on? The really old version of libav on those servers is causing problems for people uploading videos in certain formats. Since API and appservers are done, we're left with the jobrunners (for which the conversion is already done), the imagescalers and the videoscalers. We are working right now on the imagescaler conversion, it will require some preparation work and some testing, but it won't take too long hopefully. Cheers, Giuseppe -- Giuseppe Lavagetto Wikimedia Foundation - TechOps Team ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] All non-api traffic is now served by HHVM
On 03/12/14 18:03, Giuseppe Lavagetto wrote: Hi all, [CUT] The API traffic is still being partially served by mod_php, but that will not be for long! As promised, all our API traffic is on HHVM as well as of now. The effects on CPU usage have been quite drastic on this cluster, where the load is higher: http://bit.ly/1Abwwzi Cheers, Giuseppe -- Giuseppe Lavagetto Wikimedia Foundation - TechOps Team ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] All non-api traffic is now served by HHVM
On 12/3/14, Giuseppe Lavagetto glavage...@wikimedia.org wrote: Hi all, it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature. Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous task that would require me weeks to complete, even with the improved automation we built lately. Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM. This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era). The API traffic is still being partially served by mod_php, but that will not be for long! Cheers, Giuseppe -- Giuseppe Lavagetto Wikimedia Foundation - TechOps Team ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l Awesome. Any chance the video scalars could be put near the top of the list for servers to upgrade Ubuntu on? The really old version of libav on those servers is causing problems for people uploading videos in certain formats. --bawolff ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] All non-api traffic is now served by HHVM
This deserves a 1 meter barnstar to Alex, Giuseppe and Yuvi! The impact is so visible on * the last three graphs at https://gdash.wikimedia.org/dashboards/editpage/ and * the last four at https://gdash.wikimedia.org/dashboards/totalphp/ , that one tends to suspect an error in the graphs. Nemo ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] All non-api traffic is now served by HHVM
Hi all, it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature. Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous task that would require me weeks to complete, even with the improved automation we built lately. Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM. This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era). The API traffic is still being partially served by mod_php, but that will not be for long! Cheers, Giuseppe -- Giuseppe Lavagetto Wikimedia Foundation - TechOps Team ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] All non-api traffic is now served by HHVM
Amazing work. Kudos for this staggered deployment! On Dec 3, 2014 6:04 PM, Giuseppe Lavagetto glavage...@wikimedia.org wrote: Hi all, it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature. Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous task that would require me weeks to complete, even with the improved automation we built lately. Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM. This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era). The API traffic is still being partially served by mod_php, but that will not be for long! Cheers, Giuseppe -- Giuseppe Lavagetto Wikimedia Foundation - TechOps Team ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] All non-api traffic is now served by HHVM
On 3 December 2014 at 17:03, Giuseppe Lavagetto glavage...@wikimedia.org wrote: it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature. Excellent! Excellent! Does that make us the second-largest HHVM site? What sort of bugs needed squashing? (Is there a list, or a suitable Phabricator query?) This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era). How is the 1.23 tarball under HHVM? Would backports of fixes be accepted for the LTS? - d. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] All non-api traffic is now served by HHVM
On Wed, Dec 3, 2014 at 8:19 PM, David Gerard dger...@gmail.com wrote: What sort of bugs needed squashing? (Is there a list, or a suitable Phabricator query?) A HHVM tag exists: https://phabricator.wikimedia.org/tag/hhvm/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] All non-api traffic is now served by HHVM
Very nice. The impact can is also reflected and easy to spot in the Cluster CPU graphs on Ganglia: https://ganglia.wikimedia.org/latest/?r=monthc=Application+servers+eqiad https://ganglia.wikimedia.org/latest/?r=yearc=Application+servers+eqiad — Krinkle On 3 Dec 2014, at 17:03, Giuseppe Lavagetto glavage...@wikimedia.org wrote: Hi all, it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature. Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous task that would require me weeks to complete, even with the improved automation we built lately. Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM. This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era). The API traffic is still being partially served by mod_php, but that will not be for long! Cheers, Giuseppe -- Giuseppe Lavagetto Wikimedia Foundation - TechOps Team ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
