Thanks a lot Christian :) I had not meant by any mean last Friday to overload the cluster ... I did it nonetheless. Your page on how to 'keep an eye on it' will really be useful! Cheers Joseph
On Sun, Mar 8, 2015 at 8:26 PM, Leila Zia <[email protected]> wrote: > This is really useful, Christian. Thanks for explaining and documenting it. > > Leila > > On Sat, Mar 7, 2015 at 6:14 AM, Christian Aistleitner < > [email protected]> wrote: > >> Hi, >> >> around running jobs on the Analytics cluster, I've sometime seen >> people say in IRC: “Let's run this heavy job. I'll keep an eye on it”. >> >> But more often than not, this seems to have meant: >> “Let's just run this heavy job and wait. If QChris joins IRC, let's >> hope he doesn't ping us about having overloaded the cluster.” >> >> That's not nice^Wscalable ;-) >> >> So just in case someone is vague on how to “keep an eye on it”, I did >> a short write-up at: >> >> https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Load >> >> which details on detecting how the cluster is doing on a very high >> level. >> Especially, it allows you to detect if the cluster got stalled, and if >> it did, it tells you what to do. >> >> Have fun, >> Christian >> >> P.S.: The above URL has diagrams! Click the URL! >> >> -- >> ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- >> Companies' registry: 360296y in Linz >> Christian Aistleitner >> Kefermarkterstrasze 6a/3 Email: [email protected] >> 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 >> Fax: +43 7946 / 20 5 81 >> Homepage: http://quelltextlich.at/ >> --------------------------------------------------------------- >> >> _______________________________________________ >> Analytics mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/analytics >> >> > > _______________________________________________ > Analytics mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/analytics > > -- *Joseph Allemandou* Data Engineer @ Wikimedia Foundation IRC: joal
_______________________________________________ Analytics mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/analytics
