============================================ #fedora-meeting: Infrastructure (2015-01-15) ============================================
Meeting started by nirik at 18:00:27 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2015-01-15/infrastructure.2015-01-15-18.00.log.html . Meeting summary --------------- * aloha (nirik, 18:00:27) * New folks introductions and Apprentice tasks. (nirik, 18:03:34) * Applications status / discussion (nirik, 18:09:01) * blockerbugs nearing rhel7/ansible migration ready (nirik, 18:10:35) * LINK: http://play.grafana.org/ does look nice (pingou, 18:23:04) * Sysadmin status / discussion (nirik, 18:23:40) * Upcoming Tasks/Items (nirik, 18:30:42) * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/ (nirik, 18:30:42) * An App overview - koji (nirik, 18:35:34) * Open Floor (nirik, 18:58:26) * LINK: https://apps.fedoraproject.org/notifications (threebean, 19:01:14) Meeting ended at 19:06:54 UTC. Action Items ------------ Action Items, by person ----------------------- * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * nirik (124) * threebean (31) * herlo (28) * pingou (25) * mizdebsk (18) * sborza (17) * smooge (13) * dgilmore (9) * zodbot (7) * tflink (6) * michel_slm (5) * mhurron (3) * fale (3) * webpigeon (1) * relrod (1) * mirek-hm (1) * abompard (1) * puiterwijk (0) * abadger1999 (0) * lmacken (0) * mdomsch (0) -- 18:00:27 <nirik> #startmeeting Infrastructure (2015-01-15) 18:00:27 <zodbot> Meeting started Thu Jan 15 18:00:27 2015 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:27 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:27 <nirik> #meetingname infrastructure 18:00:27 <nirik> #topic aloha 18:00:27 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:27 <zodbot> The meeting name has been set to 'infrastructure' 18:00:27 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:32 * pingou here 18:00:58 <smooge> here 18:01:04 <webpigeon> o/ 18:01:19 * threebean 18:01:39 * relrod here 18:02:55 * michel_slm here 18:03:22 * sborza here 18:03:24 <nirik> alright, I guess lets get started then... 18:03:26 * herlo here 18:03:34 <nirik> #topic New folks introductions and Apprentice tasks. 18:03:42 <nirik> any new folks like to introduce themselves? 18:03:48 <nirik> or apprentices with questions or comments? 18:04:10 * sborza raises hand 18:04:23 * fale is here too 18:04:28 * abompard here too :-) 18:04:29 <sborza> my ssh access may not be working now that i've moved accounts? 18:04:50 <nirik> sborza: it should have activated on the new account... we can debug after the meeting in #fedora-admin? 18:04:55 <mhurron> actually, management has moved from puppet to ansible, correct? 18:04:56 <sborza> seems lke the pub key is getting denied...i've updated the config based on: http://infrastructure.fedoraproject.org/infra/docs/sshaccess.txt 18:05:04 <sborza> nirik: sure thing, thanks 18:05:27 <pingou> mhurron: is moving, yes 18:05:34 <nirik> mhurron: we are in the process of doing so yeah... 18:05:43 <nirik> we have ~50 machines left in puppet... 18:05:45 <mhurron> the apprentice wiki page has a workflow with puppet, is there one for ansible? or a page that details what is in puppet and what is in ansible? 18:06:09 * tflink shows up late :-/ 18:06:13 <nirik> there's a migration page for puppet->ansible move, but yeah we should update things for ansible 18:06:44 * herlo blames tflink 18:06:48 <herlo> :) 18:07:26 <nirik> mhurron: I'll try and update that page, or provide info to someone else who wants to. ;) 18:07:39 <pingou> <hint> <hint> 18:07:47 <tflink> herlo: penalty for showing up late is being blamed for ... everything? 18:07:53 <herlo> tflink: exactly. :D 18:07:59 <herlo> tflink: it shows we love you. 18:08:03 <nirik> ha 18:08:13 <mhurron> thanks ... i'll do it if I can get the information 18:08:29 <tflink> huh, didn't realize it worked that way 18:08:37 <nirik> mhurron: excellent. Happy to answer questions in #fedora-admin, #fedora-noc, etc. ;) 18:08:45 <herlo> tflink: who better than those that care to give you a hard time? 18:09:01 <nirik> #topic Applications status / discussion 18:09:11 <nirik> any application news, status or info this week? 18:09:12 * tflink will have to remember that one :) 18:09:31 <tflink> the last el7 builds we needed for blockerbugs have been done and are in epel7-testing 18:09:40 <herlo> nirik: woud new application suggestions go here? 18:09:57 <tflink> waiting on final review for application changes to support el7 but that should be done this week 18:10:19 <nirik> tflink: awesome. Happy to work on migrating it. 18:10:21 <nirik> herlo: sure. 18:10:35 <nirik> #info blockerbugs nearing rhel7/ansible migration ready 18:10:42 <herlo> I wanted to propose we have a look at d-note. https://pthree.org/2014/06/09/officially-announcing-d-note-version-1-0/ 18:10:54 <herlo> This is what it does: https://secrets.xmission.com/ 18:11:06 <herlo> and it's on github here: https://github.com/atoponce/d-note 18:11:15 <herlo> I'm working on packaging it for Fedora now. 18:11:39 <pingou> I've been mostly working on pkgdb2 this week 18:11:55 <nirik> herlo: interesting. 18:11:58 * herlo is willing to deploy and maintain it... 18:12:07 <pingou> to adjust it as discuss with rel-eng to migrate the new package / new branch processes to pkgdb (instead of bugzilla) 18:12:17 <herlo> of course, I'll need some ansible guidance. :) 18:12:34 <nirik> herlo: well, you need a few more folks. I am very much not willing to deploy anything that only one person works on. 18:12:43 <nirik> and we need to decide if it would be of use to us. 18:12:49 <herlo> nirik: sure... which is why I am bringing it up now. 18:12:51 <nirik> but yeah, packaging it up would be first step 18:13:12 <sborza> in that line, i'd like to propose we look at graphite-web for metrics visualization: https://github.com/graphite-project/graphite-web 18:13:21 <herlo> nirik: the conversation of usefulness is in the concept of being able to share secrets in a place we trust. 18:13:21 <sborza> as well as updating our (way out of date) collectd configs 18:13:29 <herlo> sborza: we use graphite at LF. It's pretty nice. 18:13:33 <pingou> herlo: your secrets.xmission link doesn't work here 18:13:49 <herlo> pingou: really? 18:13:51 <herlo> weird... 18:13:57 <sborza> herlo: a lot of changes happening now, would love to get this added to the EPEL repos too 18:13:57 <fale> it works on my pc 18:13:58 <herlo> pingou: it's https. 18:14:02 <pingou> yes 18:14:06 <herlo> sborza: indeed... 18:14:17 <nirik> herlo: can you also use that to make open ones that can stay around? or are they always destroyed on one view? 18:14:27 * nirik is wondering if it could replace paste. 18:14:34 <herlo> pingou: well, you can stand up an instance quickly. from my repo at https://github.com/herlo/d-note 18:14:42 <sborza> nirik: would be happy to work with someone to get this tested/integrated (re: graphite-web) 18:14:43 <herlo> nirik: no. It's not intended for that. 18:14:49 <nirik> ok. 18:14:58 <herlo> nirik: he does have plans for user profiles and such. 18:15:09 <threebean> sborza: I'm interested in it, but can't volunteer too much time. there's lots to consider. 18:15:18 * nirik notes we have a RFR process for these things. ;) 18:15:25 <herlo> nirik: and there's a possibility of allowing multiple people to see the note. Only each instance would be deleted upon viewing, I think. 18:15:26 <threebean> sborza: most of the work would probably be tied up in porting our custom plugins over. 18:15:39 <herlo> nirik: I just wanted to get a feeling if it would be useful first. :) 18:15:52 <sborza> threebean: fair enough, doesn't have to be asap but at some point soon 18:16:00 <nirik> herlo: sure. 18:16:10 <fale> It can create both private and public notes, but it missing the code highlight feature 18:16:27 <nirik> I'm not saying no to anything, just noting that we are very busy. :) 18:16:39 <threebean> sborza: first thing to look into would be requesting that all the components/packages get branched for EPEL7 (they already exist in EPEL6 which is nice) 18:17:08 <sborza> threebean: sounds good re: collectd...if we want the same for graphite-web should I do that myself? 18:17:21 <sborza> threebean: full disclosure, i'm part of the graphite-web steering committee :) 18:17:32 <herlo> nirik: as usual. Busy people are busy. :) 18:17:48 <threebean> sborza: cool ;p. yeah, I actually just meant graphite-web needs the branch requests. we already have collectd on el7. 18:17:55 * herlo is done proposing. The interest level is enough to get me started for sure. 18:18:19 <sborza> threebean: awesome, let's talk more in #fedora-admin (or #fedora-devel?) 18:18:40 <threebean> sborza: -admin works for this 18:18:58 <nirik> herlo: it looks like a cool thing in any case. Even if we don't have a instance. 18:18:59 <pingou> sborza: can it handle any apps, other than web app? 18:19:26 <michel_slm> sborza: are you planning to also package a prettified front-end like Team Dashboard or Graphene? 18:19:36 <herlo> nirik: right. The point I was hoping to make was that we could have 'locations' of trustiness. 18:19:49 <nirik> sure, could be handy 18:20:47 <nirik> ok, any other application type news? 18:21:04 <sborza> pingou: not sure what you're asking, example? 18:21:20 <pingou> sborza: we currently use collectd to monitor the load on fedmsg 18:21:39 <pingou> and on the queue of some application relying on fedmsg 18:21:45 <sborza> michel_slm: can do...graphene is awesome, metrilyx is decent and grafana is phenomenal 18:22:40 <threebean> sborza, pingou: those are the custom plugins I was talking about. I'm almost certain graphite's tools can handle those cases. 18:22:47 <sborza> pingou: graphite-web can handle a variety of metrics collectors, including anything custom 18:22:56 <pingou> cool :) 18:23:04 <pingou> http://play.grafana.org/ does look nice 18:23:07 <michel_slm> sborza: awesome, let me know if you need reviewers 18:23:34 * mirek-hm is late but here 18:23:40 <nirik> #topic Sysadmin status / discussion 18:23:49 <nirik> ok, on the sysadmin side... the usual stuff. ;) 18:24:06 <nirik> We got a few bad drives replaced yesterday. Thanks puiterwijk for working with folks to get that done 18:24:26 <nirik> I've been working on getting the new rhel7/ansible koji hubs done so we can switch to them. 18:24:32 <sborza> michel_slm: fo sho, will do 18:24:41 <nirik> smooge has been burried in hardware ordering and budget fun. ;) 18:25:07 <smooge> hardware is ordered 18:25:13 <smooge> s390 box is being racked 18:25:16 <nirik> we also got a gigantic spike in web requests this morning that saturated our proxies for about 10min or so. Not sure what caused it... 18:25:20 <smooge> qa boxes will be installed this afternoon 18:25:22 <smooge> I broke DNS 18:25:42 <nirik> smooge: thats a new s390 koji hub? 18:25:54 <smooge> yes. I am getitjng it installed in our new racks 18:26:05 <nirik> I really would like to set it up with ansible and in our normal machine setup 18:27:32 <smooge> I have no idea how the current one is setup 18:27:35 <nirik> Nagios has been very noisy of late... hopefully we can settle it down some. 18:27:47 <smooge> since the s390 is somewhere not in redhat 18:27:57 <nirik> smooge: well, when you get the new one installed, let me know. I can talk to sharkcz and we can get it sorted. ' 18:28:15 <nirik> it's up in boston I think... 18:28:40 <nirik> anyhow, I want to get all the secondary stuff setup in ansible and in our normal processes. 18:28:42 <smooge> probably. I thought at one time it was at IBM but not sure 18:28:48 <nirik> so they have 2fa and get regular updates and all that 18:29:01 <smooge> I am fine with that 18:29:44 <nirik> Oh, I made a db01.stg instance... and moved some databases from the old db02.stg to it. 18:30:03 <nirik> there's still a bunch more to move. If any sysadmin-mainer folks want to do that, please feel free. ;) 18:30:42 <nirik> #topic Upcoming Tasks/Items 18:30:42 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:30:52 <nirik> anyone have upcoming tasks they would like to schedule or note? 18:31:26 <pingou> fosdem and devconf in early Feb 18:32:13 <nirik> also smooge and I will be at our main datacenter in feb 18:32:27 <nirik> likely we will need some outages then... to add memory to machines. 18:32:57 <nirik> So, we have some time left in the meeting and I wanted to see about trying something new... 18:33:15 <threebean> oh? 18:33:20 <pingou> suspens :) 18:33:38 <nirik> I thought it would be nice when we have extra time to talk about one specific app of ours. 18:33:45 <nirik> how it's setup, what it does, etc. 18:33:55 <michel_slm> +1 18:33:58 <nirik> This week I could talk about koji or ansible... anyone have a preference? 18:35:20 <michel_slm> nirik: seems like you get to decide 18:35:20 <threebean> hm. cool. 18:35:27 <nirik> ok, I guess I will pick one... koji, since I have been working on it. 18:35:34 <nirik> #topic An App overview - koji 18:35:54 <nirik> so, koji is our buildsystem. It has several parts... 18:36:18 <nirik> a hub that runs a wsgi / httpd application - right now this is koji03 18:36:48 <nirik> a secondary hub (could take over if primary died) that right now runs some cron jobs and kojira 18:36:52 <nirik> this is koji04 18:37:17 <nirik> kojira is a process that launches things like buildroot rebuilds and the like. You can see it in the web interface... jobs owned by it. 18:37:36 <nirik> There's also a pile of builders. They all talk to the hub to get jobs and report on status and such. 18:38:06 <nirik> koji uses a self signed cert with its own CA... because it uses certs to identify builders and people 18:38:33 <nirik> we also have a squid proxy - kojipkgs02 currently that sits in front of the koji packages urls... 18:38:50 <nirik> builders and other things that download from koji hit that squid where it's cached. 18:39:00 <nirik> (for packages and most things) 18:39:27 <nirik> builders run the 'kojid' process. This talks to the hub, and identifies with a cert. 18:40:13 <nirik> Current koji (rhel6/puppet) uses heartbeat to keep a application ip on the active hub (usually koji03) 18:40:28 <nirik> new koji I am working on (rhel7/ansible) will use keepalived 18:40:51 <nirik> Storage for koji is a nfs mounted volume from the netapp. 18:41:12 <nirik> It has to be mounted on the hub and any builders that do newrepo tasks (but not on builders that don't) 18:41:35 <threebean> question 18:41:35 <nirik> Thats kind of the high level... questions? 18:41:57 <threebean> those builders that do newrepo tasks 18:42:01 <threebean> do they also do other tasks? 18:42:04 <nirik> yep. 18:42:08 * threebean nods 18:42:12 <nirik> koji has the concept of 'channels' 18:42:28 <nirik> you can setup channels for things and then tell it specific builders do those channels. 18:42:32 <threebean> so, a job submitted by joe-user can potentially get run on a builder that has write-access to the netapp mount? 18:42:55 <nirik> yep. It would be in a mock chroot tho. 18:43:02 * threebean nods 18:43:08 <threebean> cool. 18:43:10 <nirik> that chroot shouldn't have any access... 18:43:34 <pingou> ! 18:43:48 <nirik> We do have some special channels/rules... there's some special builders for secure boot for example that only do builds of packages that need secure boot signing. 18:44:10 <nirik> pingou: yes? 18:44:20 <pingou> Are all the koji hosts el6 still? (can the builders be moved to el7 separatly from the masters? (Although I imagine heartbeet/keepalive might be the problem)) 18:45:02 <nirik> pingou: most of the builders are now f21. ;) 18:45:07 <pingou> oh true :) 18:45:36 <nirik> all the arm ones, the buildvm and buildhw ones are all f21. 18:45:49 <nirik> the bkernel ones are still f20. I need to upgrade them. 18:46:09 <nirik> the buildppc ones are rhel6 I think... but we only need ppc for epel 18:46:53 <nirik> koji03/04 and kojipkgs02 are all rhel6/puppet. I am working now to move them to koji01/02, kojipkgs01 that is ansible and rhel7 18:48:13 <nirik> oh, there's also a db host... I already moved it to rhel7 18:48:54 <nirik> ok, any other questions? does someone want to do some other app next week? :) 18:49:47 <threebean> oh - here's a question. the new koschei service seems to be hitting koji pretty hard. 18:50:13 <threebean> is koji handling that new load OK? 18:50:18 <nirik> yeah, it has a limiter... I think it only ever does 40 jobs at once or something. 18:50:25 <pingou> .load 18:50:25 <zodbot> pingou: Error: You don't have the owner capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified. 18:50:28 <nirik> yeah, perfectly fine. 18:50:31 <mizdebsk> 30* 18:50:54 <nirik> it's actually been nice when I had a problem with the f21 buildvm's... I could count on it submitting jobs so I could see if we fixed it or not. 18:51:01 <nirik> mizdebsk: sorry, my mistake. ;) 18:51:01 <threebean> ;p 18:51:08 <mizdebsk> and koji load must be <50% at the time of executing new build by koschei 18:51:19 <mizdebsk> all that is configurable, feedback is welcome 18:51:42 <nirik> .buildload 18:51:43 <zodbot> nirik: Load: 116.9 Total: 272.0 Use: 43.0% (Medium Load) 18:52:16 <threebean> mizdebsk: looks fine here. just curious about how its playing out. 18:52:16 <nirik> I think it's been fine. I haven't seen any issues with it. 18:52:41 <nirik> During mass rebuilds we get a nice backlog (submitting faster than it can build), but even then it chruns thru them pretty fast. 18:52:48 <nirik> we have 91 builders currently enabled. 18:53:06 <pingou> .builders 18:53:06 <zodbot> pingou: Enabled: 91 Ready: 78 Disabled: 69 18:53:26 <pingou> being used == Enabled - Ready? 18:53:42 <nirik> 46 arm, 27 buildvm, 12 buildhw, 2 bkernel, 2 buildppc 18:53:58 <mizdebsk> ready means load is low and builder is accepting tasks 18:54:12 <nirik> pingou: koji uses a loading setup... so you can tell it how much load a builder can handle 18:54:22 <nirik> when it gets that load or more it stops being ready to accept new jobs. 18:54:35 <pingou> ok 18:54:54 <nirik> % koji list-hosts 18:54:54 <nirik> Hostname Enb Rdy Load/Cap Arches Last Update 18:54:54 <nirik> arm02-builder00.arm.fedoraproject.org Y N 6.0/2.0 armhfp 2015-01-15 18:53:06 18:55:09 <mizdebsk> it's obvious that koschei needs hardware - there were plans to add new hardware to koji instead having separate pool just for koschei 18:55:13 <nirik> so that builder can handle a load of 2.0, but it has a 6.0 job... so its not ready currently 18:55:36 <mizdebsk> also during mass rebuilds i'm suspending koschei 18:55:51 <pingou> nirik: cool, thanks 18:56:28 <nirik> mizdebsk: I wonder if we could do something with out staging koji... (which is currently not working). If it got regular builds it would help us know it's working along ok 18:57:08 <nirik> just an idle thought. 18:57:30 <nirik> pingou / threebean: either of you have an app you might want to talk about next week? ;) 18:57:34 <mizdebsk> maybe stagging koschei could use it? i started playing with ansible setup for koschei, but i don't have access to bastion and such 18:57:35 <threebean> sure :P 18:57:40 <smooge> mizdebsk, you said there were plans ... 18:57:45 <smooge> have they changed? 18:57:47 <pingou> nirik: I might be able to find one, maybe two ;-) 18:58:17 <nirik> mizdebsk: might work. I need to fix the issue where it's not starting builders, but then it would be good to look into 18:58:20 <nirik> cool. 18:58:26 <nirik> #topic Open Floor 18:58:32 <nirik> Anyone have anything for open floor? 18:58:33 <mizdebsk> smooge, currently i've been told that there will be budget for new hardware if it's needed 18:58:56 <mizdebsk> but so far it wasn't necessary - afaict koji is handling current load just fain 18:59:12 <mizdebsk> fine* 18:59:14 <threebean> mizdebsk: would you like a 'koschei' cert for koschei, so it can submit builds as itself instead of msimacek ? 18:59:27 <mizdebsk> threebean, yes, i've filled ticket for that 18:59:33 <threebean> cool, cool. 18:59:53 <smooge> mizdebsk, thanks. I just wanted to make sure if I needed to put in an order before the end of the year or not (or prepare for boxes to arrive) 19:00:00 <mizdebsk> threebean, https://fedorahosted.org/rel-eng/ticket/5941 19:00:14 * pingou gtg 19:00:20 <nirik> we really need to get thru more releng tickets in the releng meetings. ;) we will get there 19:00:25 <pingou> thanks for the meeting everyone, thanks for chairing nirik 19:00:25 <mizdebsk> i assume cert will be generated as part of RFR 19:00:58 <nirik> mizdebsk: yeah, we can get it sorted. 19:01:05 <nirik> ok, thanks for coming everyone... 19:01:11 <threebean> A couple things I forgot to mention in the apps section 19:01:12 <nirik> will close in a min if nothing else. 19:01:12 <threebean> 1) A new release of FMN got deployed as promised last week. Lots of bugfixes and enhancements, but mostly a new set of nice defaults: 19:01:14 <threebean> https://apps.fedoraproject.org/notifications 19:01:15 <nirik> oh, go ahead 19:01:16 <threebean> You there's a button on each messaging context that let's you reset your filters to the new defaults if you'd like to test it out. 19:01:18 <threebean> 2) A new release of the-new-hotness got deployed to stg that follows up on bugs about real builds, not just scratch builds. There's a little more work to be done before we can put it in production. Should be done with that in a week or two. 19:01:20 <threebean> 3) We had some issues with FAS in fedmsg the past few weeks. Two of the fas servers weren't publishing any messages. Should be all fixed now. 19:02:21 <nirik> threebean: thanks for tracking that down. 19:02:21 <threebean> (heh, that's all.. ;p) 19:02:33 <threebean> np 19:02:36 <dgilmore> mizdebsk: its causeing issues 19:02:39 <nirik> I really need to redo fas01 too... but I want to try and confirm the cert stuff is all ok in asible 19:02:41 <nirik> ansible 19:02:53 <dgilmore> mizdebsk: we did some things to have java builds mostly go to x86 19:03:19 <dgilmore> mizdebsk: and because of the load of the tasks its not happening causeing complaints from the java folks 19:03:45 <nirik> dgilmore: huh... was this eclipse folks? or ? 19:04:01 <dgilmore> nirik: no. the people doing other java builds 19:04:04 <mizdebsk> dgilmore, koschei runs some tasks with --arch-override x86_64 because arm builders are not usable for java imho 19:04:10 <mizdebsk> that may be the reason 19:04:36 <dgilmore> mizdebsk: its causing issues 19:04:39 <nirik> ok, because one of the arm builders in the eclipse channel is down with a dead drive... so it's only got 1 left I think, making it slow 19:05:06 <dgilmore> java on arm is missing a jit so its slow 19:05:08 <mizdebsk> dgilmore, but this is really issue related to arm and java, not koschei itself 19:05:21 <mizdebsk> are there are any other issues with koschei? 19:05:33 <dgilmore> it is something that is going to need some planning and coordination 19:05:48 <nirik> perhaps we could continue this over in #fedora-releng? we are over time on our meeting. ;) 19:05:56 <dgilmore> mizdebsk: yes but koschei is breaking what we are doing to deal with it 19:06:18 * mizdebsk is fine with moving to different channel 19:06:49 <dgilmore> nirik: sure 19:06:50 <nirik> Thanks for coming folks. 19:06:54 <nirik> #endmeeting
pgpsG2TEIFWR1.pgp
Description: OpenPGP digital signature
_______________________________________________ infrastructure mailing list [email protected] https://admin.fedoraproject.org/mailman/listinfo/infrastructure
