The IRC channel is #sakai on the freenode network, you can join using a web client at http://webchat.freenode.net/?channels=sakai and you can view the complete logs at http://sakai.iriscouch.com/irc/_design/viewer/index.html
15:03 <SakaiGithub> [nakamura-ruby/master] Merge pull request #21 from croby/KERN-2979 - D. Stuart Freeman 16:13 <raydavis> But zacht, what was this totally awesome conference called? 16:16 <zacht> raydavis No Fluff Just Stuff. 16:16 <raydavis> Thanks! 16:17 <zacht> raydavis I've actually been twice this year. I got a volume discount. :-) 16:21 <raydavis> Woot! All integration tests pass! I can't remember the last time I saw that. :) 16:21 <raydavis> Thanks for getting the lid on those, zacht 16:21 <zacht> raydavis My pleasure. 16:22 <zacht> raydavis Now to roll it into continuous integration. 16:22 <ctweney> zacht: what was that link for sonar again? 16:23 <stuartf> I'm actually seeing one integration test fail here 16:23 <zacht> http://sakai-oae.sonar.cloudbees.com 16:23 <stuartf> test_paging_of_messages(TC_Kern2113Test) [/home/stuart/src/nakamura/testscripts/SlingRuby/kerns/kern-2113.rb 16:23 <raydavis> stuartf does it pass when run separately? 16:23 <stuartf> raydavis: I haven't tried that 16:24 <raydavis> stuartf that tells me if it's some sort of timing/threading problem. 16:25 <ctweney> zacht: sonar wants me to log in… 16:25 <stuartf> running it standalone now 16:25 <raydavis> Usually a problem with the test rather than with OAE, although there have been exceptions. 16:25 <zacht> ctweney Yeah, as I mentioned on the standup call, you won't be able to see it. 16:26 <MrVisser> integration tests have bailed me out quite a bit before I commit code. 16:26 <MrVisser> glad to see them restore confidence :) 16:27 <raydavis> mrvisser, me too, but it's been a drag having to keep a copy of all the tests-that-failed-before-my-change for comparison. 16:27 <MrVisser> raydavis: for sure. 16:28 <stuartf> raydavis: yeah, it passes when run by itself 16:29 <raydavis> stuartf, great (kind of), that means there's a bug in the test code. Should get fixed to avoid false positives in continuous integration. 16:30 <raydavis> (I think I've found maybe a half-dozen times when it was a genuine contention bug in Nakamura/Sparse, but usually it's bad assumptions in the Ruby script.) 16:31 <PhysX> Hi bug bashers! today we'll be testing against the 1.4.0 release candidate. Everything is fair game so try and test a variety of areas in the system. The server is live at https://qa20-us.sakaiproject.org:8088 16:46 <Bob_> Added self from old account as member of group - is there any way to distinguish people with the same name - any unique identifier of who someone is that is visible to users? 16:49 <stuartf> the javascript cut/copy/paste in a sakaidoc doesn't work on firefox 16:50 <stuartf> https://developer.mozilla.org/en/Midas/Security_preferences 16:50 <zacht> Bob_ Their home url will have their unique id in it. Is that what you had in mind? 16:56 <Bob_> zacht - here's the problem - I want to share an item to John Smith, but there are three John Smiths in the system - how do I know which one I'm sharing it with? 16:56 <nicolaas> Bob: this problem is being worked into the current designs 16:57 <nicolaas> will be shown to the URG next week 16:57 <Bob_> Cool - thanks! 16:58 <PhysX> dean440: did you see some strange behavior in your inbox? 16:59 <dean440> PhysX No, I didn't see anything. What are you seeing? 16:59 <dean440> Ah, I see your message 16:59 <stuartf> autosave dialog shows 'yes' as a button and 'no' as a link 17:01 <PhysX> half an hour left bug bashers! 17:08 <Bob_> Oops - screen freeze - happen to anyone else? (Will file as bug with screen shot if not - happened while dragging an item in a collection to another collection) 17:09 <nicolaas> not seeing anything over here 17:10 <stuartf> same here, I did just create a rather large sakai-doc 17:10 <Bob_> Still frozen 17:17 <ctweney> nicolaas, PhysX : I noticed a minor issue with the collector. If I have the collector open while I add a doc to a collection, the count inside the collector doesn't update. Known issue? 17:17 <Bob_> Refreshing unfroze screen - all working normally now 17:18 <PhysX> Bob_ are you able to reproduce the frozen screen? 17:19 <PhysX> ctweney: don't think it's a know issue but we probably don't catch the event that's sent out 17:19 <Bob_> Phys X Haven't been able to so far - will keep trying 17:20 <PhysX> everyone, there's about 10 minutes remaining in today's bug bash. If there's anything you'd still like to try out give it a go 17:21 <dean440> Anyone having 2 collection counts update when you drag an item into a collection? 17:21 <PhysX> dean440: library and collection? 17:21 <dean440> No, 2 different collections. 17:22 <dean440> PhysX ^ 17:22 <PhysX> dean440: yeah I can reproduce it too 17:22 <PhysX> can you make a ticket? 17:22 <dean440> PhysX Yep 17:22 <stuartf> oops: http://i.imgur.com/p5Nfw.png 17:23 <nicolaas> stuartf: I haven't seen that yet 17:23 <Bob_> Minor counter issue - if I drag an item from a collection into My Library (in the Collector window) that's already in the Library, I get the counter going up each time (like if I drag it in three times, the counter goes from 15 to 18) - but then when I go to the Library, there is (properly) only one copy and the counter is at 15, where it shoud be 17:24 <Bob_> dean 440 - similar to your issue? 17:24 <stuartf> I was pasting 3M of text (War and Peace) and chrome didn't like it 17:25 <dean440> Man, Collection Counts are buggy. >_< 17:26 <Bob_> Now it's counting by twos - I drag an item from a Collection into the Library collection, and the counter goes up by two. - but, again, when I open the Library itself, the counter is correct 17:28 <zacht> ctweney What's the path to the activity feed? 17:30 <zacht> ctweney Nevermind. I've got it. 17:30 <PhysX> it's time to wrap up the bash everyone, thank you for participating. Please file any issues that you encountered. 17:37 <zacht> ctweney assigned KERN-3050 17:38 <Bob_> S' Long 17:39 <stuartf> ha, I got it to work: https://qa20-us.sakaiproject.org:8088/content#p=mmQeqyQOKs/War%20and%20Peace 17:53 <ctweney> zacht, kentfitz, the reason for kern-3050 seems to be that the activitysource field doesn't exist in the solr index 17:53 <kentfitz> ctweney: As in it's not getting indexed ever? 17:54 <ctweney> as in it's not part of the schema, I don't think 17:54 <zacht> ctweney But it's in the indexing handler? 17:54 <ctweney> it should be, let me double check 17:54 <ixmati> I'm not sure I'll be able to finish War & Peace today 17:55 <zacht> ixmati Hey, that's just what the preview processor said! 17:55 <ctweney> yes zacht it's in the indexer. So I'd expect to see a few errors in the log pertaining to that 17:56 <zacht> ctweney I'll try a grep now. 17:56 <ctweney> kentfitz, zacht, did qa20 recently change to have solr on a separate server? 17:56 <kentfitz> ctweney: It did not, never had time to get that configured 17:57 <ixmati> zacht: LOL 17:57 <ctweney> ok, there goes one hunch.. 17:59 <zacht> kentfitz Which directory is oae running from on qa20? 18:00 <kentfitz> weird, last message did not send. It was launched from /home/sakai/13 18:00 <kentfitz> the code is in /home/sakai/14 18:02 <zacht> kentfitz if you begin a message with a forward slash, IRC interprets it as an IRC command. 18:03 <kentfitz> zacht: that would have done it 18:03 <stuartf> /you can start the line with a space though 18:04 <zacht> ctweney no mention of ActivityIndexer in the log. Is there something else that I should look for? 18:05 <ctweney> zacht: try another grep for ActivityIndexing 18:12 <zacht> ctweney I think I've got something. 18:12 <zacht> ctweney I created a new DEBUG logger for activity search. 18:13 <zacht> ctweney When I added a piece of content, it was ResourceUpdateIndexingHandler which popped up in the log. 18:14 <zacht> ctweney But ActivityIndexingHandler is the only one that makes any mention of activitysource 18:14 <ctweney> ResourceUpdate is just for the most-active-content feed 18:14 <ctweney> which is unrelated 18:15 <ctweney> you don't see ActivityIndexingHandler fire at all? 18:15 <zacht> ctweney no. 18:16 <ctweney> very strange.. but obviously it must have worked for some documents, because they're there in the search results! 18:16 <ctweney> err wait 18:16 <ctweney> no, the search when you're logged in has results 18:16 <ctweney> although it seems like it shouldn't 18:17 <zacht> ctweney If you want to try some ad hoc solr queries, you can here: http://qa20-us.sakaiproject.org:8080/system/console/ 18:19 <zacht> ctweney I'm now seeing some ActivityIndexingHandler action in the log. 18:20 <ctweney> curious, zacht, did somethign change or is it just catching up with a bunch of activity? 18:21 <zacht> ctweney No, they're all for the same authorizable: a:selenium/private/activityFeed 18:23 <zacht> ctweney Ok, I did see a log message for a piece of content I added. http://pastebin.com/7u9bkK9S 18:25 <zacht> ctweney Could the problem be that it's using sakai:activity-source for the property name? 18:25 <ctweney> no - that's the sparse property name. In Solr that gets translated to "activitysource" 18:25 <ctweney> zacht: can you grep with a few lines of context around that last log entry you pasted? I'd expect to see some errors from Solr around there. 18:26 <ctweney> since in the Solr query console if you search "activitysource:*" it says that field does not exist. 18:29 <zacht> ctweney I think I need logging on a different package. 18:29 <zacht> ctweney Though that error tells us there is something wrong with our schema definition. 18:30 <ctweney> actually zacht the content of that Solr doc is wrong. That looks like an obsolete version of ActivityIndexingHandler. 18:31 <ctweney> zacht: here's what a proper log entry should look like : http://pastebin.com/LL27LUtx 18:32 <ctweney> note the activitysource field 18:33 <stuartf> zacht: so, should I just make a branch and change all the 1.4.0-SNAPSHOT to 1.5.0-SNAPSHOT? or do it in an existing branch? 18:33 <zacht> stuartf make a branch. 18:34 <zacht> stuartf the caveat I was going to tell you about is that you'll find the our solr bundle version number at 1.4.3-SNAPSHOT (or something close to that). It should also become 1.5.0-SNAPSHOT. 18:34 <stuartf> ok, and was there some other thing you hinted at on the call? 18:34 <stuartf> ah 18:34 <stuartf> got it 18:37 <zacht> ctweney I'm going to test a hunch. 18:50 <stuartf> zacht: it looks like solr is already at 1.5.0-SNAPSHOT in master 18:51 <zacht> stuartf ok. No problem then. 18:55 <zacht> ctweney I think the build didn't have your changes in it for KERN-3036 19:41 <zacht> kentfitz I have somehow broken qa20 19:41 <kentfitz> zacht: way to go! ;) 19:41 <zacht> I know. 19:42 <MrVisser> my god puppet is awesome 19:42 <zacht> kentfitz I wanted to start up with a rebuilt 1.4.0 jar, but I got the "Cannot authenticate request." problem. 19:43 <MrVisser> it… it… it…. just works. 19:43 <MrVisser> no surprises. 19:43 <kentfitz> zacht: Where did you launch from? 19:43 <zacht> kentfitz /home/sakai/13 19:44 <kentfitz> zacht: Hmmmm, yeah, something isn't right, it's using derby rather than postgres. Let me look a little further 19:44 <zacht> kentfitz So I tried starting with a new sling directory. 19:45 <zacht> kentfitz So at a minimum, it needs the correct config copied back into it. 19:45 <kentfitz> zacht: Ahhhhhh, well, that explains derby, but it's going to need a clean DB as well if you did that 19:45 <zacht> kentfitz How do we use the existing data set? 19:46 <kentfitz> zacht: I'd have to restore the 1.3 dataset then re run the entire migration 19:46 <kentfitz> zacht: I was waiting to snapshot 1.4 until we released 19:46 <zacht> kentfitz I don't mind running the migration. 19:47 <kentfitz> zacht: run /home/sakai/13/scripts/wipeandrestorefromsavedstate.sh 19:47 <zacht> kentfitz++ 19:47 <kentfitz> zacht: That will drop the db, restore from backup, and restore the sling and store directories from backup 19:49 <zacht> kentfitz And then I follow the steps on https://confluence.sakaiproject.org/display/3AK/Upgrading+OAE ? 19:50 <kentfitz> zacht: Yep. So launch the 1.3 code from home/sakai/13 for the first step. Then launch the code from home/sakai/14 for the remainder 19:51 <zacht> kentfitz got it. Thanks. 19:51 <kentfitz> home/sakai/13/startsakai.sh is set up to launch the 1.4 code in that 14 directory, but the 1.3 start up is commented out if you want to use it 19:51 <zacht> kentfitz ok, I'll use that. 19:53 <zacht> mrvisser are you using a puppet master? 19:53 <zacht> mrvisser I got a good demo of that over the weekend. 19:54 <MrVisser> zacht: no puppet master involved. for now I am the puppet master. 19:54 <MrVisser> zacht: I have munin server + nodes installed on the cluster nodes, which will monitor system resources 19:55 <MrVisser> and it just worked. no tears 19:59 <zacht> kentfitz reindex underway. I'm going to step out for a bite to eat while it's running. 19:59 <kentfitz> zacht: Makes sense, probably 30-40 minutes there 19:59 <kentfitz> zacht: I have to leave in about 30, but I'll try to check mail periodically this evening 20:04 <ctweney> huh, zacht and kentfitz, qa20's got some oddities in /var. specifically, /var/scm-version.json is 404. 20:05 <kentfitz> ctweney: it's in the middle of the upgrade steps now, so I wonder if that's related 20:06 <ctweney> could be.. though /var should be loaded as soon as all the bundles start up. 20:44 <zacht> ctweney All the upgrade steps are complete. 20:44 <zacht> ctweney And recent activity works. :-) 20:45 <ctweney> I still smell a rat on qa20, zacht 20:45 <ctweney> try clicking the Sakai logo at lower left to get the revision number 20:45 <ctweney> the file that feeds that is 404 20:45 <zacht> ctweney comes from the doc bundle, right? 20:45 <ctweney> /var/scm-version.json 20:45 <ctweney> it's from the resources bundle actually 20:47 <zacht> ctweney http://qa20-us.sakaiproject.org:8080/system/console/contentreloader 20:47 <zacht> ctweney It's that weird symptom we saw last week where not all the initial content went in. 20:47 <ctweney> weird 20:48 <zacht> ctweney except this time, I didn't add /var using curl. 20:48 <ctweney> on my local build (fresh, not an upgrade) the resources bundle doesn't appear in content loader, although the /var/scm-version file is there as expected 20:49 <ctweney> zacht: did you delete /var during the upgrade? 20:49 <zacht> ctweney yeah. 20:50 <ctweney> and you deleted sling/startup and sling/felix per usual? 20:50 <zacht> ctweney Yes. Should I try re-installing bundles a la carte? 20:51 <ctweney> unfortunately I think resources is one of those you can't reinstall w/o a restart 20:51 <ctweney> but it might be worth a try 20:51 <ctweney> I'm getting nervous about the content loading strangeness though 20:52 <ctweney> since it doesn't follow a pattern I can discern 20:52 <zacht> Yeah, it's not cool. 20:55 <zacht> ctweney Here's a clue. I tried stopping and restarting the doc bundle. Stack trace. 20:55 <ctweney> hm, intriguing 20:55 <zacht> ctweney http://pastebin.com/XNfQbqbt 20:56 <zacht> ctweney What do they mean by no child node definition? 21:00 <ctweney> not sure what that means, zacht, but maybe it's to do with /var getting deleted. I'm testing it out on my local box. 21:01 <zacht> /var has definitely been recreated. 21:01 <ctweney> I think scm-version is one of the only files that's a direct child of /var 21:02 <MrVisser> ping froese 21:02 <froese> pong mrvisser 21:03 <MrVisser> hey froese, I need to open up a port. do you have a preferred puppet module for that? there is an iptables puppet module I could use. 21:03 <froese> i don't actually. I'm used to running most of the cluster on a private network 21:04 <froese> but there are iptables modules. if puppet labs distributes on that'd be great 21:04 <MrVisser> ok I'll check that first. thanks. 21:07 <ctweney> zacht: I found a clue here http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201103.mbox/%[email protected]%3E 21:08 <ctweney> zacht: on qa20 /var has "jcr:primaryType": "nt:folder" 21:08 <ctweney> zacht: whereas on my machine it's sling:Folder, which allows for unstructured content underneath 21:09 <zacht> ctweney That explains step 1b. https://confluence.sakaiproject.org/display/3AK/Upgrading+OAE 21:09 <ctweney> yeah 21:09 <ctweney> but when we included step 1b, didn't we have problems with some pieces not loading? 21:11 <zacht> ctweney We assumed that was the cause. 21:13 <ctweney> I'm kind of stumped zacht 21:13 <zacht> ctweney I'm going to delete var, and see if we have better luck with some of these bundles. 21:14 <zacht> ctweney ok, now we have a sling:Folder 21:15 <ctweney> did you explicitly recreate /var zacht? 21:15 <zacht> ctweney I did. 21:15 <ctweney> ok, good experiment.. 21:15 <zacht> ctweney I stopped and started the doc bundle, and now we have /var/scm-version.json 21:16 <ctweney> sadly, /var/search/activity/all.json is now 404 21:16 <ctweney> oh now it's 200 21:16 <zacht> ctweney Not sad, expected! 21:16 <ctweney> I was too quick :) 21:17 <zacht> ctweney According to the content reloader console, all the content is back in place. 21:18 <ctweney> very quick smoke test shows everything ok 21:18 <zacht> ctweney I wonder why this worked before, after kentfitz dropped step 1b from the upgrade. 21:24 <ctweney> zacht: maybe sometimes /var gets recreated as nt:folder and other times as sling:Folder? 21:24 <zacht> ctweney It could depend on the order the bundles start up in. 21:26 <zacht> ctweney The search/impl bundle includes var.json My guess is that if search starts first, its properties will apply. 21:26 <ctweney> that'd be my guess too zacht 21:27 <ctweney> I'll see if I can repro here locally 21:31 <ctweney> this is some strange, strange stuff zacht 21:31 <ctweney> when I delete /var on my local machine there is still content there! 21:31 <zacht> ctweney Did you see any symptoms? 21:31 <zacht> ctweney Woah. 21:31 <ctweney> I delete var thusly: curle -u admin:admin http://localhost:8080/var -F":operation=delete" 21:32 <ctweney> and look at the content! http://pastebin.com/sQyN6bJW 21:33 <zacht> ctweney Does it really have no jcr:primaryType? 21:34 <zacht> ctweney Maybe .1.json is suppressing var's properties. 21:36 <MrVisser> ping froese: trying to add anthony as a user, but it's giving me an error on oae-community.sakaiproject.org: 21:36 <MrVisser> err: /Stage[main]/People::Users/User[arwhyte]/ensure: change from absent to present failed: Could not create user arwhyte: Execution of '/usr/sbin/useradd -u 799 -d /home/arwhyte -G wheel -m arwhyte' returned 9: useradd: group arwhyte exists - if you want to add this user to that group, use -g. 21:37 <froese> don't realize(Group['arwhyte']) in the people/manifest/init.pp 21:37 <MrVisser> froese: I see. why is that? everyone else is.. 21:37 <froese> i think i forgot to take that out 21:38 <froese> it hung me up for a while, i removed it manually on the machine, I think I just forgot to commit it 21:38 <froese> i never commit from the nodes themselves 21:38 <MrVisser> ahh I see. delete everyone's? 21:38 <froese> yeah you can remove the realize(Group* lines 21:38 <ctweney> zacht: can I delete /var on qa20 for an experiment? 21:39 <zacht> ctweney yeah, go for it. 21:41 <MrVisser> froese: that's not working. 21:41 <MrVisser> same error 21:41 <froese> groupdel arwhyte 21:42 <ctweney> zacht: same result on qa20, /var still there after delete : http://pastebin.com/B3nMEwAe 21:42 <zacht> ctweney That's a strange notion of delete. 21:43 <ctweney> we are well and truly through the looking glass without a paddle 21:43 <MrVisser> froese good call. all better. thanks! 21:43 <froese> np. thanks for picking this up! 21:44 <froese> (not assigning it to you mrvisser, just good to have someone else who understands it. 21:44 <froese> ) 21:44 <froese> (i hate (unclosed) parentheses) 21:45 <MrVisser> me too, thanks for fixing it up for me :) 21:46 <_mst> hope everyone's OK with unopened ones :) 21:46 <_mst> community service: ((((((( 21:53 <MrVisser> haha nice… ))))))) 21:54 <MrVisser> froese: anthony is installing the SSL cert right now on oae-performance, is there any puppeting that should be done to accommodate this to avoid wreckage on the next puppet update? 21:56 <froese> yes! 21:56 <froese> if you give me the certs and keys ill put them in puppet 21:56 <froese> then add them to the apache::ghost delcarations 21:56 <froese> *apache::vhost 21:57 <froese> https://github.com/efroese/puppet-oae-example/blob/master/environments/community-cluster/modules/localconfig/manifests/nodes.pp#L64 21:58 <froese> they take more arguments https://github.com/efroese/puppet-apache/blob/master/manifests/vhost-ssl.pp#L35 21:59 <froese> damn you _mst 21:59 <froese> you make my OCD fly off the handle 22:00 <froese> i close my smiley-face parens 22:02 <MrVisser> froese: key and cert actually won't be ready until tomorrow. anthony is generating a csr.. 22:02 <froese> womp womp 22:04 <MrVisser> needs them regenerated because the attrs aren't quite right. 22:14 <GitHub51> [nakamura] zathomas tagged 1.4.0-RC2 at 0baa48c: http://git.io/nxsViQ 22:14 <GitHub51> [nakamura/1.4.0-RC2] SOLR-72 add telemetry, resolve merge conflict - Carl Hall 22:14 <GitHub51> [nakamura/1.4.0-RC2] bind to sparsemap 1.3.6 and solr 1.4.3 - Zach A. Thomas 22:14 <GitHub51> [nakamura/1.4.0-RC2] release_pre_process: Moving config files from 1.4.0-SNAPSHOT to 1.4.0 - Zach A. Thomas 22:17 <GitHub31> [nakamura] zathomas pushed 6 new commits to 1.4.0: http://git.io/h8xq7w 22:17 <GitHub31> [nakamura/1.4.0] SOLR-72 add telemetry, resolve merge conflict - Carl Hall 22:17 <GitHub31> [nakamura/1.4.0] bind to sparsemap 1.3.6 and solr 1.4.3 - Zach A. Thomas 22:17 <GitHub31> [nakamura/1.4.0] release_pre_process: Moving config files from 1.4.0-SNAPSHOT to 1.4.0 - Zach A. Thomas 22:29 <_mst> froese: one of the curses of IRCing from emacs is it highlights the matching paren whenever someone uses a smiley... 22:38 <elicochran> anyone know what's up with qa20? I wanted to regress a bug but the SignIn button appears to be unresponsive. 22:45 <ctweney> elicochran: qa20's a bit hosed at the moment due to some testing zacht and I were doing 22:45 <elicochran> ctweney: thanks 22:45 <zacht> ctweney Do we just need the initial content back? 22:45 <ctweney> I think that's all we need zacht 22:45 <zacht> ctweney I can restart certain bundles, which is what worked last time. 22:46 <ctweney> if that'll restore everything then go for it zacht 22:52 <ctweney> elicochran: you should be good to go on qa20 now 22:52 <ctweney> thanks zacht 22:53 <zacht> ctweney They didn't all come back in a whoosh this time. I had to tell each one of them to update. 22:53 <elicochran> ctweney: thank you, sir! 23:08 <elicochran> ctweney: all done, if you need it back... but I'm guessing you don't 23:24 <MrVisser> froese: are these amazon instance firewalls configured elsewhere? the iptables suggest that everything is accepted, but I'm having trouble opening up some ports. Is there an admin interface where I coud look at firewall settings? 23:26 <froese> kyle campos from rSmart manages it. 23:26 <froese> there are restrictions on communications between hosts 23:26 <froese> I'm not sure what they all are 23:27 <froese> mrvisser, what are you trying to configure? 23:28 <MrVisser> froese: just trying to open 4949/tcp for munin (os monitoring) _______________________________________________ oae-dev mailing list [email protected] http://collab.sakaiproject.org/mailman/listinfo/oae-dev
