Members present: Noah, kocolosk, Jarda, jan____, wendall911, benoitc, drsm79, daleharvey_, rnewson, dch, chewbranca, Kxepal
---------------- Meeting summary: ---------------- 1. Preface a. last weeks** minutes here http://mail-archives.apache.org/mod_mbox/couchdb-dev/201301.mbox/%3ccapajbx5x6eo6tab91k+d6700sbn662a4t07a-utx0jkm8-h...@mail.gmail.com%3E (dch, 1) 2. 1.3.x blockers b. http://s.apache.org/couchdb_130_blockers is useful btw. Only 1654 remains. (dch, 2) 3. rcouch merge y. benoitc working on introducting rebar to couch, as part of rcouch merge (dch, 3) 4. fauxon, bring me all your UX goodness. >. ASF CLAs underway for fauxtoneers (dch, 4) >. jan to add fauxton component to JIRA (jan____, 4) 5. 1.3.x CORS 6. couchdbconf was awesome d. jan to mail about CouchDB Conf (jan____, 6) 7. benchmarking a. http://adamlofts.blogspot.co.at/2012/07/couchdb-benchmarking-i.html (dch, 7) j. https://github.com/chewbranca/all-the-numbers (dch, 7) r. dch to find some people with large docs/complex views for testing (dch, 7) >. kxepal to contribute real world docs and views for testing - huge thanks! (dch, 7) 8. packages a. wendall911 to come back to ML about EPEL packages (dch, 8) -------- Actions: -------- - benoitc working on introducting rebar to couch, as part of rcouch merge (dch, 20:36:09) - ASF CLAs underway for fauxtoneers (dch, 20:43:49) - jan to add fauxton component to JIRA (jan____, 20:51:04) - jan to mail about CouchDB Conf (jan____, 20:55:34) - dch to find some people with large docs/complex views for testing (dch, 21:04:23) - kxepal to contribute real world docs and views for testing - huge thanks! (dch, 21:18:26) - wendall911 to come back to ML about EPEL packages (dch, 21:21:22) IRC log follows: # 1. Preface # 20:02:17 [dch]: #info last weeks** minutes here http://mail-archives.apache.org/mod_mbox/couchdb-dev/201301.mbox/%3ccapajbx5x6eo6tab91k+d6700sbn662a4t07a-utx0jkm8-h...@mail.gmail.com%3E 20:03:39 [dch]: so topics our usual 1.3.x where are we at, I'm keen to hear what went on at couchhack after I had to return, items from last meeting etc. 20:03:54 [dch]: hey Adam welcome :-) 20:04:02 [kocolosk]: hiya 20:04:17 [Noah]: dch: we should also move around each member in the channel to ask if they have any business to discuss 20:04:24 [rnewson]: the view sig thing is a blocker. 20:04:26 [Noah]: dch: or some variation of that 20:04:32 [daleharvey_]: aloha 20:04:54 [dch]: ok, let's do that then 20:05:02 [dch]: Filipe welcome too :-) 20:05:02 [dch]: and dale 20:05:39 [drsm79]: hello meeting room 20:05:49 [dch]: welcome Simon, I have a face for the name now . 20:05:56 [drsm79]: :0 20:06:02 [drsm79]: :) 20:06:03 [drsm79]: a beard for the name, too 20:06:09 [dch]: OK, quick round check for items, then lets get cracking. 20:06:17 [dch]: fdmanana: Jarda JasonSmith any other topics? 20:06:39 [dch]: kocolosk: Noah rnewson ? Other than a fine meal chez Newson atm. 20:06:50 [dch]: benoitc: chewbranca daleharvey_ anything? 20:07:02 [Noah]: dch: i am thinking it might be easier to just go round each person - topics will come up naturally 20:07:02 [Jarda]: I'm actually intersted in the upcoming Futon and if it needs some extra hands? (If this is a suitable forum for that) 20:07:09 [Noah]: dch: collecting topics at the start just seems to slow us down 20:07:17 [benoitc]: sorry 20:07:17 [dch]: Noah: you need to decide if you're running the meeting or not. 20:07:18 [Noah]: dch sorry i will step back 20:07:26 [chewbranca]: Jarda: yeah definitely lots to be done 20:07:26 [dch]: :-) 20:07:32 [jan____]: Jarda: this is a great place 20:07:32 [Noah]: i am in a reduced capacity - but also caffine saturated ;) 20:07:32 [benoitc]: so yeah viez sig,, is there anyrthoing to do 20:07:40 [benoitc]: how do e bench the new view server 20:07:41 [Noah]: ACTION steps slowly away from the keyboard 20:07:42 [dch]: ok let's rool # 2. 1.3.x blockers # 20:08:10 [benoitc]: would like also to discuss a little about the rcouch merge if ize zill have some time 20:08:24 [dch]: #link http://s.apache.org/couchdb_130_blockers is useful btw. Only 1654 remains. 20:08:25 [dch]: benoitc: noted! 20:08:25 [chewbranca]: dch: Fauxton is coming along, I'm hoping to get some feedback on all-the-numbers and if there are any suggestions for better tests for views 20:08:47 [dch]: chewbranca: noted! 20:09:02 [rnewson]: I should add that davisp plans to do something on the view server but he's not here to elaborate, sadly. 20:09:24 [dch]: so the guts of this is that the new view server in master/1.3.x requires rebuilding your views as a result of changed file location and a different way of calculating the sig. 20:09:40 [rnewson]: and the contents of the view file have changed in incompatible ways. 20:09:50 [rnewson]: just changing the sig and location is insufficient. 20:09:54 [dch]: davisp has a suspicion he did that for a reason but nobody recalls the reason. It might just be an overdose of mustard-based bbq sauce though. 20:10:09 [dch]: right, just shuffling the view file from 1.2.x location into 1.3.x won't work either. 20:10:17 [rnewson]: correct. 20:10:25 [benoitc]: i remeber it was to previent similarities in view 20:10:32 [rnewson]: this is because the record name of the header has changed. 20:10:32 [benoitc]: s 20:10:39 [dch]: unless anybody has a miracle up their sleeve, rnewson and davisp have this in hand 20:11:25 [dch]: presumably this means we need to keep the old view code around in some form, so we can read the old views, until the next view compaction or regeneration? 20:11:39 [rnewson]: nah, it's easier than that. 20:11:39 [jan____]: I can help too but iâd need guideance. 20:11:40 [benoitc]: we need to make sure we wont't lost any new possible ffeature 20:11:41 [dch]: ACTION phew 20:11:47 [benoitc]: whatever the change we do 20:11:49 [rnewson]: the question is whether we migrate to the new scheme or preserve the old forever. 20:11:54 [dch]: sure 20:12:17 [rnewson]: I think we agree that the "Rebuild all views" approach is a non-starter. 20:12:24 [jan____]: +1 20:12:24 [benoitc]: or maybe we do like for dbs 20:12:26 [dch]: rnewson: i.e. if chewbranca's + others performance tests come back unfavourable, then revert to 1.2.x approach? 20:12:39 [benoitc]: ie zork zith old sche;e and on compaction go to the new one 20:12:56 [jan____]: benoitc: Iâd like that approach 20:13:02 [dch]: in the bigger picture, one of the reasons for this change was to make other indexing schemes more pluggable (e.g. vmx' R-tree, and anybody else game enouhg). 20:13:17 [dch]: benoitc: that seems sound. 20:13:18 [jan____]: bit I have not seriously looked at the code 20:13:24 [jan____]: *but 20:13:26 [benoitc]: it should be possible 20:13:39 [rnewson]: dch: imo, yes. 20:13:47 [chewbranca]: the very simple test I did comparing 1.2.x view engine vs 1.3.x showed a minimal improvement for 1.3.x, but the test was far from exhaustive 20:13:47 [dch]: is the view server easier to extend, as a result? That would be a big long-term win. 20:13:54 [benoitc]: imo the new view server design can introduce some latency but not so much 20:14:03 [benoitc]: i noticed differences recently 20:14:18 [benoitc]: so that may be just a commit we did recently 20:14:25 [benoitc]: but we need benchs anyway 20:14:33 [jan____]: lets focus this discussion a little 20:14:34 [rnewson]: I wouldn't expect a performance difference btw, it just seems to be an abstracting out of the common pattern. 20:14:40 [jan____]: can we move performance questions to the back? 20:14:57 [chewbranca]: benoitc: if you notice anywhere in particular where 1.3.x views are slower, please give me a test case and sample data set and I'll add it to the benchmarks 20:15:18 [dch]: sure, let's keep on this for a few more minutes. jan____ anything specific in mind? 20:16:03 [jan____]: rnewson: you have looked at the code, do you think it is feasible to add some code that can treat a 1.2.x view index as is, and even write to it in the 1.2.x format, and only upgrade to the new format on view compaction? 20:16:18 [jan____]: dch: I donât want to cut off anything, but Iâd like to not talk about 10 things at once 20:16:33 [benoitc]: chewbranca: well if you have already bench tools then share them so e can have figures using the same process 20:16:34 [jan____]: we have a) file incompat, update badness b) various ways to handle a) and c) performance stuff 20:17:10 [jan____]: c) might lead to a) being unnecessary for now (because weâd revert) but frankly I am not seeing that, and we need to address view index upgrades regardless) 20:17:20 [benoitc]: chewbranca: i actually can't share the data so i need to think a little on how to test it accurately 20:18:03 [chewbranca]: benoitc: https://github.com/chewbranca/all-the-numbers 20:18:05 [benoitc]: will do that but not before next week 20:18:13 [benoitc]: thanks. 20:19:19 [jan____]: I understand rnewson is out for dinner, davisp isnât here. is there anyone else who can comment on whether the proposed strategy would work? 20:19:25 [dch]: so (a) is all about the view signatures themselves, as a result of the changed data format? I'm not clear on the issue, but it sounds like rnewson is saying the new sig is a requirement of the new view engine, we can't keep the old sig? 20:19:55 [benoitc]: dch: the thing is that you have a big index you don't want to rebuild it online 20:20:13 [jan____]: dch: it is the filename and the file format 20:20:33 [dch]: benoitc: clear, you'd want to do the COPY shuffle for example, or plan your compaction. 20:20:41 [benoitc]: dch: yup 20:20:48 [jan____]: that leads to 1.3.x (as I understand) invalidate any previous view indexes which leads to what benoitc just said 20:20:55 [wendall911]: jan____: not having support for both 1.2.x and 1.3.x view formats would mean downtime. Huge downtime for some cases. 20:21:03 [benoitc]: for old sig vs new sig, i can see that the new signature make them more unique 20:21:10 [dch]: yes, and there's no way to avoid it either. 20:21:10 [jan____]: wendall911: yes, we want to avoid that 20:21:28 [jan____]: we can defer it to view-compaction though 20:21:39 [kocolosk]: I'd have to think we can add some upgrade clauses to read the old signature format internally (though i've not looked closely at it yet) 20:21:39 [benoitc]: i think we need for that some feedback from davisp 20:21:54 [jan____]: wel *can* modulo code that the db file handling already has (not that we can copy anything I think) 20:22:02 [kocolosk]: the change in location is a bit more annoying but should be manageable, right? 20:22:02 [jan____]: *well 20:22:17 [benoitc]: kocolosk: yes should be possible, need to check if it works well with the server that maintain states 20:22:24 [kocolosk]: right 20:22:39 [benoitc]: kocolosk: the location change is easier to solve imo 20:22:47 [jan____]: kocolosk: wanna take a stab at it? 20:22:54 [kocolosk]: try new location, if missing try old one? 20:23:33 [kocolosk]: jan____: if i can't scare up davisp, yeah 20:24:03 [jan____]: kocolosk: woot, I am up for testing this tomorrow, if you can tag-team this over night 20:24:10 [benoitc]: kocolosk: yes that can work 20:24:25 [kocolosk]: jan____: kk 20:24:56 [dch]: is this then per-db & view, or per view? and how will we know that all the views are completely migrated/updated, and we can transfer the old ones to .delete for example? 20:25:12 [dch]: ACTION thinking out loud, not expecting an answer. 20:25:33 [benoitc]: chewbranca: i think this test lacks of some datasets types (size of docs and real latency) i will try it anyway 20:26:10 [benoitc]: dch: can be done when accessing the view group maybe 20:26:27 [chewbranca]: benoitc: definitely does, just put it together over the weekend, lets chat later in this meeting about what can be done to make it comprehensive 20:26:48 [dch]: ok I think we are ready to move to next topic, unless people have more to add? 20:27:06 [dch]: anything else people want to add for 1.3.x issues? 20:27:11 [benoitc]: chewbranca: fine for me 20:27:19 [benoitc]: let's try tonmorrow though 20:27:27 [benoitc]: i'm a bit toired tonight 20:27:48 [benoitc]: dch: the doc link maybe 20:27:55 [benoitc]: in fiuton 20:28:04 [jan____]: one more point for the release team: this might push us over the desired Feb 1st call-a-vote date. 20:28:05 [dch]: I have one, I added a jira a while back to do sanity checking that (a) we can read/write to key locations like logfiles, view and dbs, and if not, fail with a graceful(ish) erlang message. I will take a whack at this tomorrow, it will not be a lot of code 20:28:10 [benoitc]: maybe if docs aren't build we can provide direct link to readthe doc 20:28:33 [dch]: benoitc: I am +1 on a direct link to docs.couchdb.org if the docs aren't built. 20:28:48 [jan____]: I liked the idae of doing the fake index.html with the meta refresh 20:28:48 [dch]: I haven't looked at Kxepal's patch yet. 20:29:06 [jan____]: neither, dch wanna do that tomorrow? 20:29:19 [dch]: jan____: read the patch? for sure. 20:29:39 [benoitc]: dch: mm i would just display it, misued the word direct 20:29:42 [benoitc]: misused 20:29:49 [benoitc]: ACTION is training a new kbd 20:30:32 [dch]: ok, next up isâ¦. # 3. rcouch merge # 20:30:40 [dch]: benoitc: all yours 20:30:54 [rnewson]: jan____: what kocolosk said. 20:31:04 [jan____]: rnewson: thx 20:31:47 [benoitc]: ok 20:31:54 [benoitc]: so i am thinking to start it next week 20:32:09 [benoitc]: by first introducing rebar and a way to build without autoconf 20:32:24 [benoitc]: i won't try to fight with autotools for now 20:32:32 [benoitc]: except if one think it's desired 20:33:02 [dch]: benoitc: good call. and then work on the NIFs? I should be able to help with that. 20:33:09 [benoitc]: yes 20:33:18 [benoitc]: i can say i have already all the nifs in rcouch 20:33:32 [benoitc]: couch_collate for ex 20:33:50 [dch]: yup, that should be pretty good. I think I forgot one piece in jiffy for windows, but I assume we are sticking with ejson for the moment? 20:33:54 [benoitc]: wich replace couch_drv & couch_ejson_compare 20:34:09 [benoitc]: un didn't thin k baout that 20:34:24 [benoitc]: yes so i will put this change in another commit 20:34:32 [jan____]: dch: yeah, lets not make it more comlpicated than needed 20:34:34 [jan____]: we can jiffy things in at any time later 20:34:47 [benoitc]: going to jiffy is a 5 line patch anyway 20:35:09 [jan____]: yes, but it adds complexity for all the systems we support 20:35:17 [benoitc]: i agree 20:35:24 [jan____]: cool 20:35:47 [dch]: alright! sounds great, so what was actiomnable there? 20:35:54 [benoitc]: i am not sure how to handle the spidermonkey case for now 20:36:09 [dch]: #action benoitc working on introducting rebar to couch, as part of rcouch merge 20:36:32 [dch]: I think for spidermonkey we should seriously consider consolidating on 1.8.5 only. 20:36:32 [benoitc]: yes let put an action, what version should be supported 20:36:39 [benoitc]: presumably we donyt replace everything for now 20:36:54 [benoitc]: yes i'm +1 on it 20:37:02 [dch]: for 1.3.x, no, this would come 1.4 or later not sure how that is covered in semver. 20:37:04 [jan____]: for master? +1 20:37:17 [benoitc]: yes for master 20:37:32 [dch]: it's very S&M tonight 20:37:41 [dch]: benoitc: anything else, or shall we move topic? 20:37:56 [benoitc]: i think i have some answer 20:38:09 [benoitc]: will make spome update on the ml next week 20:38:17 [jan____]: cool 20:38:19 [benoitc]: ACTION is disappearing for the fosdem 20:38:24 [dch]: looking forwards to it! 20:38:32 [dch]: have fun benoitc. # 4. fauxon, bring me all your UX goodness. # 20:38:49 [dch]: drsm79: how does your 3G go? 20:39:04 [dch]: or chewbranca maybe it was your topic (sorry!) 20:39:32 [chewbranca]: fauxton is coming along well, plugin system is solid and we've been getting some help from Garren and a few others 20:40:09 [chewbranca]: we're in the process of getting CLAs taken care of so that we can move the code into the primary repo and use standard ticketing tools and what not 20:40:11 [dch]: great! are we ready to set up an integration branch for this or are you still githubbing? 20:40:24 [jan____]: Simon got @gr2m interested in hacking too 20:40:33 [jan____]: I want to move the code to asf asap 20:40:39 [dch]: chewbranca: right. let me know if you need a hand with this, the actual process is easy. 20:40:47 [jan____]: Iâll handle the apperwork, I hope I get this done tomorrow 20:40:47 [drsm79]: dch what chewbranca said + more hands welcome 20:40:47 [jan____]: *paper 20:40:54 [drsm79]: dch: it's on a branch in a fork of couch 20:40:54 [drsm79]: we can switch it over whenever I think 20:41:02 [jan____]: then I want to kick off votes for all fauxton committers 20:41:04 [dch]: fun-tastic! 20:41:10 [chewbranca]: plenty left to do, will have a view editor working this week that would be good to get some feedback 20:41:32 [chewbranca]: yeah jan____ is helping getting the paperwork taken care of, I'll get my CLA signed and scanned and sent off to jan____ today 20:41:40 [drsm79]: I'll do mine tomorrow 20:41:47 [jan____]: also, @gr2m is a little frustrated with fauxton development. I asked him what heâd like it to look at and he wrote us up a fake readme: http://minutes.jit.su 20:41:47 [jan____]: oops 20:41:56 [jan____]: that fake readme is actually in https://gist.github.com/4672813 20:42:24 [drsm79]: 3g is failing me 20:42:47 [drsm79]: will read it 20:42:54 [chewbranca]: jan____: sure, things could be easier, and we just need to throw deps into a package.json and then you can just npm install 20:42:54 [Jarda]: jan____: that looks awesome to me 20:43:09 [jan____]: Jarda: yeah, sadly it isnât that for now 20:43:33 [chewbranca]: I'll work on simplifying the setup process a bit this week 20:43:40 [jan____]: but I think we can approximate things 20:43:47 [benoitc]: mmm btw does npp or other can keep deps offline ? 20:43:49 [dch]: #action ASF CLAs underway for fauxtoneers 20:43:54 [jan____]: I wonder how folks feel about a separate repo for fauxton? 20:43:55 [benoitc]: for a n archive release i mean 20:43:56 [dch]: benoitc: yes. 20:44:09 [daleharvey_]: I think a seperate repo is a good idea 20:44:12 [dch]: benoitc: yes you can in principle commit them. 20:44:17 [benoitc]: also everything can be solved later during full integraton 20:44:17 [Jarda]: jan____: so what is the tech stack for current fauxton? 20:44:24 [jan____]: chewbranca: awesome, Iâm happy to help test and nudge towards MOAR EASY so we can get the gr2m types aboard. 20:44:24 [benoitc]: dch: mm not commit 20:44:39 [benoitc]: or maybe 20:44:54 [jan____]: Jarda: chewbranca knows ;) 20:45:04 [chewbranca]: Jarda: its a backbone app with grunt for building/tasks and a handful of other tools like backbone layoutmanager 20:45:09 [Jarda]: I have a solid background in client-side javascript (backbone/requirejs/etc) 20:45:11 [benoitc]: we need to integrate sonme depenqncies on release but don't really need all in the repo 20:45:24 [Jarda]: We also use yeoman in our current project at work 20:45:39 [chewbranca]: current code is at: https://github.com/cloudant-labs/couchdb/tree/fauxton/src/fauxton 20:45:54 [Jarda]: I might be able to help if there are clear issues to tackle 20:45:57 [jan____]: Jarda: perfect :) 20:46:02 [dch]: Jarda: feel free to pitch in, we'd love to have some helping hands! 20:46:39 [drsm79]: jan____: I'd be happy to have it in it's own repo etc 20:46:40 [drsm79]: but don't know all the nitty gritty around what ASF require 20:46:54 [jan____]: just admin stuff 20:46:57 [jan____]: we can have more repos 20:47:02 [drsm79]: moar 20:47:09 [dch]: ok, moar fauxton or next topic? 20:47:10 [jan____]: the only thing we need to solve then is how to get it into `make dist` 20:47:17 [drsm79]: do issues have to go in jira or can we do it all in github? 20:47:17 [jan____]: moarepos 20:47:17 [dch]: chewbranca: drsm79: great update. 20:47:32 [chewbranca]: would be good to decide on a place to handle tickets 20:47:54 [benoitc]: jira once the integration is done i think 20:47:57 [drsm79]: dch: not bad considering being on a train ;) 20:48:04 [jan____]: drsm79: once code is in ASF, issue go into jira, PRs can go into github 20:48:09 [chewbranca]: grunt is similar to yeoman, we can get down to a npm install grunt gogo-gadget-fauxton and have it all running, but would be good to have a place where we can start recording issues 20:48:18 [chewbranca]: dch: thanks 20:48:18 [jan____]: Id suggest abusing PRs for tickets 20:48:32 [chewbranca]: jan____: that's what we've been doing 20:48:32 [jan____]: tickets proper are ASF JIRA for now 20:48:47 [jan____]: good 20:48:49 [jan____]: keep it up :) 20:49:02 [drsm79]: jan____: sounds good 20:49:11 [jan____]: new PRs would go against github.com/apache/couchdb-fauxton or whatever 20:49:24 [Jarda]: ok so is there a roadmap for fauxton somewhere? 20:49:39 [chewbranca]: should we temporarily have issues at the current fauxton repo or in the futon organization? 20:49:56 [jan____]: chewbranca: just to migrate things a week later? 20:50:02 [chewbranca]: we're starting to get enough people involved that we need somewhere for people to file issues 20:50:09 [jan____]: chewbranca: we can start recording issues in ASF jira right away 20:50:11 [chewbranca]: jan____: good point, alright scratch that 20:50:24 [benoitc]: ACTION dch ping here 20:50:24 [benoitc]: err 20:50:32 [Jarda]: if one would like to help, is the proper way to just make it running and hack around? 20:50:39 [benoitc]: z\/win 59 20:50:40 [chewbranca]: jan____: sounds good 20:50:54 [dch]: next topic? 20:50:55 [jan____]: chewbranca: aye 20:50:57 [chewbranca]: Jarda: yeah, shoot me a ping after this and I can help you get up and running 20:51:02 [Jarda]: chewbranca: cool 20:51:04 [jan____]: #action jan to add fauxton component to JIRA 20:51:09 [jan____]: +1 EOD 20:51:32 [dch]: ok I wanted to circle back to 1.3.x / CORS for a moment.. # 5. 1.3.x CORS # 20:52:02 [dch]: basically I did some testing, and I wondered if we collectively feel ok or if we should be doing some more here? 20:52:32 [benoitc]: ah more like what? 20:52:39 [jan____]: dch: we ship as experimental 20:52:40 [dch]: I got stuck reading some of the spec and translating that into "testable" stuff. 20:52:56 [jan____]: dch: I donât want to reopen this now. we donât get to ship 1.3.x ever 20:52:56 [dch]: pre-flight for example, I didn't follow that sufficiently yet. 20:53:09 [dch]: jan____: ok for me then. 20:53:11 [benoitc]: we are ok on that imo 20:53:32 [benoitc]: also i would say that having pouchdb running with it is enoiugh for me :) 20:53:32 [dch]: woot, I will see if I can put together an example or two for people to reference. 20:53:34 [daleharvey_]: trying to find my old comments, but I remember there being a few minor feature requests, but mostly good, ship as is experimental and then to be honest there was only a few things more needed 20:53:47 [jan____]: yeah, lets have it hit eager users and let them find all the issues :) 20:53:54 [dch]: yay users! 20:53:56 [jan____]: daleharvey_: I think I got through mot of your stuff eventually 20:54:03 [dch]: ok, next topic, any requests? 20:54:24 [chewbranca]: benchmarking? 20:54:34 [jan____]: I can give an update about CouchDB Conf 20:55:04 [dch]: um how about we roll with jan____ then? 20:55:04 [benoitc]: chewbranca: i need to go , as qction i would say let's try together to build some datasets 20:55:09 [Noah]: jan____: to the ML preferably 20:55:09 [benoitc]: so we can test it 20:55:09 [chewbranca]: yeah would be great to hear about CouchDB conf 20:55:18 [drsm79]: jan____: we should start planning the next one ~now 20:55:18 [chewbranca]: benoitc: sounds good 20:55:18 [drsm79]: keep the momentum # 6. couchdbconf was awesome # 20:55:26 [jan____]: Noah: aye 20:55:26 [Noah]: cool 20:55:26 [jan____]: ok 20:55:34 [jan____]: #action jan to mail about CouchDB Conf 20:55:42 [jan____]: drsm79: +1 20:55:49 [chewbranca]: I'm definitely interested in getting a CouchDB conference going out in Seattle 20:55:58 [drsm79]: chewbranca in ~4 months time? 20:56:06 [chewbranca]: yeap 20:56:12 [drsm79]: gtg train has got home 20:56:19 [jan____]: drsm79: safe home! 20:56:27 [chewbranca]: drsm79: later 20:56:27 [jan____]: chewbranca: yeah, awesome 20:56:52 [jan____]: chewbranca: for me around JSConf US or NodeConf 20:57:04 [jan____]: otherwise I can skype in 20:57:25 [chewbranca]: jan____: sounds good 20:58:09 [dch]: ok we have a choice, run over for 5-10 minutes and talk about benchmarking or do a sprint for the finish. 20:58:47 [jan____]: Iâm all actioned out, but you go ahead 20:58:47 [dch]: chewbranca: any actions related to seattlecouch? # 7. benchmarking # 20:59:32 [dch]: #link http://adamlofts.blogspot.co.at/2012/07/couchdb-benchmarking-i.html 20:59:32 [chewbranca]: dch: if it sounds good to everyone I'll start looking into it 20:59:56 [chewbranca]: nice, good link 21:00:17 [jan____]: nice link! 21:00:24 [jan____]: yeah! :) 21:00:24 [dch]: chewbranca: I would love to do some more work with what Adam put together in Tsung, its on my list for a while. But I need to keep focused on the other stuff first. 21:01:17 [dch]: Also I use this a lot https://github.com/fdmanana/seatoncouch its ruby, set it up with a while true; do ⦠and run multiple instances pointing to different DBs. This is a nice testing approach. 21:01:17 [chewbranca]: so my idea for all-the-numbers is to get a simple view benchmark in place that will easily compare vanilla spidermonkey view engine, with Jason's node.js engine, and then also compare a native erlang view engine, to get some ideas on overall performance and cost paid for serialization 21:01:32 [chewbranca]: I've got this going: https://github.com/chewbranca/all-the-numbers 21:01:39 [dch]: #link https://github.com/chewbranca/all-the-numbers 21:01:54 [chewbranca]: which is not Tsung by any means, but works ok and saves the results back into couchdb for easy comparisons 21:02:24 [chewbranca]: not opposed to using something else, but I want to understand the performance characteristics of the different options, and I think it would be good to get that information out 21:02:40 [dch]: chewbranca: great, the view engine changes are a good place to start. 21:03:00 [chewbranca]: dch: yeah I sent out an email to dev@ comparing 1.2.x with 1.3.x 21:03:01 [dch]: there's also a python view server, I think kxepal wrote it ? 21:03:38 [chewbranca]: I can take care of adding a few of the different engines, but what I could use help with is determining the appropriate sample data sets and proper views to comprehensively compare the engines 21:04:02 [dch]: chewbranca: tricky to get real-world stuff. Um I know a few people with large docs and complex views, maybe they can share stuff. 21:04:23 [dch]: #action dch to find some people with large docs/complex views for testing 21:04:45 [chewbranca]: dch: agreed, if there are some known edge cases it would be good to get those in 21:05:08 [chewbranca]: I want to have a range of doc sizes and what not so you can compare serialization cost as a function of doc size and number of views per ddoc and what not 21:05:23 [Kxepal]: dch: you want to benchmark it too? 21:05:23 [dch]: chewbranca: a point made a while back is that testing should ideally be done with concurrent runs. That doesn't need to be phase 1 but its good to add it in future. 21:05:41 [dch]: Kxepal: more to the point, are you interested in helping out and giving it a test too ? 21:05:54 [dch]: you know how it fits together so you'd be a great help :) 21:06:08 [chewbranca]: dch: yeah I manually did 3 runs for the tests, but I'm planning on adding automatic iterations 21:07:10 [wendall911]: it isn't always ideal to run concurrently, especially building large indexes. You may just get to the point where you're actually just benchmarking the hardware, not the software. 21:07:39 [Kxepal]: dch: no problem. however, for python there is some set of cases when it will be faster due to rich stdlib and caching imports. but mostly it's slower on long running tasks unless it's not a pypy 21:08:18 [chewbranca]: what other concurrent requests would be good to do? basically right now I'm just throwing a bunch of docs at the db and then grabbing the view with limit=1 21:08:25 [dch]: chewbranca: I'd like to run this on windows as well, um maybe I send you a patch to extract the OS etc for the reports. 21:08:25 [chewbranca]: not testing incremental updates or anything else right now 21:08:33 [chewbranca]: dch: sounds good! 21:08:49 [dch]: compaction 21:09:33 [chewbranca]: compaction as in bench marking compaction as well, or running compaction while bench marking the view? 21:09:54 [wendall911]: Is it possible for the test runner to use tmpfs for the databases? I fear that you'd be just doing IO testing on the hardware for some of this. 21:09:56 [dch]: e.g. run tests, compact, re-run tests. 21:09:57 [chewbranca]: maybe action item for this is start a thread on the ML about proper benchmarking scenarios and start adding those in 21:10:17 [dch]: chewbranca: yes, or adding them into a jira ticket list. 21:10:30 [chewbranca]: dch: ahhh, yeah I was just loading the db, deleting, reloading the db 21:10:32 [dch]: wendall911: I think we probably want both. 21:10:53 [wendall911]: dch: I don't see that figuring out your hardware is slow is really a useful benchmark 21:11:00 [dch]: I definitely want to see difference between SSD and cloudy, and real disk. 21:11:08 [wendall911]: hmm, ok 21:11:31 [dch]: wendall911: because for example the serialisation cost is high, possibly comparable in some circumstances to disk IO slowness. 21:11:44 [dch]: rampant speculation that. 21:11:59 [wendall911]: so long as the benchmarks are run on hardware that the test is the only thing running 21:12:01 [dch]: but I've seen 650K individual docs going to the JS view server. 21:12:07 [chewbranca]: understanding the actual serialization costs was one of my big motivations here 21:12:07 [dch]: wendall911: yup. 21:12:15 [dch]: chewbranca: I have some ideas on testing serialisation costs. 21:12:45 [chewbranca]: dch: cool, definitely want to pick your brain on it 21:12:50 [dch]: I can instrument erlang and node directly (ignoring sandboxes) and we should be able to get fine-grained results. 21:12:57 [chewbranca]: nice 21:13:12 [dch]: but thats a wee bit away. I have the basic pieces working but not massaging the data. 21:13:19 [wendall911]: and we should only publish the tmpfs without fsync, so we can be like all the other kids 21:13:27 [dch]: wendall911: noatime FTW. 21:13:34 [wendall911]: :) 21:13:57 [dch]: chewbranca: if you're feeling frisky we could add with / without snappy compression. 21:14:12 [chewbranca]: dch: yeah the more tests we can do the merrier 21:14:19 [dch]: there must be a generic "run tests with various combinations" nodejs library out there. 21:14:49 [chewbranca]: yeah worth looking into 21:15:12 [dch]: so anything more? I'm getting calls to come to bed. 21:15:14 [dch]: (sorry for the abrupt topic change) 21:15:20 [chewbranca]: nope, glad to get the discussion going 21:15:50 [Kxepal]: chewbranca: if you'd like I could share db for 3M of production-world documents for testing - around 6GB of data and they are complex to create interesting views that really maps and reduces. 21:15:57 [dch]: yes, I am totally enthused. 21:16:32 [dch]: Kxepal: that would be *awesome*. I am happy to sanitise the content, and then we could potentially keep it as reference, if you agree. 21:16:39 [dch]: and maybe we could rewrite some of the views in erlang. 21:16:47 [dch]: for comparison. 21:16:55 [wendall911]: dch: is there a place to have a hosted copy? Would be nice to be able to run tests without distributing the data per-se, just sync with hosted copy. 21:17:02 [chewbranca]: Kxepal: yeah that would be great, especially if we can throw that up on a CouchDB instance somewhere so people can replicate it in for testing 21:17:24 [Kxepal]: dch: simple erlang view that build index for single field runs half hour for my server (: never tried python nor js for this db 21:17:25 [dch]: wendall911: I am pretty sure between cloudant and iriscouch we will find a way. Or use my mac as an endpoint. 21:17:39 [wendall911]: haha 21:18:04 [Kxepal]: chewbranca: ok, I'll prepare db on this week. ping me if I didn't you 21:18:09 [dch]: wendall911: what do you mean by hosted copy? a way to retrieve the doc set for testing? 21:18:17 [wendall911]: dch: yes 21:18:24 [chewbranca]: yeah worst case scenario I've got a dedicated box with couchdb running and plenty of bandwidth 21:18:26 [dch]: #action kxepal to contribute real world docs and views for testing - huge thanks! 21:18:46 [dch]: chewbranca: like a spare 80 node cluster I'll bet. 21:18:53 [dch]: ok any more topics? 21:19:00 [dch]: we covered a lot 21:19:01 [chewbranca]: lol 21:19:23 [wendall911]: On a side note, I may be able to step into a co-maintainer role on the epel packages (for centos, rhel, fedora) 21:19:31 [wendall911]: working on it 21:20:08 [wendall911]: But not really project related, just frustrated with the slowness of releases downstream, so proactively trying to fix. 21:20:18 [dch]: wendall911: that would be fantastic, we are working on build bots (via jenkins etc) so you might be interested in that too. 21:20:44 [wendall911]: ahh, would be nice. I think the show stopper for me is js 1.7 on centos, it just sucks # 8. packages # 21:21:22 [dch]: #action wendall911 to come back to ML about EPEL packages 21:21:30 [dch]: wendall911: hopefully that's generic enough to remind you. 21:21:58 [wendall911]: dch: thanks, I would have done more in this area, just work is overly busy atm :( 21:22:12 [dch]: wendall911: damn that work. 21:22:42 [dch]: imma closing the meeting unless we have more?
