I couldn't attend the meeting. But still having such a summary is awesome :)
Thanks! Klaus On Wed, 2013-01-30 at 21:39 +0000, ASF IRC Services wrote: > Members present: Noah, kocolosk, Jarda, jan____, wendall911, benoitc, drsm79, > daleharvey_, rnewson, dch, chewbranca, Kxepal > > ---------------- > Meeting summary: > ---------------- > > 1. Preface > a. last weeks** minutes here > http://mail-archives.apache.org/mod_mbox/couchdb-dev/201301.mbox/%3ccapajbx5x6eo6tab91k+d6700sbn662a4t07a-utx0jkm8-h...@mail.gmail.com%3E > (dch, 1) > > 2. 1.3.x blockers > b. http://s.apache.org/couchdb_130_blockers is useful btw. Only 1654 > remains. (dch, 2) > > 3. rcouch merge > y. benoitc working on introducting rebar to couch, as part of rcouch merge > (dch, 3) > > 4. fauxon, bring me all your UX goodness. > >. ASF CLAs underway for fauxtoneers (dch, 4) > >. jan to add fauxton component to JIRA (jan____, 4) > > 5. 1.3.x CORS > > 6. couchdbconf was awesome > d. jan to mail about CouchDB Conf (jan____, 6) > > 7. benchmarking > a. http://adamlofts.blogspot.co.at/2012/07/couchdb-benchmarking-i.html > (dch, 7) > j. https://github.com/chewbranca/all-the-numbers (dch, 7) > r. dch to find some people with large docs/complex views for testing (dch, > 7) > >. kxepal to contribute real world docs and views for testing - huge > thanks! (dch, 7) > > 8. packages > a. wendall911 to come back to ML about EPEL packages (dch, 8) > > > -------- > Actions: > -------- > - benoitc working on introducting rebar to couch, as part of rcouch merge > (dch, 20:36:09) > - ASF CLAs underway for fauxtoneers (dch, 20:43:49) > - jan to add fauxton component to JIRA (jan____, 20:51:04) > - jan to mail about CouchDB Conf (jan____, 20:55:34) > - dch to find some people with large docs/complex views for testing (dch, > 21:04:23) > - kxepal to contribute real world docs and views for testing - huge thanks! > (dch, 21:18:26) > - wendall911 to come back to ML about EPEL packages (dch, 21:21:22) > > IRC log follows: > > > # 1. Preface # > 20:02:17 [dch]: #info last weeks** minutes here > http://mail-archives.apache.org/mod_mbox/couchdb-dev/201301.mbox/%3ccapajbx5x6eo6tab91k+d6700sbn662a4t07a-utx0jkm8-h...@mail.gmail.com%3E > 20:03:39 [dch]: so topics our usual 1.3.x where are we at, I'm keen to hear > what went on at couchhack after I had to return, items from last meeting etc. > 20:03:54 [dch]: hey Adam welcome :-) > 20:04:02 [kocolosk]: hiya > 20:04:17 [Noah]: dch: we should also move around each member in the channel > to ask if they have any business to discuss > 20:04:24 [rnewson]: the view sig thing is a blocker. > 20:04:26 [Noah]: dch: or some variation of that > 20:04:32 [daleharvey_]: aloha > 20:04:54 [dch]: ok, let's do that then > 20:05:02 [dch]: Filipe welcome too :-) > 20:05:02 [dch]: and dale > 20:05:39 [drsm79]: hello meeting room > 20:05:49 [dch]: welcome Simon, I have a face for the name now . > 20:05:56 [drsm79]: :0 > 20:06:02 [drsm79]: :) > 20:06:03 [drsm79]: a beard for the name, too > 20:06:09 [dch]: OK, quick round check for items, then lets get cracking. > 20:06:17 [dch]: fdmanana: Jarda JasonSmith any other topics? > 20:06:39 [dch]: kocolosk: Noah rnewson ? Other than a fine meal chez Newson > atm. > 20:06:50 [dch]: benoitc: chewbranca daleharvey_ anything? > 20:07:02 [Noah]: dch: i am thinking it might be easier to just go round each > person - topics will come up naturally > 20:07:02 [Jarda]: I'm actually intersted in the upcoming Futon and if it > needs some extra hands? (If this is a suitable forum for that) > 20:07:09 [Noah]: dch: collecting topics at the start just seems to slow us > down > 20:07:17 [benoitc]: sorry > 20:07:17 [dch]: Noah: you need to decide if you're running the meeting or not. > 20:07:18 [Noah]: dch sorry i will step back > 20:07:26 [chewbranca]: Jarda: yeah definitely lots to be done > 20:07:26 [dch]: :-) > 20:07:32 [jan____]: Jarda: this is a great place > 20:07:32 [Noah]: i am in a reduced capacity - but also caffine saturated ;) > 20:07:32 [benoitc]: so yeah viez sig,, is there anyrthoing to do > 20:07:40 [benoitc]: how do e bench the new view server > 20:07:41 [Noah]: ACTION steps slowly away from the keyboard > 20:07:42 [dch]: ok let's rool > > > # 2. 1.3.x blockers # > 20:08:10 [benoitc]: would like also to discuss a little about the rcouch > merge if ize zill have some time > 20:08:24 [dch]: #link http://s.apache.org/couchdb_130_blockers is useful btw. > Only 1654 remains. > 20:08:25 [dch]: benoitc: noted! > 20:08:25 [chewbranca]: dch: Fauxton is coming along, I'm hoping to get some > feedback on all-the-numbers and if there are any suggestions for better tests > for views > 20:08:47 [dch]: chewbranca: noted! > 20:09:02 [rnewson]: I should add that davisp plans to do something on the > view server but he's not here to elaborate, sadly. > 20:09:24 [dch]: so the guts of this is that the new view server in > master/1.3.x requires rebuilding your views as a result of changed file > location and a different way of calculating the sig. > 20:09:40 [rnewson]: and the contents of the view file have changed in > incompatible ways. > 20:09:50 [rnewson]: just changing the sig and location is insufficient. > 20:09:54 [dch]: davisp has a suspicion he did that for a reason but nobody > recalls the reason. It might just be an overdose of mustard-based bbq sauce > though. > 20:10:09 [dch]: right, just shuffling the view file from 1.2.x location into > 1.3.x won't work either. > 20:10:17 [rnewson]: correct. > 20:10:25 [benoitc]: i remeber it was to previent similarities in view > 20:10:32 [rnewson]: this is because the record name of the header has changed. > 20:10:32 [benoitc]: s > 20:10:39 [dch]: unless anybody has a miracle up their sleeve, rnewson and > davisp have this in hand > 20:11:25 [dch]: presumably this means we need to keep the old view code > around in some form, so we can read the old views, until the next view > compaction or regeneration? > 20:11:39 [rnewson]: nah, it's easier than that. > 20:11:39 [jan____]: I can help too but id need guideance. > 20:11:40 [benoitc]: we need to make sure we wont't lost any new possible > ffeature > 20:11:41 [dch]: ACTION phew > 20:11:47 [benoitc]: whatever the change we do > 20:11:49 [rnewson]: the question is whether we migrate to the new scheme or > preserve the old forever. > 20:11:54 [dch]: sure > 20:12:17 [rnewson]: I think we agree that the "Rebuild all views" approach is > a non-starter. > 20:12:24 [jan____]: +1 > 20:12:24 [benoitc]: or maybe we do like for dbs > 20:12:26 [dch]: rnewson: i.e. if chewbranca's + others performance tests come > back unfavourable, then revert to 1.2.x approach? > 20:12:39 [benoitc]: ie zork zith old sche;e and on compaction go to the new > one > 20:12:56 [jan____]: benoitc: Id like that approach > 20:13:02 [dch]: in the bigger picture, one of the reasons for this change was > to make other indexing schemes more pluggable (e.g. vmx' R-tree, and anybody > else game enouhg). > 20:13:17 [dch]: benoitc: that seems sound. > 20:13:18 [jan____]: bit I have not seriously looked at the code > 20:13:24 [jan____]: *but > 20:13:26 [benoitc]: it should be possible > 20:13:39 [rnewson]: dch: imo, yes. > 20:13:47 [chewbranca]: the very simple test I did comparing 1.2.x view engine > vs 1.3.x showed a minimal improvement for 1.3.x, but the test was far from > exhaustive > 20:13:47 [dch]: is the view server easier to extend, as a result? That would > be a big long-term win. > 20:13:54 [benoitc]: imo the new view server design can introduce some latency > but not so much > 20:14:03 [benoitc]: i noticed differences recently > 20:14:18 [benoitc]: so that may be just a commit we did recently > 20:14:25 [benoitc]: but we need benchs anyway > 20:14:33 [jan____]: lets focus this discussion a little > 20:14:34 [rnewson]: I wouldn't expect a performance difference btw, it just > seems to be an abstracting out of the common pattern. > 20:14:40 [jan____]: can we move performance questions to the back? > 20:14:57 [chewbranca]: benoitc: if you notice anywhere in particular where > 1.3.x views are slower, please give me a test case and sample data set and > I'll add it to the benchmarks > 20:15:18 [dch]: sure, let's keep on this for a few more minutes. jan____ > anything specific in mind? > 20:16:03 [jan____]: rnewson: you have looked at the code, do you think it is > feasible to add some code that can treat a 1.2.x view index as is, and even > write to it in the 1.2.x format, and only upgrade to the new format on view > compaction? > 20:16:18 [jan____]: dch: I dont want to cut off anything, but Id like to not > talk about 10 things at once > 20:16:33 [benoitc]: chewbranca: well if you have already bench tools then > share them so e can have figures using the same process > 20:16:34 [jan____]: we have a) file incompat, update badness b) various ways > to handle a) and c) performance stuff > 20:17:10 [jan____]: c) might lead to a) being unnecessary for now (because > wed revert) but frankly I am not seeing that, and we need to address view > index upgrades regardless) > 20:17:20 [benoitc]: chewbranca: i actually can't share the data so i need to > think a little on how to test it accurately > 20:18:03 [chewbranca]: benoitc: https://github.com/chewbranca/all-the-numbers > 20:18:05 [benoitc]: will do that but not before next week > 20:18:13 [benoitc]: thanks. > 20:19:19 [jan____]: I understand rnewson is out for dinner, davisp isnt here. > is there anyone else who can comment on whether the proposed strategy would > work? > 20:19:25 [dch]: so (a) is all about the view signatures themselves, as a > result of the changed data format? I'm not clear on the issue, but it sounds > like rnewson is saying the new sig is a requirement of the new view engine, > we can't keep the old sig? > 20:19:55 [benoitc]: dch: the thing is that you have a big index you don't > want to rebuild it online > 20:20:13 [jan____]: dch: it is the filename and the file format > 20:20:33 [dch]: benoitc: clear, you'd want to do the COPY shuffle for > example, or plan your compaction. > 20:20:41 [benoitc]: dch: yup > 20:20:48 [jan____]: that leads to 1.3.x (as I understand) invalidate any > previous view indexes which leads to what benoitc just said > 20:20:55 [wendall911]: jan____: not having support for both 1.2.x and 1.3.x > view formats would mean downtime. Huge downtime for some cases. > 20:21:03 [benoitc]: for old sig vs new sig, i can see that the new signature > make them more unique > 20:21:10 [dch]: yes, and there's no way to avoid it either. > 20:21:10 [jan____]: wendall911: yes, we want to avoid that > 20:21:28 [jan____]: we can defer it to view-compaction though > 20:21:39 [kocolosk]: I'd have to think we can add some upgrade clauses to > read the old signature format internally (though i've not looked closely at > it yet) > 20:21:39 [benoitc]: i think we need for that some feedback from davisp > 20:21:54 [jan____]: wel *can* modulo code that the db file handling already > has (not that we can copy anything I think) > 20:22:02 [kocolosk]: the change in location is a bit more annoying but should > be manageable, right? > 20:22:02 [jan____]: *well > 20:22:17 [benoitc]: kocolosk: yes should be possible, need to check if it > works well with the server that maintain states > 20:22:24 [kocolosk]: right > 20:22:39 [benoitc]: kocolosk: the location change is easier to solve imo > 20:22:47 [jan____]: kocolosk: wanna take a stab at it? > 20:22:54 [kocolosk]: try new location, if missing try old one? > 20:23:33 [kocolosk]: jan____: if i can't scare up davisp, yeah > 20:24:03 [jan____]: kocolosk: woot, I am up for testing this tomorrow, if you > can tag-team this over night > 20:24:10 [benoitc]: kocolosk: yes that can work > 20:24:25 [kocolosk]: jan____: kk > 20:24:56 [dch]: is this then per-db & view, or per view? and how will we know > that all the views are completely migrated/updated, and we can transfer the > old ones to .delete for example? > 20:25:12 [dch]: ACTION thinking out loud, not expecting an answer. > 20:25:33 [benoitc]: chewbranca: i think this test lacks of some datasets > types (size of docs and real latency) i will try it anyway > 20:26:10 [benoitc]: dch: can be done when accessing the view group maybe > 20:26:27 [chewbranca]: benoitc: definitely does, just put it together over > the weekend, lets chat later in this meeting about what can be done to make > it comprehensive > 20:26:48 [dch]: ok I think we are ready to move to next topic, unless people > have more to add? > 20:27:06 [dch]: anything else people want to add for 1.3.x issues? > 20:27:11 [benoitc]: chewbranca: fine for me > 20:27:19 [benoitc]: let's try tonmorrow though > 20:27:27 [benoitc]: i'm a bit toired tonight > 20:27:48 [benoitc]: dch: the doc link maybe > 20:27:55 [benoitc]: in fiuton > 20:28:04 [jan____]: one more point for the release team: this might push us > over the desired Feb 1st call-a-vote date. > 20:28:05 [dch]: I have one, I added a jira a while back to do sanity checking > that (a) we can read/write to key locations like logfiles, view and dbs, and > if not, fail with a graceful(ish) erlang message. I will take a whack at this > tomorrow, it will not be a lot of code > 20:28:10 [benoitc]: maybe if docs aren't build we can provide direct link to > readthe doc > 20:28:33 [dch]: benoitc: I am +1 on a direct link to docs.couchdb.org if the > docs aren't built. > 20:28:48 [jan____]: I liked the idae of doing the fake index.html with the > meta refresh > 20:28:48 [dch]: I haven't looked at Kxepal's patch yet. > 20:29:06 [jan____]: neither, dch wanna do that tomorrow? > 20:29:19 [dch]: jan____: read the patch? for sure. > 20:29:39 [benoitc]: dch: mm i would just display it, misued the word direct > 20:29:42 [benoitc]: misused > 20:29:49 [benoitc]: ACTION is training a new kbd > 20:30:32 [dch]: ok, next up is. > > > # 3. rcouch merge # > 20:30:40 [dch]: benoitc: all yours > 20:30:54 [rnewson]: jan____: what kocolosk said. > 20:31:04 [jan____]: rnewson: thx > 20:31:47 [benoitc]: ok > 20:31:54 [benoitc]: so i am thinking to start it next week > 20:32:09 [benoitc]: by first introducing rebar and a way to build without > autoconf > 20:32:24 [benoitc]: i won't try to fight with autotools for now > 20:32:32 [benoitc]: except if one think it's desired > 20:33:02 [dch]: benoitc: good call. and then work on the NIFs? I should be > able to help with that. > 20:33:09 [benoitc]: yes > 20:33:18 [benoitc]: i can say i have already all the nifs in rcouch > 20:33:32 [benoitc]: couch_collate for ex > 20:33:50 [dch]: yup, that should be pretty good. I think I forgot one piece > in jiffy for windows, but I assume we are sticking with ejson for the moment? > 20:33:54 [benoitc]: wich replace couch_drv & couch_ejson_compare > 20:34:09 [benoitc]: un didn't thin k baout that > 20:34:24 [benoitc]: yes so i will put this change in another commit > 20:34:32 [jan____]: dch: yeah, lets not make it more comlpicated than needed > 20:34:34 [jan____]: we can jiffy things in at any time later > 20:34:47 [benoitc]: going to jiffy is a 5 line patch anyway > 20:35:09 [jan____]: yes, but it adds complexity for all the systems we support > 20:35:17 [benoitc]: i agree > 20:35:24 [jan____]: cool > 20:35:47 [dch]: alright! sounds great, so what was actiomnable there? > 20:35:54 [benoitc]: i am not sure how to handle the spidermonkey case for now > 20:36:09 [dch]: #action benoitc working on introducting rebar to couch, as > part of rcouch merge > 20:36:32 [dch]: I think for spidermonkey we should seriously consider > consolidating on 1.8.5 only. > 20:36:32 [benoitc]: yes let put an action, what version should be supported > 20:36:39 [benoitc]: presumably we donyt replace everything for now > 20:36:54 [benoitc]: yes i'm +1 on it > 20:37:02 [dch]: for 1.3.x, no, this would come 1.4 or later not sure how that > is covered in semver. > 20:37:04 [jan____]: for master? +1 > 20:37:17 [benoitc]: yes for master > 20:37:32 [dch]: it's very S&M tonight > 20:37:41 [dch]: benoitc: anything else, or shall we move topic? > 20:37:56 [benoitc]: i think i have some answer > 20:38:09 [benoitc]: will make spome update on the ml next week > 20:38:17 [jan____]: cool > 20:38:19 [benoitc]: ACTION is disappearing for the fosdem > 20:38:24 [dch]: looking forwards to it! > 20:38:32 [dch]: have fun benoitc. > > > # 4. fauxon, bring me all your UX goodness. # > 20:38:49 [dch]: drsm79: how does your 3G go? > 20:39:04 [dch]: or chewbranca maybe it was your topic (sorry!) > 20:39:32 [chewbranca]: fauxton is coming along well, plugin system is solid > and we've been getting some help from Garren and a few others > 20:40:09 [chewbranca]: we're in the process of getting CLAs taken care of so > that we can move the code into the primary repo and use standard ticketing > tools and what not > 20:40:11 [dch]: great! are we ready to set up an integration branch for this > or are you still githubbing? > 20:40:24 [jan____]: Simon got @gr2m interested in hacking too > 20:40:33 [jan____]: I want to move the code to asf asap > 20:40:39 [dch]: chewbranca: right. let me know if you need a hand with this, > the actual process is easy. > 20:40:47 [jan____]: Ill handle the apperwork, I hope I get this done tomorrow > 20:40:47 [drsm79]: dch what chewbranca said + more hands welcome > 20:40:47 [jan____]: *paper > 20:40:54 [drsm79]: dch: it's on a branch in a fork of couch > 20:40:54 [drsm79]: we can switch it over whenever I think > 20:41:02 [jan____]: then I want to kick off votes for all fauxton committers > 20:41:04 [dch]: fun-tastic! > 20:41:10 [chewbranca]: plenty left to do, will have a view editor working > this week that would be good to get some feedback > 20:41:32 [chewbranca]: yeah jan____ is helping getting the paperwork taken > care of, I'll get my CLA signed and scanned and sent off to jan____ today > 20:41:40 [drsm79]: I'll do mine tomorrow > 20:41:47 [jan____]: also, @gr2m is a little frustrated with fauxton > development. I asked him what hed like it to look at and he wrote us up a > fake readme: http://minutes.jit.su > 20:41:47 [jan____]: oops > 20:41:56 [jan____]: that fake readme is actually in > https://gist.github.com/4672813 > 20:42:24 [drsm79]: 3g is failing me > 20:42:47 [drsm79]: will read it > 20:42:54 [chewbranca]: jan____: sure, things could be easier, and we just > need to throw deps into a package.json and then you can just npm install > 20:42:54 [Jarda]: jan____: that looks awesome to me > 20:43:09 [jan____]: Jarda: yeah, sadly it isnt that for now > 20:43:33 [chewbranca]: I'll work on simplifying the setup process a bit this > week > 20:43:40 [jan____]: but I think we can approximate things > 20:43:47 [benoitc]: mmm btw does npp or other can keep deps offline ? > 20:43:49 [dch]: #action ASF CLAs underway for fauxtoneers > 20:43:54 [jan____]: I wonder how folks feel about a separate repo for fauxton? > 20:43:55 [benoitc]: for a n archive release i mean > 20:43:56 [dch]: benoitc: yes. > 20:44:09 [daleharvey_]: I think a seperate repo is a good idea > 20:44:12 [dch]: benoitc: yes you can in principle commit them. > 20:44:17 [benoitc]: also everything can be solved later during full integraton > 20:44:17 [Jarda]: jan____: so what is the tech stack for current fauxton? > 20:44:24 [jan____]: chewbranca: awesome, Im happy to help test and nudge > towards MOAR EASY so we can get the gr2m types aboard. > 20:44:24 [benoitc]: dch: mm not commit > 20:44:39 [benoitc]: or maybe > 20:44:54 [jan____]: Jarda: chewbranca knows ;) > 20:45:04 [chewbranca]: Jarda: its a backbone app with grunt for > building/tasks and a handful of other tools like backbone layoutmanager > 20:45:09 [Jarda]: I have a solid background in client-side javascript > (backbone/requirejs/etc) > 20:45:11 [benoitc]: we need to integrate sonme depenqncies on release but > don't really need all in the repo > 20:45:24 [Jarda]: We also use yeoman in our current project at work > 20:45:39 [chewbranca]: current code is at: > https://github.com/cloudant-labs/couchdb/tree/fauxton/src/fauxton > 20:45:54 [Jarda]: I might be able to help if there are clear issues to tackle > 20:45:57 [jan____]: Jarda: perfect :) > 20:46:02 [dch]: Jarda: feel free to pitch in, we'd love to have some helping > hands! > 20:46:39 [drsm79]: jan____: I'd be happy to have it in it's own repo etc > 20:46:40 [drsm79]: but don't know all the nitty gritty around what ASF require > 20:46:54 [jan____]: just admin stuff > 20:46:57 [jan____]: we can have more repos > 20:47:02 [drsm79]: moar > 20:47:09 [dch]: ok, moar fauxton or next topic? > 20:47:10 [jan____]: the only thing we need to solve then is how to get it > into `make dist` > 20:47:17 [drsm79]: do issues have to go in jira or can we do it all in github? > 20:47:17 [jan____]: moarepos > 20:47:17 [dch]: chewbranca: drsm79: great update. > 20:47:32 [chewbranca]: would be good to decide on a place to handle tickets > 20:47:54 [benoitc]: jira once the integration is done i think > 20:47:57 [drsm79]: dch: not bad considering being on a train ;) > 20:48:04 [jan____]: drsm79: once code is in ASF, issue go into jira, PRs can > go into github > 20:48:09 [chewbranca]: grunt is similar to yeoman, we can get down to a npm > install grunt gogo-gadget-fauxton and have it all running, but would be good > to have a place where we can start recording issues > 20:48:18 [chewbranca]: dch: thanks > 20:48:18 [jan____]: Id suggest abusing PRs for tickets > 20:48:32 [chewbranca]: jan____: that's what we've been doing > 20:48:32 [jan____]: tickets proper are ASF JIRA for now > 20:48:47 [jan____]: good > 20:48:49 [jan____]: keep it up :) > 20:49:02 [drsm79]: jan____: sounds good > 20:49:11 [jan____]: new PRs would go against > github.com/apache/couchdb-fauxton or whatever > 20:49:24 [Jarda]: ok so is there a roadmap for fauxton somewhere? > 20:49:39 [chewbranca]: should we temporarily have issues at the current > fauxton repo or in the futon organization? > 20:49:56 [jan____]: chewbranca: just to migrate things a week later? > 20:50:02 [chewbranca]: we're starting to get enough people involved that we > need somewhere for people to file issues > 20:50:09 [jan____]: chewbranca: we can start recording issues in ASF jira > right away > 20:50:11 [chewbranca]: jan____: good point, alright scratch that > 20:50:24 [benoitc]: ACTION dch ping here > 20:50:24 [benoitc]: err > 20:50:32 [Jarda]: if one would like to help, is the proper way to just make > it running and hack around? > 20:50:39 [benoitc]: z\/win 59 > 20:50:40 [chewbranca]: jan____: sounds good > 20:50:54 [dch]: next topic? > 20:50:55 [jan____]: chewbranca: aye > 20:50:57 [chewbranca]: Jarda: yeah, shoot me a ping after this and I can help > you get up and running > 20:51:02 [Jarda]: chewbranca: cool > 20:51:04 [jan____]: #action jan to add fauxton component to JIRA > 20:51:09 [jan____]: +1 EOD > 20:51:32 [dch]: ok I wanted to circle back to 1.3.x / CORS for a moment.. > > > # 5. 1.3.x CORS # > 20:52:02 [dch]: basically I did some testing, and I wondered if we > collectively feel ok or if we should be doing some more here? > 20:52:32 [benoitc]: ah more like what? > 20:52:39 [jan____]: dch: we ship as experimental > 20:52:40 [dch]: I got stuck reading some of the spec and translating that > into "testable" stuff. > 20:52:56 [jan____]: dch: I dont want to reopen this now. we dont get to ship > 1.3.x ever > 20:52:56 [dch]: pre-flight for example, I didn't follow that sufficiently yet. > 20:53:09 [dch]: jan____: ok for me then. > 20:53:11 [benoitc]: we are ok on that imo > 20:53:32 [benoitc]: also i would say that having pouchdb running with it is > enoiugh for me :) > 20:53:32 [dch]: woot, I will see if I can put together an example or two for > people to reference. > 20:53:34 [daleharvey_]: trying to find my old comments, but I remember there > being a few minor feature requests, but mostly good, ship as is experimental > and then to be honest there was only a few things more needed > 20:53:47 [jan____]: yeah, lets have it hit eager users and let them find all > the issues :) > 20:53:54 [dch]: yay users! > 20:53:56 [jan____]: daleharvey_: I think I got through mot of your stuff > eventually > 20:54:03 [dch]: ok, next topic, any requests? > 20:54:24 [chewbranca]: benchmarking? > 20:54:34 [jan____]: I can give an update about CouchDB Conf > 20:55:04 [dch]: um how about we roll with jan____ then? > 20:55:04 [benoitc]: chewbranca: i need to go , as qction i would say let's > try together to build some datasets > 20:55:09 [Noah]: jan____: to the ML preferably > 20:55:09 [benoitc]: so we can test it > 20:55:09 [chewbranca]: yeah would be great to hear about CouchDB conf > 20:55:18 [drsm79]: jan____: we should start planning the next one ~now > 20:55:18 [chewbranca]: benoitc: sounds good > 20:55:18 [drsm79]: keep the momentum > > > # 6. couchdbconf was awesome # > 20:55:26 [jan____]: Noah: aye > 20:55:26 [Noah]: cool > 20:55:26 [jan____]: ok > 20:55:34 [jan____]: #action jan to mail about CouchDB Conf > 20:55:42 [jan____]: drsm79: +1 > 20:55:49 [chewbranca]: I'm definitely interested in getting a CouchDB > conference going out in Seattle > 20:55:58 [drsm79]: chewbranca in ~4 months time? > 20:56:06 [chewbranca]: yeap > 20:56:12 [drsm79]: gtg train has got home > 20:56:19 [jan____]: drsm79: safe home! > 20:56:27 [chewbranca]: drsm79: later > 20:56:27 [jan____]: chewbranca: yeah, awesome > 20:56:52 [jan____]: chewbranca: for me around JSConf US or NodeConf > 20:57:04 [jan____]: otherwise I can skype in > 20:57:25 [chewbranca]: jan____: sounds good > 20:58:09 [dch]: ok we have a choice, run over for 5-10 minutes and talk about > benchmarking or do a sprint for the finish. > 20:58:47 [jan____]: Im all actioned out, but you go ahead > 20:58:47 [dch]: chewbranca: any actions related to seattlecouch? > > > # 7. benchmarking # > 20:59:32 [dch]: #link > http://adamlofts.blogspot.co.at/2012/07/couchdb-benchmarking-i.html > 20:59:32 [chewbranca]: dch: if it sounds good to everyone I'll start looking > into it > 20:59:56 [chewbranca]: nice, good link > 21:00:17 [jan____]: nice link! > 21:00:24 [jan____]: yeah! :) > 21:00:24 [dch]: chewbranca: I would love to do some more work with what Adam > put together in Tsung, its on my list for a while. But I need to keep focused > on the other stuff first. > 21:01:17 [dch]: Also I use this a lot https://github.com/fdmanana/seatoncouch > its ruby, set it up with a while true; do and run multiple instances > pointing to different DBs. This is a nice testing approach. > 21:01:17 [chewbranca]: so my idea for all-the-numbers is to get a simple view > benchmark in place that will easily compare vanilla spidermonkey view engine, > with Jason's node.js engine, and then also compare a native erlang view > engine, to get some ideas on overall performance and cost paid for > serialization > 21:01:32 [chewbranca]: I've got this going: > https://github.com/chewbranca/all-the-numbers > 21:01:39 [dch]: #link https://github.com/chewbranca/all-the-numbers > 21:01:54 [chewbranca]: which is not Tsung by any means, but works ok and > saves the results back into couchdb for easy comparisons > 21:02:24 [chewbranca]: not opposed to using something else, but I want to > understand the performance characteristics of the different options, and I > think it would be good to get that information out > 21:02:40 [dch]: chewbranca: great, the view engine changes are a good place > to start. > 21:03:00 [chewbranca]: dch: yeah I sent out an email to dev@ comparing 1.2.x > with 1.3.x > 21:03:01 [dch]: there's also a python view server, I think kxepal wrote it ? > 21:03:38 [chewbranca]: I can take care of adding a few of the different > engines, but what I could use help with is determining the appropriate sample > data sets and proper views to comprehensively compare the engines > 21:04:02 [dch]: chewbranca: tricky to get real-world stuff. Um I know a few > people with large docs and complex views, maybe they can share stuff. > 21:04:23 [dch]: #action dch to find some people with large docs/complex views > for testing > 21:04:45 [chewbranca]: dch: agreed, if there are some known edge cases it > would be good to get those in > 21:05:08 [chewbranca]: I want to have a range of doc sizes and what not so > you can compare serialization cost as a function of doc size and number of > views per ddoc and what not > 21:05:23 [Kxepal]: dch: you want to benchmark it too? > 21:05:23 [dch]: chewbranca: a point made a while back is that testing should > ideally be done with concurrent runs. That doesn't need to be phase 1 but its > good to add it in future. > 21:05:41 [dch]: Kxepal: more to the point, are you interested in helping out > and giving it a test too ? > 21:05:54 [dch]: you know how it fits together so you'd be a great help :) > 21:06:08 [chewbranca]: dch: yeah I manually did 3 runs for the tests, but I'm > planning on adding automatic iterations > 21:07:10 [wendall911]: it isn't always ideal to run concurrently, especially > building large indexes. You may just get to the point where you're actually > just benchmarking the hardware, not the software. > 21:07:39 [Kxepal]: dch: no problem. however, for python there is some set of > cases when it will be faster due to rich stdlib and caching imports. but > mostly it's slower on long running tasks unless it's not a pypy > 21:08:18 [chewbranca]: what other concurrent requests would be good to do? > basically right now I'm just throwing a bunch of docs at the db and then > grabbing the view with limit=1 > 21:08:25 [dch]: chewbranca: I'd like to run this on windows as well, um maybe > I send you a patch to extract the OS etc for the reports. > 21:08:25 [chewbranca]: not testing incremental updates or anything else right > now > 21:08:33 [chewbranca]: dch: sounds good! > 21:08:49 [dch]: compaction > 21:09:33 [chewbranca]: compaction as in bench marking compaction as well, or > running compaction while bench marking the view? > 21:09:54 [wendall911]: Is it possible for the test runner to use tmpfs for > the databases? I fear that you'd be just doing IO testing on the hardware for > some of this. > 21:09:56 [dch]: e.g. run tests, compact, re-run tests. > 21:09:57 [chewbranca]: maybe action item for this is start a thread on the ML > about proper benchmarking scenarios and start adding those in > 21:10:17 [dch]: chewbranca: yes, or adding them into a jira ticket list. > 21:10:30 [chewbranca]: dch: ahhh, yeah I was just loading the db, deleting, > reloading the db > 21:10:32 [dch]: wendall911: I think we probably want both. > 21:10:53 [wendall911]: dch: I don't see that figuring out your hardware is > slow is really a useful benchmark > 21:11:00 [dch]: I definitely want to see difference between SSD and cloudy, > and real disk. > 21:11:08 [wendall911]: hmm, ok > 21:11:31 [dch]: wendall911: because for example the serialisation cost is > high, possibly comparable in some circumstances to disk IO slowness. > 21:11:44 [dch]: rampant speculation that. > 21:11:59 [wendall911]: so long as the benchmarks are run on hardware that the > test is the only thing running > 21:12:01 [dch]: but I've seen 650K individual docs going to the JS view > server. > 21:12:07 [chewbranca]: understanding the actual serialization costs was one > of my big motivations here > 21:12:07 [dch]: wendall911: yup. > 21:12:15 [dch]: chewbranca: I have some ideas on testing serialisation costs. > 21:12:45 [chewbranca]: dch: cool, definitely want to pick your brain on it > 21:12:50 [dch]: I can instrument erlang and node directly (ignoring > sandboxes) and we should be able to get fine-grained results. > 21:12:57 [chewbranca]: nice > 21:13:12 [dch]: but thats a wee bit away. I have the basic pieces working but > not massaging the data. > 21:13:19 [wendall911]: and we should only publish the tmpfs without fsync, so > we can be like all the other kids > 21:13:27 [dch]: wendall911: noatime FTW. > 21:13:34 [wendall911]: :) > 21:13:57 [dch]: chewbranca: if you're feeling frisky we could add with / > without snappy compression. > 21:14:12 [chewbranca]: dch: yeah the more tests we can do the merrier > 21:14:19 [dch]: there must be a generic "run tests with various combinations" > nodejs library out there. > 21:14:49 [chewbranca]: yeah worth looking into > 21:15:12 [dch]: so anything more? I'm getting calls to come to bed. > 21:15:14 [dch]: (sorry for the abrupt topic change) > 21:15:20 [chewbranca]: nope, glad to get the discussion going > 21:15:50 [Kxepal]: chewbranca: if you'd like I could share db for 3M of > production-world documents for testing - around 6GB of data and they are > complex to create interesting views that really maps and reduces. > 21:15:57 [dch]: yes, I am totally enthused. > 21:16:32 [dch]: Kxepal: that would be *awesome*. I am happy to sanitise the > content, and then we could potentially keep it as reference, if you agree. > 21:16:39 [dch]: and maybe we could rewrite some of the views in erlang. > 21:16:47 [dch]: for comparison. > 21:16:55 [wendall911]: dch: is there a place to have a hosted copy? Would be > nice to be able to run tests without distributing the data per-se, just sync > with hosted copy. > 21:17:02 [chewbranca]: Kxepal: yeah that would be great, especially if we can > throw that up on a CouchDB instance somewhere so people can replicate it in > for testing > 21:17:24 [Kxepal]: dch: simple erlang view that build index for single field > runs half hour for my server (: never tried python nor js for this db > 21:17:25 [dch]: wendall911: I am pretty sure between cloudant and iriscouch > we will find a way. Or use my mac as an endpoint. > 21:17:39 [wendall911]: haha > 21:18:04 [Kxepal]: chewbranca: ok, I'll prepare db on this week. ping me if I > didn't you > 21:18:09 [dch]: wendall911: what do you mean by hosted copy? a way to > retrieve the doc set for testing? > 21:18:17 [wendall911]: dch: yes > 21:18:24 [chewbranca]: yeah worst case scenario I've got a dedicated box with > couchdb running and plenty of bandwidth > 21:18:26 [dch]: #action kxepal to contribute real world docs and views for > testing - huge thanks! > 21:18:46 [dch]: chewbranca: like a spare 80 node cluster I'll bet. > 21:18:53 [dch]: ok any more topics? > 21:19:00 [dch]: we covered a lot > 21:19:01 [chewbranca]: lol > 21:19:23 [wendall911]: On a side note, I may be able to step into a > co-maintainer role on the epel packages (for centos, rhel, fedora) > 21:19:31 [wendall911]: working on it > 21:20:08 [wendall911]: But not really project related, just frustrated with > the slowness of releases downstream, so proactively trying to fix. > 21:20:18 [dch]: wendall911: that would be fantastic, we are working on build > bots (via jenkins etc) so you might be interested in that too. > 21:20:44 [wendall911]: ahh, would be nice. I think the show stopper for me is > js 1.7 on centos, it just sucks > > > # 8. packages # > 21:21:22 [dch]: #action wendall911 to come back to ML about EPEL packages > 21:21:30 [dch]: wendall911: hopefully that's generic enough to remind you. > 21:21:58 [wendall911]: dch: thanks, I would have done more in this area, just > work is overly busy atm :( > 21:22:12 [dch]: wendall911: damn that work. > 21:22:42 [dch]: imma closing the meeting unless we have more? >
