Hey, sounds great overall. One minor thing, though: I submitted
a binary patch a while back to finalize incr/decr transport. I think
it's important that that goes in before clients can be ready.
While mercurial is obviously better than git at everything but
managing svn repos, I'd be quite happy if I could simply have my
changes published somewhere so I feel they weren't lost or becoming
stale and others could try them and tell me what I did wrong (mostly
so I could declare my client done sooner).
Would it make sense for someone to provide a repo conversion to
work from outside of danga and just declare svn officially unsupported?
--
Dustin Sallings (mobile)
On Feb 7, 2008, at 0:56, dormando <[EMAIL PROTECTED]> wrote:
Yo all,
The original plan for today was to pop on the list and yell "Hey!
Have some -rc's to test! Go nuts! Thanks for all the patches!" - but
my awesome fever for the last few days has pushed that off by ... a
few more days.
But it's totally cool, because it's happening. For the first time in
a long time memcached will act like it's releasing early, often.
Thanks to everyone for all of the recent code submissions and
bugfixes :)
So we have two -rc's which are coming up as soon as I can wrap a few
bug reports and merge all this code:
1.3.0-rc (the binary tree, in its current state)
1.2.5-rc (non-binary branch with all of the recent fixes and patches)
There might not be an immediate merge of 1.2.5 -> 1.3.0, but I'm not
convinced that matters immediately.
I believe should probably give the binary tree a couple of real
development releases until we've had a few client implementations.
So in the case of the protocol having to change due to issues with
actual client integration, we won't be backing out on a stable API.
SO! I'd like to lay out my own wishlist for memcached, some of which
is underway, some not:
- Binary protocol. It's been too long, lets finish it. We need more
native tests, and we need to benchmark its speed relative to the
text protocol, make sure we're actually coming out ahead here.
- Dynamic storage engines. Toru has a promising proof of concept
that I'd love to merge for an eventual 1.4 release.
Most of the memcached forks have been due to folks not liking the
storage method. While I disagree with how a lot of these forks are
done, there's definitely merit in the flexibility. A lot of these
folks start from scratch, but I believe memcached should be a
simple, efficient, network platform for dead simple key=value
datastores, no matter how they're actually implemented.
- Dedupe the error messages. We have 11 instances of "SERVER_ERROR
out of memory". This makes it impossible to even guess what
happened, even with full context. Did we fail to allocate a msghdr?
Did do_item_alloc fail? Who knows. We can do this for 1.2.5 if
there're people on board. I'll float the patches.
- Add stats about failures. I'd love counters for all of the times
something failed without causing an eviction. Sometimes machines
will be hosed for a long time before we realize the hit rate is way
down due to the OS refusing 50% of the mallocs. Hard to tell if
you've hit maxconns, blah blah blah. More easy work, maybe for 1.2.5.
- Method of dumping refcounts of "locked" items. I'm tracking down
what I suspect is a refcount bug, and I expect us to have more of
these in the future. It'll be great to be able to verify these
offhand. If someone isolates a bum server, they can dump slabs,
refcount, etc, and send to the list.
- Full item dump. I guess we should do this with the appropriate
caveats. There's a potential patch floating around, we'll see about
it.
- Fixing a lot of dynamic memory buffers. Per connection, per
freelist, whatever. A lot of these can be turned into linked lists
if, say, the 'item' struct had a 'next' pointer. This should remove
most of the various realloc logic.
- Clean up the network iov's to work like above, and also manage
generic freelisted buffers. Which would remove the need for the
specialized cas suffix buffer. These should be wrapped so they can
be linklisted, instead of managed via dynamic arrays per connection.
Could also be used with the binary protocol to reduce syscalls.
- TCP_CORK/TCP_NOPUSH where appropriate. I totally agree here, and
it used to explicitly do this.
- New slabber engine. I view this as a bonus to making the slabber
code dynamic. We can start cooking an experimental new slabber to do
slab-reassignment, fragmentation avoidance, etc. Without really
hosing mainline.
- More tests. We need better test coverage. Please submit tests!
- "Standard benchmarks". Memcached's contributions have historically
been based on "we ran this in production, here're some tests and
proof it works fine". Recently it's been more about rapid cleanups
and the new storage engine independence. It'd be really great to
have some (say libmemcached based) tests to exercize (under
realistic AND unrealistic measures!) everything.
- More documentation. I figure eventually someone will write docs
with better english, but there're tons of ideas floating around. We
should get them all down. Along with standard troubleshooting info,
of which there is practically none!
- Cache::Memcached updates. Uh, whoops. This doesn't support any of
the recent features.
---
On community related measures:
- Buildbot. I totally started on this, but have not completed it. I
want to offer a buildbot network to the developer community. If you
want a platform supported, provide a slave. Otherwise we will not
target it (besides a couple of standby's).
- GIT. SVN is dead. We'll need to move on to git or hg in order to
scale the developer pace. It's my understanding that just as many
folks on the list prefer GIT as they do Mercurial, and SVN is mainly
a convenience. Anything will be better. I could crack out weekly
releases if I didn't have to worry about SVN branches/merges.
- Add more committers. I promise you guys; we're in an awkward
position with commit access, but people at sixapart absolutely
acknowledge and care about it. I've been having meetings,
discussions, etc, about dealing with this sooner than later. I
mostly help with memcached on my own time. However, sixapart does
love memcached and the rest of the danga projects. I can't say
anything definitive, of course, but I feel like I can share the
feeling of "we give a shit" with all of you folks.
Part of having more committers means we need more dedicated
reviewers. Memcached is a stable project trusted by many, many
websites.
- We should do more, less formal, hackathons. These are great for
hashing out minor issues, and I feel with more frequence they'd also
be really good for cranking out code, too ;)
---
That's all for now.
I'm missing a number of line items (please don't be offended if I
missed mentioning your submissions!) If it's in my inbox, it'll get
a fair review and probably go in.
have fun,
-Dormando
