Github user kocolosk commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
Wow, great stuff @jaydoane
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user davisp commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
Good work and persistence, @jaydoane!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user davisp commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user davisp commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
Yep, +1 to squashing with that commit message
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user jaydoane commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
@davisp, formatting fixed in latest commit.
Here's a proposed commit message:
> Current behavior attempts to read a header at each block, starting at the
eof and working
Github user davisp commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
Oh, and we'll want to squash this down into a single commit that has a good
explanation of the change and the basics of how it works.
---
If your project is set up for it, you can reply to
Github user theburge commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
Thanks for doing the additional investigation @jaydoane. I'm glad the
performance remains consistently improved.
From memory, I've observed deeply buried headers frequently in MT,
Github user jaydoane commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
The following image adds vmpage_io.memory.in vmpage_io.memory.out to the
experiment. All experiments were searching 4 files in parallel. The first
starts around 23:48, using the current
Github user davisp commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
That seems fairly convincingly just all around better than the default
implementation. The only real worry I had was RAM usage with the vectored reads
but that appears to be a non issue (and
Github user theburge commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
@jaydoane Thanks for running and documenting those experiments, I think
those results look very promising. Reducing time taken and user-space memory
usage can only be good. :)
FWIW,
Github user jaydoane commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
@davisp, @theburge, et al, can you guys take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user jaydoane commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
I have simplified the pread/2 based header search, and hopefully addressed
most of the issues raised against the previous implementations. For a simple
speed comparison using the same file
Github user davisp commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
@jaydoane pointed out that its a WIP commit currently so I stopped
iterating style points until he's finished.
Also, reading through this its getting complicated pretty quickly between
Github user jaydoane commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
@davisp, thanks for the explanation. You were of course correct that I was
confusing `pread/3` with `pread/2`. I think it makes sense to look at an
alternate version of header search which,
Github user theburge commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
@jaydoane: One additional thought I had was that it's taking good care of
the page cache with `posix_fadvise`, but perhaps not quite such good care of
the BEAM's memory usage. Have you
Github user davisp commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
The implementation looks better. A few style issues I'll not nit pick on
till later.
I only meant "wrong" in so much in that it was a common recursion error in
pulling out an
Github user jaydoane commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
Thank you @davisp for the thorough review! I will address your points in
turn:
> the timing numbers reported are in microseconds so I'm not super
convinced on them. What size of
Github user davisp commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
I like the idea here but I think we need to rethink the implementation a
bit. First, the timing numbers reported are in microseconds so I'm not super
convinced on them. What size of file are
Github user jaydoane commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
@theburge, @kocolosk, @davisp, @eiri: can I get some feedback on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user jaydoane commented on the issue:
https://github.com/apache/couchdb-couch/pull/185
I've been testing with this repl command:
```erlang
_time_find_header = fun(File, Algorithm) when is_list(File) ->
FileSize = filelib:file_size(File),
{ok, Fd} =
20 matches
Mail list logo
