davisp commented on pull request #3127: URL: https://github.com/apache/couchdb/pull/3127#issuecomment-689002928
@garrensmith I've split the behavior/implementation between `couch_views_batch` (behavior) and `couch_views_batch_impl` (implementation). I don't believe this behavior is nearly abstract enough to cover anything besides couch_views so it doesn't make a lot of sense to me to create a new application to hold a new module that only couch_views will use. @iilyak For the scope I agree that this is a lot more narrow. I started by trying to figure out the AIMD/congestion algorithms in both couch_rate and couch_replicator but after thinking awhile I realized that having an AIMD based algorithm isn't appropriate for batch sizes since we want to avoid failing batches as much as possible because every failed batch is wasted time when building a view. I did a number of different approaches to try and measure optimal batch sizes and behaviors. An earlier version of this work actually tried to optimize the rate of indexing within the bounds of transaction sizes. That lead to the realization that just maximizing the work done in a single transaction was most beneficial overall for indexing throughput. That's how it ended up being so simple and narrow in the end. For the meck-via-Elixir, that would have worked if this test were running in the same VM but those tests are run separately against an instance of `dev/run`. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
