Note: this appears to result in a slight performance regression,
I think, but I'm on a busy system...
In theory I expect it to work since smaller work units means
better work distribution when we're dealing with a few gigantic
inboxes (LKML) and many smaller ones; but perhaps the extra
IPC overhead from the multitudes of small inboxes on my lore
mirror ends up hurting.
Instead of dumping all Xapian shards of an inbox with a
single xap_helper command, issue one xap_helper command
per-shard (as we already do with the cidx shards) to get
smaller work units and improve work sharing.
---
lib/PublicInbox/CodeSearchIdx.pm | 18 +++++++++++-------
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/CodeSearchIdx.pm b/lib/PublicInbox/CodeSearchIdx.pm
index a9a7d313..d6348740 100644
--- a/lib/PublicInbox/CodeSearchIdx.pm
+++ b/lib/PublicInbox/CodeSearchIdx.pm
@@ -551,14 +551,18 @@ sub dump_roots_start {
sub dump_ibx { # sends to xap_helper.h
my ($self, $ibx_id) = @_;
my $ibx = $IBX[$ibx_id] // die "BUG: no IBX[$ibx_id]";
- my @cmd = ('dump_ibx', $ibx->isrch->xh_args,
- (map { ('-A', $_) } @ASSOC_PFX),
- $ibx_id, $QRY_STR);
- pipe(my ($r, $w)) or die "pipe: $!";
- $XHC->mkreq([$DUMP_IBX_WPIPE, $w], @cmd);
+ my @args = map { ('-A', $_) } @ASSOC_PFX;
my $ekey = $ibx->eidx_key;
- $self->{PENDING}->{$ekey} = $TODO{associate};
- PublicInbox::CidxXapHelperAux->new($r, $self, $ekey);
+ my $es = $ibx->isrch->{es};
+ push(@args, '-O', $ekey) if defined $es;
+ push(@args, $ibx_id, $QRY_STR);
+ for my $d ($es ? $es->shard_dirs : $ibx->isrch->shard_dirs) {
+ pipe(my ($r, $w)) or die "pipe: $!";
+ $XHC->mkreq([$DUMP_IBX_WPIPE, $w], qw(dump_ibx -d), $d, @args);
+ my $desc = "$ekey -d $d";
+ $self->{PENDING}->{$desc} = $TODO{associate};
+ PublicInbox::CidxXapHelperAux->new($r, $self, $desc);
+ }
}
sub dump_ibx_start {
