Hi Adrien,
Great suggestion. If I understand correctly, then what you are suggesting is
something along the lines of ( subject to exact message details, which we can
trash out in a PR ):
diff --git
a/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java
b/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java
index 4500d5cf7ce..76f7ea2a9f2 100644
--- a/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java
+++ b/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java
@@ -691,13 +691,23 @@ public class ConcurrentMergeScheduler extends
MergeScheduler {
public void run() {
try {
if (verbose()) {
- message(" merge thread: start");
+ message(String.format(Locale.ROOT, "merge thread %s start",
this.getName()));
}
doMerge(mergeSource, merge);
if (verbose()) {
- message(" merge thread: done");
+ message(
+ String.format(
+ Locale.ROOT,
+ "merge thread %s done estSize=%.1f MB (written=%.1f MB)
runTime=%.1fs (stopped=%.1fs, paused=%.1fs) rate=%s",
+ this.getName(),
+ bytesToMB(merge.estimatedMergeBytes),
+ bytesToMB(rateLimiter.getTotalBytesWritten()),
+ nsToSec(System.nanoTime() - merge.mergeStartNS),
+ nsToSec(rateLimiter.getTotalStoppedNS()),
+ nsToSec(rateLimiter.getTotalPausedNS()),
+ rateToString(rateLimiter.getMBPerSec())));
}
runOnMergeFinished(mergeSource);
} catch (Throwable exc) {
Then ES can leverage such from the infoStream, right? ( thus avoiding the need
for ES extract the inaccessible information directly itself, while also being
more generally useful in Lucene logs ).
Or have I misinterpreted your comment?
-Chris.
> On 22 Sep 2021, at 10:12, Adrien Grand <[email protected]> wrote:
>
> Hi Chris,
>
> I looked into this and Elasticsearch seems to only need access to the rate
> limiter for logging purposes, without adding any information that Lucene
> doesn't have.
> Maybe another option would consist of moving the logging to Lucene? Having
> information in the IndexWriter's InfoStream about rate limiting for each
> completed merge sounds like something that would generally be useful.
>
