Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

[Fikavec F]

   I was able to create a collection with "solr.SimpleTextCodecFactory" codecFactory and solr can proces (return) only 2x more documents per second from it (214 410 documents per second vs 115 000 "solr.SchemaCodecFactory" with compression). I expected much much more, because this is a simple iteration and sending small fields to the output. Is this enough to make sure that the Solr limit of processing 115 000 documents per second is not due only  to compression, but something else? Or is the speed of SimpleTextCodecFactory in this case not an indicator for correct testing and yet it is necessary to create my own codecFactory class without compression? I also tried to create a collection with a standard codec of 8 shards for the test, the  documents iteration rate is the same about 115 000 small documents per second.

 

P.S. As a <codecFactory class="my.Lucene87CodecWithNoFieldCompression"/> in solrconfig.xml, I currently can't connect even a simple codec layer:

package my;

import org.apache.lucene.codecs.FilterCodec;

import org.apache.lucene.codecs.StoredFieldsFormat;

import org.apache.lucene.codecs.lucene87.Lucene87Codec;

import org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat;

 

public final class Lucene87CodecWithNoFieldCompression extends FilterCodec {

    private final StoredFieldsFormat storedFieldsFormat; 

 

    public Lucene87CodecWithNoFieldCompression() {

        super("Lucene87CodecWithNoFieldCompression", new Lucene87Codec());

        storedFieldsFormat = new Lucene87StoredFieldsFormat();

    }

    @Override

    public StoredFieldsFormat storedFieldsFormat() {

        return storedFieldsFormat;

    }

    @Override

    public String toString() {

      return getClass().getSimpleName();

    }

}

 

Best Regards,