[
https://issues.apache.org/jira/browse/SOLR-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716226#comment-14716226
]
Shalin Shekhar Mangar commented on SOLR-7971:
---------------------------------------------
Thanks Yonik for reviewing.
I wrote a small JMH benchmark between different approaches at
https://github.com/shalinmangar/solr-jmh-tests
# JavaBinCodecBenchmark.testDefaultWriteStr measures unmodified JavaBinCodec
# JavaBinCodecBenchmark.testDirectBufferWriteStr is the direct buffer code
which made 3 copies
# JavaBinCodecBenchmark.testDirectBufferNoScratchWriteStr is direct buffer code
which writes to off-heap buffer without using an intermediate scratch. A 8KB
scratch is used to write contents of off-heap buffer
# JavaBinCodecBenchmark.testDoublePassWriteStr uses Mikhail's double pass
approach. We calculate the length first using a method which avoids all the
bitwise operations and then writes the bytes directly to the FastOutputStream
Here are the results:
{code}
10 MB JSON
------------------------------------------------------------------------------------------
java -server -Xmx2048M -Xms2048M -jar target/benchmarks.jar -wi 3 -i 3 -gc true
".*JavaBinCodecBenchmark.*"
# Run complete. Total time: 00:05:31
Benchmark Mode Cnt Score
Error Units
JavaBinCodecBenchmark.testDefaultWriteStr thrpt 30 47.662 ±
5.691 ops/s
JavaBinCodecBenchmark.testDirectBufferNoScratchWriteStr thrpt 30 43.079 ±
2.693 ops/s
JavaBinCodecBenchmark.testDirectBufferWriteStr thrpt 30 28.578 ±
1.290 ops/s
JavaBinCodecBenchmark.testDoublePassWriteStr thrpt 30 34.191 ±
1.268 ops/s
100 MB JSON
------------------------------------------------------------------------------------------
java -server -Xmx2048M -Xms2048M -jar target/benchmarks.jar -wi 3 -i 3 -gc true
".*JavaBinCodecBenchmark.*"
# Run complete. Total time: 00:06:38
Benchmark Mode Cnt Score
Error Units
JavaBinCodecBenchmark.testDefaultWriteStr thrpt 30 5.304 ±
0.318 ops/s
JavaBinCodecBenchmark.testDirectBufferNoScratchWriteStr thrpt 30 4.890 ±
0.514 ops/s
JavaBinCodecBenchmark.testDirectBufferWriteStr thrpt 30 3.274 ±
0.091 ops/s
JavaBinCodecBenchmark.testDoublePassWriteStr thrpt 30 2.194 ±
0.295 ops/s
{code}
I am not sure how well JMH works with IO benchmarks but it seems that the
double pass approach is too slow and using direct buffer without scratch works
best.
> Reduce memory allocated by JavaBinCodec to encode large strings
> ---------------------------------------------------------------
>
> Key: SOLR-7971
> URL: https://issues.apache.org/jira/browse/SOLR-7971
> Project: Solr
> Issue Type: Sub-task
> Components: Response Writers, SolrCloud
> Reporter: Shalin Shekhar Mangar
> Assignee: Shalin Shekhar Mangar
> Priority: Minor
> Fix For: Trunk, 5.4
>
> Attachments: SOLR-7971-directbuffer.patch,
> SOLR-7971-directbuffer.patch, SOLR-7971.patch
>
>
> As discussed in SOLR-7927, we can reduce the buffer memory allocated by
> JavaBinCodec while writing large strings.
> https://issues.apache.org/jira/browse/SOLR-7927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700420#comment-14700420
> {quote}
> The maximum Unicode code point (as of Unicode 8 anyway) is U+10FFFF
> ([http://www.unicode.org/glossary/#code_point]). This is encoded in UTF-16
> as surrogate pair {{\uDBFF\uDFFF}}, which takes up two Java chars, and is
> represented in UTF-8 as the 4-byte sequence {{F4 8F BF BF}}. This is likely
> where the mistaken 4-bytes-per-Java-char formulation came from: the maximum
> number of UTF-8 bytes required to represent a Unicode *code point* is 4.
> The maximum Java char is {{\uFFFF}}, which is represented in UTF-8 as the
> 3-byte sequence {{EF BF BF}}.
> So I think it's safe to switch to using 3 bytes per Java char (the unit of
> measurement returned by {{String.length()}}), like
> {{CompressingStoredFieldsWriter.writeField()}} does.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
