[GitHub] [kafka] cadonna commented on a change in pull request #10651: MINOR: Kafka Streams code samples formating unification

Fri, 21 May 2021 02:00:18 -0700 


cadonna commented on a change in pull request #10651:
URL: https://github.com/apache/kafka/pull/10651#discussion_r636733013



##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -171,44 +158,42 @@ <h2><a class="toc-backref" href="#id3">RocksDB</a><a 
class="headerlink" href="#r
         <code class="docutils literal"><span 
class="pre">rocksdb.config.setter</span></code> configuration.</p>
       <p>Also, we recommend changing RocksDB's default memory allocator, 
because the default allocator may lead to increased memory consumption.
         To change the memory allocator to <code>jemalloc</code>, you need to 
set the environment variable <code>LD_PRELOAD</code>before you start your Kafka 
Streams application:</p>
-      <pre>
-# example: install jemalloc (on Debian)
+      <pre class="line-numbers"><code class="language-bash"># example: install 
jemalloc (on Debian)
 $ apt install -y libjemalloc-dev
 # set LD_PRELOAD before you start your Kafka Streams application
 $ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so”
-      </pre>
+      </code></pre>
       <p> As of 2.3.0 the memory usage across all instances can be bounded, 
limiting the total off-heap memory of your Kafka Streams application. To do so 
you must configure RocksDB to cache the index and filter blocks in the block 
cache, limit the memtable memory through a shared <a class="reference external" 
href="https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager";>WriteBufferManager</a>
 and count its memory against the block cache, and then pass the same Cache 
object to each instance. See <a class="reference external" 
href="https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB";>RocksDB 
Memory Usage</a> for details. An example RocksDBConfigSetter implementing this 
is shown below:</p>
+      <pre class="line-numbers"><code class="language-java">public static 
class BoundedMemoryRocksDBConfig implements RocksDBConfigSetter {
 
-      <div class="highlight-java"><div class="highlight"><pre><span></span>    
<span class="kd">public</span> <span class="kd">static</span> <span 
class="kd">class</span> <span class="nc">BoundedMemoryRocksDBConfig</span> 
<span class="kd">implements</span> <span class="n">RocksDBConfigSetter</span> 
<span class="o">{</span>
-
-       <span class="kd">private</span> <span class="kt">static</span> <span 
class="n">org.rocksdb.Cache</span> <span class="n">cache</span> <span 
class="o">=</span> <span class="k">new</span> <span class="n">org</span><span 
class="o">.</span><span class="na">rocksdb</span><span class="o">.</span><span 
class="na">LRUCache</span><span class="o">(</span><span 
class="mi">TOTAL_OFF_HEAP_MEMORY</span><span class="o">,</span> <span 
class="n">-1</span><span class="o">,</span> <span class="n">false</span><span 
class="o">,</span> <span class="n">INDEX_FILTER_BLOCK_RATIO</span><span 
class="o">);</span><sup><a href="#fn1" id="ref1">1</a></sup>
-       <span class="kd">private</span> <span class="kt">static</span> <span 
class="n">org.rocksdb.WriteBufferManager</span> <span 
class="n">writeBufferManager</span> <span class="o">=</span> <span 
class="k">new</span> <span class="n">org</span><span class="o">.</span><span 
class="na">rocksdb</span><span class="o">.</span><span 
class="na">WriteBufferManager</span><span class="o">(</span><span 
class="mi">TOTAL_MEMTABLE_MEMORY</span><span class="o">,</span> cache<span 
class="o">);</span>
+   private static org.rocksdb.Cache cache = new 
org.rocksdb.LRUCache(TOTAL_OFF_HEAP_MEMORY, -1, false, 
INDEX_FILTER_BLOCK_RATIO);1

Review comment:
       The `1` at the end should be a superscript reference to a footnote. Is 
there a way to get again a superscript here? If not what would be the 
alternatives? 

##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -171,44 +158,42 @@ <h2><a class="toc-backref" href="#id3">RocksDB</a><a 
class="headerlink" href="#r
         <code class="docutils literal"><span 
class="pre">rocksdb.config.setter</span></code> configuration.</p>
       <p>Also, we recommend changing RocksDB's default memory allocator, 
because the default allocator may lead to increased memory consumption.
         To change the memory allocator to <code>jemalloc</code>, you need to 
set the environment variable <code>LD_PRELOAD</code>before you start your Kafka 
Streams application:</p>
-      <pre>
-# example: install jemalloc (on Debian)
+      <pre class="line-numbers"><code class="language-bash"># example: install 
jemalloc (on Debian)
 $ apt install -y libjemalloc-dev
 # set LD_PRELOAD before you start your Kafka Streams application
 $ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so”
-      </pre>
+      </code></pre>
       <p> As of 2.3.0 the memory usage across all instances can be bounded, 
limiting the total off-heap memory of your Kafka Streams application. To do so 
you must configure RocksDB to cache the index and filter blocks in the block 
cache, limit the memtable memory through a shared <a class="reference external" 
href="https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager";>WriteBufferManager</a>
 and count its memory against the block cache, and then pass the same Cache 
object to each instance. See <a class="reference external" 
href="https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB";>RocksDB 
Memory Usage</a> for details. An example RocksDBConfigSetter implementing this 
is shown below:</p>
+      <pre class="line-numbers"><code class="language-java">public static 
class BoundedMemoryRocksDBConfig implements RocksDBConfigSetter {
 
-      <div class="highlight-java"><div class="highlight"><pre><span></span>    
<span class="kd">public</span> <span class="kd">static</span> <span 
class="kd">class</span> <span class="nc">BoundedMemoryRocksDBConfig</span> 
<span class="kd">implements</span> <span class="n">RocksDBConfigSetter</span> 
<span class="o">{</span>
-
-       <span class="kd">private</span> <span class="kt">static</span> <span 
class="n">org.rocksdb.Cache</span> <span class="n">cache</span> <span 
class="o">=</span> <span class="k">new</span> <span class="n">org</span><span 
class="o">.</span><span class="na">rocksdb</span><span class="o">.</span><span 
class="na">LRUCache</span><span class="o">(</span><span 
class="mi">TOTAL_OFF_HEAP_MEMORY</span><span class="o">,</span> <span 
class="n">-1</span><span class="o">,</span> <span class="n">false</span><span 
class="o">,</span> <span class="n">INDEX_FILTER_BLOCK_RATIO</span><span 
class="o">);</span><sup><a href="#fn1" id="ref1">1</a></sup>
-       <span class="kd">private</span> <span class="kt">static</span> <span 
class="n">org.rocksdb.WriteBufferManager</span> <span 
class="n">writeBufferManager</span> <span class="o">=</span> <span 
class="k">new</span> <span class="n">org</span><span class="o">.</span><span 
class="na">rocksdb</span><span class="o">.</span><span 
class="na">WriteBufferManager</span><span class="o">(</span><span 
class="mi">TOTAL_MEMTABLE_MEMORY</span><span class="o">,</span> cache<span 
class="o">);</span>
+   private static org.rocksdb.Cache cache = new 
org.rocksdb.LRUCache(TOTAL_OFF_HEAP_MEMORY, -1, false, 
INDEX_FILTER_BLOCK_RATIO);1
+   private static org.rocksdb.WriteBufferManager writeBufferManager = new 
org.rocksdb.WriteBufferManager(TOTAL_MEMTABLE_MEMORY, cache);
 
-       <span class="nd">@Override</span>
-       <span class="kd">public</span> <span class="kt">void</span> <span 
class="nf">setConfig</span><span class="o">(</span><span 
class="kd">final</span> <span class="n">String</span> <span 
class="n">storeName</span><span class="o">,</span> <span 
class="kd">final</span> <span class="n">Options</span> <span 
class="n">options</span><span class="o">,</span> <span class="kd">final</span> 
<span class="n">Map</span><span class="o">&lt;</span><span 
class="n">String</span><span class="o">,</span> <span 
class="n">Object</span><span class="o">&gt;</span> <span 
class="n">configs</span><span class="o">)</span> <span class="o">{</span>
+   @Override
+   public void setConfig(final String storeName, final Options options, final 
Map&lt;String, Object&gt; configs) {
 
-         <span class="n">BlockBasedTableConfig</span> <span 
class="n">tableConfig</span> <span class="o">=</span> <span 
class="k">(BlockBasedTableConfig)</span> <span 
class="n">options</span><span><span class="o">.</span><span 
class="na">tableFormatConfig</span><span class="o">();</span>
+     BlockBasedTableConfig tableConfig = (BlockBasedTableConfig) 
options.tableFormatConfig();
 
-         <span class="c1"> // These three options in combination will limit 
the memory used by RocksDB to the size passed to the block cache 
(TOTAL_OFF_HEAP_MEMORY)</span>
-         <span class="n">tableConfig</span><span class="o">.</span><span 
class="na">setBlockCache</span><span class="o">(</span><span 
class="mi">cache</span><span class="o">);</span>
-         <span class="n">tableConfig</span><span class="o">.</span><span 
class="na">setCacheIndexAndFilterBlocks</span><span class="o">(</span><span 
class="kc">true</span><span class="o">);</span>
-         <span class="n">options</span><span class="o">.</span><span 
class="na">setWriteBufferManager</span><span class="o">(</span><span 
class="mi">writeBufferManager</span><span class="o">);</span>
+      // These three options in combination will limit the memory used by 
RocksDB to the size passed to the block cache (TOTAL_OFF_HEAP_MEMORY)
+     tableConfig.setBlockCache(cache);
+     tableConfig.setCacheIndexAndFilterBlocks(true);
+     options.setWriteBufferManager(writeBufferManager);
 
-         <span class="c1"> // These options are recommended to be set when 
bounding the total memory</span>
-         <span class="n">tableConfig</span><span class="o">.</span><span 
class="na">setCacheIndexAndFilterBlocksWithHighPriority</span><span 
class="o">(</span><span class="mi">true</span><span class="o">);</span><sup><a 
href="#fn2" id="ref2">2</a></sup>
-         <span class="n">tableConfig</span><span class="o">.</span><span 
class="na">setPinTopLevelIndexAndFilter</span><span class="o">(</span><span 
class="mi">true</span><span class="o">);</span>
-         <span class="n">tableConfig</span><span class="o">.</span><span 
class="na">setBlockSize</span><span class="o">(</span><span 
class="mi">BLOCK_SIZE</span><span class="o">);</span><sup><a href="#fn3" 
id="ref3">3</a></sup>
-         <span class="n">options</span><span class="o">.</span><span 
class="na">setMaxWriteBufferNumber</span><span class="o">(</span><span 
class="mi">N_MEMTABLES</span><span class="o">);</span>
-         <span class="n">options</span><span class="o">.</span><span 
class="na">setWriteBufferSize</span><span class="o">(</span><span 
class="mi">MEMTABLE_SIZE</span><span class="o">);</span>
+      // These options are recommended to be set when bounding the total memory
+     tableConfig.setCacheIndexAndFilterBlocksWithHighPriority(true);2

Review comment:
       See my comment above for the `2` at the end.

##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -171,44 +158,42 @@ <h2><a class="toc-backref" href="#id3">RocksDB</a><a 
class="headerlink" href="#r
         <code class="docutils literal"><span 
class="pre">rocksdb.config.setter</span></code> configuration.</p>
       <p>Also, we recommend changing RocksDB's default memory allocator, 
because the default allocator may lead to increased memory consumption.
         To change the memory allocator to <code>jemalloc</code>, you need to 
set the environment variable <code>LD_PRELOAD</code>before you start your Kafka 
Streams application:</p>
-      <pre>
-# example: install jemalloc (on Debian)
+      <pre class="line-numbers"><code class="language-bash"># example: install 
jemalloc (on Debian)
 $ apt install -y libjemalloc-dev
 # set LD_PRELOAD before you start your Kafka Streams application
 $ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so”
-      </pre>
+      </code></pre>
       <p> As of 2.3.0 the memory usage across all instances can be bounded, 
limiting the total off-heap memory of your Kafka Streams application. To do so 
you must configure RocksDB to cache the index and filter blocks in the block 
cache, limit the memtable memory through a shared <a class="reference external" 
href="https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager";>WriteBufferManager</a>
 and count its memory against the block cache, and then pass the same Cache 
object to each instance. See <a class="reference external" 
href="https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB";>RocksDB 
Memory Usage</a> for details. An example RocksDBConfigSetter implementing this 
is shown below:</p>
+      <pre class="line-numbers"><code class="language-java">public static 
class BoundedMemoryRocksDBConfig implements RocksDBConfigSetter {
 
-      <div class="highlight-java"><div class="highlight"><pre><span></span>    
<span class="kd">public</span> <span class="kd">static</span> <span 
class="kd">class</span> <span class="nc">BoundedMemoryRocksDBConfig</span> 
<span class="kd">implements</span> <span class="n">RocksDBConfigSetter</span> 
<span class="o">{</span>
-
-       <span class="kd">private</span> <span class="kt">static</span> <span 
class="n">org.rocksdb.Cache</span> <span class="n">cache</span> <span 
class="o">=</span> <span class="k">new</span> <span class="n">org</span><span 
class="o">.</span><span class="na">rocksdb</span><span class="o">.</span><span 
class="na">LRUCache</span><span class="o">(</span><span 
class="mi">TOTAL_OFF_HEAP_MEMORY</span><span class="o">,</span> <span 
class="n">-1</span><span class="o">,</span> <span class="n">false</span><span 
class="o">,</span> <span class="n">INDEX_FILTER_BLOCK_RATIO</span><span 
class="o">);</span><sup><a href="#fn1" id="ref1">1</a></sup>
-       <span class="kd">private</span> <span class="kt">static</span> <span 
class="n">org.rocksdb.WriteBufferManager</span> <span 
class="n">writeBufferManager</span> <span class="o">=</span> <span 
class="k">new</span> <span class="n">org</span><span class="o">.</span><span 
class="na">rocksdb</span><span class="o">.</span><span 
class="na">WriteBufferManager</span><span class="o">(</span><span 
class="mi">TOTAL_MEMTABLE_MEMORY</span><span class="o">,</span> cache<span 
class="o">);</span>
+   private static org.rocksdb.Cache cache = new 
org.rocksdb.LRUCache(TOTAL_OFF_HEAP_MEMORY, -1, false, 
INDEX_FILTER_BLOCK_RATIO);1
+   private static org.rocksdb.WriteBufferManager writeBufferManager = new 
org.rocksdb.WriteBufferManager(TOTAL_MEMTABLE_MEMORY, cache);
 
-       <span class="nd">@Override</span>
-       <span class="kd">public</span> <span class="kt">void</span> <span 
class="nf">setConfig</span><span class="o">(</span><span 
class="kd">final</span> <span class="n">String</span> <span 
class="n">storeName</span><span class="o">,</span> <span 
class="kd">final</span> <span class="n">Options</span> <span 
class="n">options</span><span class="o">,</span> <span class="kd">final</span> 
<span class="n">Map</span><span class="o">&lt;</span><span 
class="n">String</span><span class="o">,</span> <span 
class="n">Object</span><span class="o">&gt;</span> <span 
class="n">configs</span><span class="o">)</span> <span class="o">{</span>
+   @Override
+   public void setConfig(final String storeName, final Options options, final 
Map&lt;String, Object&gt; configs) {
 
-         <span class="n">BlockBasedTableConfig</span> <span 
class="n">tableConfig</span> <span class="o">=</span> <span 
class="k">(BlockBasedTableConfig)</span> <span 
class="n">options</span><span><span class="o">.</span><span 
class="na">tableFormatConfig</span><span class="o">();</span>
+     BlockBasedTableConfig tableConfig = (BlockBasedTableConfig) 
options.tableFormatConfig();
 
-         <span class="c1"> // These three options in combination will limit 
the memory used by RocksDB to the size passed to the block cache 
(TOTAL_OFF_HEAP_MEMORY)</span>
-         <span class="n">tableConfig</span><span class="o">.</span><span 
class="na">setBlockCache</span><span class="o">(</span><span 
class="mi">cache</span><span class="o">);</span>
-         <span class="n">tableConfig</span><span class="o">.</span><span 
class="na">setCacheIndexAndFilterBlocks</span><span class="o">(</span><span 
class="kc">true</span><span class="o">);</span>
-         <span class="n">options</span><span class="o">.</span><span 
class="na">setWriteBufferManager</span><span class="o">(</span><span 
class="mi">writeBufferManager</span><span class="o">);</span>
+      // These three options in combination will limit the memory used by 
RocksDB to the size passed to the block cache (TOTAL_OFF_HEAP_MEMORY)
+     tableConfig.setBlockCache(cache);
+     tableConfig.setCacheIndexAndFilterBlocks(true);
+     options.setWriteBufferManager(writeBufferManager);
 
-         <span class="c1"> // These options are recommended to be set when 
bounding the total memory</span>
-         <span class="n">tableConfig</span><span class="o">.</span><span 
class="na">setCacheIndexAndFilterBlocksWithHighPriority</span><span 
class="o">(</span><span class="mi">true</span><span class="o">);</span><sup><a 
href="#fn2" id="ref2">2</a></sup>
-         <span class="n">tableConfig</span><span class="o">.</span><span 
class="na">setPinTopLevelIndexAndFilter</span><span class="o">(</span><span 
class="mi">true</span><span class="o">);</span>
-         <span class="n">tableConfig</span><span class="o">.</span><span 
class="na">setBlockSize</span><span class="o">(</span><span 
class="mi">BLOCK_SIZE</span><span class="o">);</span><sup><a href="#fn3" 
id="ref3">3</a></sup>
-         <span class="n">options</span><span class="o">.</span><span 
class="na">setMaxWriteBufferNumber</span><span class="o">(</span><span 
class="mi">N_MEMTABLES</span><span class="o">);</span>
-         <span class="n">options</span><span class="o">.</span><span 
class="na">setWriteBufferSize</span><span class="o">(</span><span 
class="mi">MEMTABLE_SIZE</span><span class="o">);</span>
+      // These options are recommended to be set when bounding the total memory
+     tableConfig.setCacheIndexAndFilterBlocksWithHighPriority(true);2
+     tableConfig.setPinTopLevelIndexAndFilter(true);
+     tableConfig.setBlockSize(BLOCK_SIZE);3

Review comment:
       See my comment above for the `3` at the end.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to