This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 81f433f Raise the threshold of using MMAP for inverted index creation
from 100M to 2G (#4358)
81f433f is described below
commit 81f433fbfd24890f9d003d02aaf3881dc33487f8
Author: Xiaotian (Jackie) Jiang <[email protected]>
AuthorDate: Mon Jun 24 17:57:37 2019 -0700
Raise the threshold of using MMAP for inverted index creation from 100M to
2G (#4358)
We have experienced very high GC stopping the thread time while generating
huge inverted index with MMAP files
Writing to MMAP files could potentially cause super long stopping the
thread time and cause the server/minion lose ZK connection
Raise the threshold to 2G (use direct memory if value buffer size is
smaller than 2G) to solve the issue
Generating extremely large inverted index (over 500M values) could still
face the same issue, but we want to keep the threshold to prevent running out
of direct memory
---
.../segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
index c4e30b7..b18f5a1 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
@@ -52,8 +52,8 @@ import org.roaringbitmap.buffer.MutableRoaringBitmap;
* <p>Based on the number of values we need to store, we use direct memory or
MMap file to allocate the buffer.
*/
public final class OffHeapBitmapInvertedIndexCreator implements
InvertedIndexCreator {
- // Use MMapBuffer if the buffer size is larger than 100MB
- private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 25_000_000;
+ // Use MMapBuffer if the value buffer size is larger than 2G
+ private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX =
".fwd.idx.val.buf";
private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX =
".fwd.idx.len.buf";
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
