Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/958#discussion_r140939339
--- Diff:
exec/memory/base/src/main/java/org/apache/drill/exec/memory/Accountant.java ---
@@ -22,16 +22,55 @@
import javax.annotation.concurrent.ThreadSafe;
import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.util.AssertionUtil;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
/**
* Provides a concurrent way to manage account for memory usage without
locking. Used as basis for Allocators. All
* operations are threadsafe (except for close).
*/
@ThreadSafe
-class Accountant implements AutoCloseable {
- // private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(Accountant.class);
+@VisibleForTesting
+public class Accountant implements AutoCloseable {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(Accountant.class);
+
+ // See DRILL-5808
+ // Allow a "grace margin" above the allocator limit for those operators
+ // that are trying to stay within the limit, but are hindered by the
+ // current power-of-two allocations and random vector doublings. Instead
+ // of failing the query, the accountant allows an excess allocation
within
+ // the grace, but issues a warning to the log.
+ //
+ // Grace is allowed only when requested, and then only in production
+ // code, or if enabled in debug code by setting the following system
+ // property.
+
+ public static final String STRICT_ALLOCATOR =
"drill.memory.strict.allocator";
--- End diff --
The tricky-bit is that the allocator/accountant is below the level of
java-exec, and so does not have visibility to either Drill config or the option
manager. So, this has to be set via a system option, as in the other memory
manager controls.
A possible enhancement would be to grab the options in the Drill server
startup (or main) and set statics down in the memory layer.
---
