ilooner commented on a change in pull request #1351: DRILL-6543: Disable Hash
Join fallback, add percent_reserved_allowance_from_direct
URL: https://github.com/apache/drill/pull/1351#discussion_r199909294
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##########
@@ -490,6 +491,31 @@ private ExecConstants() {
public static DoubleValidator PERCENT_MEMORY_PER_QUERY = new
RangeDoubleValidator(
PERCENT_MEMORY_PER_QUERY_KEY, 0, 1.0);
+ /**
+ * Enforce reserving some percentage of the JVM's Direct Memory for the
+ * non-buffered operators. I.e., whether using max_query_memory_per_node or
+ * percent_per_query, the memory result (for all the buffered operators) can
+ * not exceed the size of the Direct Memory minus this reserved allowance.
+ * <p>
+ * This allowance is needed to prevent a potential OOM. In case the total
memory
+ * promised for the buffered operators is very close to the Direct Memory's
size,
+ * then if some non-buffered operators (e.g., scanners) also grab
significant memory,
+ * then the remaining memory is less than the promised memory, leading to an
OOM.
+ * </p>
+ * <p>
+ * Note that this enforcement is only good for the non concurrent case. For
multiple
+ * concurrent queries, other queries may grab some unaccounted for Direct
Memory. For
+ * concurrent work, better set the option "planner.memory.percent_per_query"
correctly.
+ * </p>
+ * <p>
+ * DEFAULT: 25%
+ * </p>
+ */
Review comment:
Thanks for adding this documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
