ctubbsii commented on code in PR #3508:
URL: https://github.com/apache/accumulo/pull/3508#discussion_r1237771541
##########
server/base/src/main/java/org/apache/accumulo/server/mem/LowMemoryDetector.java:
##########
@@ -135,48 +134,55 @@ public void logGCInfo(AccumuloConfiguration conf) {
prevGcTime.put(gcBean.getName(), time);
}
- long mem = rt.freeMemory();
- if (maxIncreaseInCollectionTime == 0) {
- gcTimeIncreasedCount = 0;
- } else {
- gcTimeIncreasedCount++;
- if (gcTimeIncreasedCount > 3 && mem < rt.maxMemory() *
freeMemoryPercentage) {
+ Runtime rt = Runtime.getRuntime();
+ final long maxConfiguredMemory = rt.maxMemory();
+ final long allocatedMemory = rt.totalMemory();
+ final long allocatedFreeMemory = rt.freeMemory();
+ final long freeMemory = maxConfiguredMemory - (allocatedMemory -
allocatedFreeMemory);
+ final double lowMemoryThreshold = maxConfiguredMemory *
freeMemoryPercentage;
Review Comment:
Should cast this to long here, so it doesn't need to be done three times
later.
##########
test/src/main/java/org/apache/accumulo/test/functional/MemoryFreeingIterator.java:
##########
@@ -28,23 +28,39 @@
import org.apache.accumulo.core.iterators.IteratorEnvironment;
import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
import org.apache.accumulo.core.iterators.WrappingIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
public class MemoryFreeingIterator extends WrappingIterator {
- @Override
- public void init(SortedKeyValueIterator<Key,Value> source,
Map<String,String> options,
- IteratorEnvironment env) throws IOException {
- super.init(source, options, env);
+ private static final Logger LOG =
LoggerFactory.getLogger(MemoryFreeingIterator.class);
+
+ @SuppressFBWarnings(value = "DM_GC", justification = "gc is okay for test")
+ public MemoryFreeingIterator() {
+ super();
+ LOG.info("Freeing consumed memory");
MemoryConsumingIterator.freeBuffers();
while (this.isRunningLowOnMemory()) {
+ System.gc();
// wait for LowMemoryDetector to recognize the memory is free.
try {
Thread.sleep(SECONDS.toMillis(1));
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
- throw new IOException("wait for low memory detector interrupted", ex);
+ throw new RuntimeException("wait for low memory detector interrupted",
ex);
}
Review Comment:
For the test code, I feel like you could just throw the
InterruptedException, instead of trying to convert it to some other type. It
doesn't matter for the test. Just throwing what's there already is shorter, and
the less code the better, especially when trying to maintain complicated tests.
It's probably not even going to be thrown in the test anyway, unless we kill
our own test... in which case, we would expect it.
##########
server/base/src/main/java/org/apache/accumulo/server/mem/LowMemoryDetector.java:
##########
@@ -36,20 +36,21 @@
public class LowMemoryDetector {
+ private static final Logger LOG =
LoggerFactory.getLogger(LowMemoryDetector.class);
+
@FunctionalInterface
- public static interface Action {
+ public interface Action {
void execute();
}
Review Comment:
For what it's worth, it seems like Runnable would suffice here. No need for
a new interface.
##########
server/base/src/main/java/org/apache/accumulo/server/mem/LowMemoryDetector.java:
##########
@@ -135,48 +134,55 @@ public void logGCInfo(AccumuloConfiguration conf) {
prevGcTime.put(gcBean.getName(), time);
}
- long mem = rt.freeMemory();
- if (maxIncreaseInCollectionTime == 0) {
- gcTimeIncreasedCount = 0;
- } else {
- gcTimeIncreasedCount++;
- if (gcTimeIncreasedCount > 3 && mem < rt.maxMemory() *
freeMemoryPercentage) {
+ Runtime rt = Runtime.getRuntime();
+ final long maxConfiguredMemory = rt.maxMemory();
+ final long allocatedMemory = rt.totalMemory();
+ final long allocatedFreeMemory = rt.freeMemory();
+ final long freeMemory = maxConfiguredMemory - (allocatedMemory -
allocatedFreeMemory);
+ final double lowMemoryThreshold = maxConfiguredMemory *
freeMemoryPercentage;
+ LOG.trace("Memory info: max={}, allocated={}, free={}, free
threshold={}",
+ maxConfiguredMemory, allocatedMemory, freeMemory, (long)
lowMemoryThreshold);
+
+ if (freeMemory < (long) lowMemoryThreshold) {
+ lowMemCount++;
+ if (lowMemCount > 3 && !runningLowOnMemory) {
runningLowOnMemory = true;
- log.warn("Running low on memory");
- gcTimeIncreasedCount = 0;
+ LOG.warn("Running low on memory: max={}, allocated={}, free={}, free
threshold={}",
+ maxConfiguredMemory, allocatedMemory, freeMemory, (long)
lowMemoryThreshold);
+ }
+ } else {
+ // If we were running low on memory, but are not any longer, than log
at warn
+ // so that it shows up in the logs
+ if (runningLowOnMemory) {
+ LOG.warn("Recovered from low memory condition");
} else {
- // If we were running low on memory, but are not any longer, than
log at warn
- // so that it shows up in the logs
- if (runningLowOnMemory) {
- log.warn("Recovered from low memory condition");
- } else {
- log.trace("Not running low on memory");
- }
- runningLowOnMemory = false;
+ LOG.trace("Not running low on memory");
}
+ runningLowOnMemory = false;
+ lowMemCount = 0;
}
- if (mem != lastMemorySize) {
+ if (freeMemory != lastMemorySize) {
sawChange = true;
}
String sign = "+";
- if (mem - lastMemorySize <= 0) {
+ if (freeMemory - lastMemorySize <= 0) {
sign = "";
}
- sb.append(String.format(" freemem=%,d(%s%,d) totalmem=%,d", mem, sign,
(mem - lastMemorySize),
- rt.totalMemory()));
+ sb.append(String.format(" freemem=%,d(%s%,d) totalmem=%,d", freeMemory,
sign,
+ (freeMemory - lastMemorySize), rt.totalMemory()));
Review Comment:
You don't need the sign variable. Java String format already supports always
showing a sign, even if it's positive. Just use `%+,d` as the format string.
##########
test/src/main/java/org/apache/accumulo/test/functional/MemoryStarvedScanIT.java:
##########
@@ -392,21 +489,28 @@ public void testBatchScanPauses() throws Exception {
assertEquals(currentCount, fetched.get());
assertTrue(SCAN_START_DELAYED.doubleValue() >= paused);
assertEquals(returned, SCAN_RETURNED_EARLY.doubleValue());
- assertEquals(1, LOW_MEM_DETECTED.get());
+ // check across multiple low memory checks and metric updates that low
memory detected
+ // remains set
+ int checkCount = 6;
+ while (checkCount-- > 0) {
Review Comment:
This currently runs 6 times (5, 4, 3, 2, 1, 0). It seems like you intended
it to run 5 times (5, 4, 3, 2, 1).
```suggestion
while (--checkCount > 0) {
```
You could also be more explicit about the range you want to iterate over by
using IntStream and forEach:
```java
IntStream.rangeClosed(1, 5).map(i -> 6-i).forEach(System.out::println); // or
IntStream.rangeClosed(-5, -1).map(i -> -i).forEach(System.out::println);
```
That example goes backwards, like yours, but it's easier to go forwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
