Re: [PR] DRILL-8484: HashJoinPOP memory leak is caused by an oom exception wh… (drill)
cgivre merged PR #2889: URL: https://github.com/apache/drill/pull/2889 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] DRILL-8484: HashJoinPOP memory leak is caused by an oom exception wh… (drill)
shfshihuafeng commented on PR #2889: URL: https://github.com/apache/drill/pull/2889#issuecomment-2021914479 > @shfshihuafeng Can you please resolve merge conflicts. it is done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] DRILL-8484: HashJoinPOP memory leak is caused by an oom exception wh… (drill)
cgivre commented on PR #2889: URL: https://github.com/apache/drill/pull/2889#issuecomment-2020523775 @shfshihuafeng Can you please resolve merge conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] DRILL-8484: HashJoinPOP memory leak is caused by an oom exception wh… (drill)
shfshihuafeng commented on code in PR #2889:
URL: https://github.com/apache/drill/pull/2889#discussion_r1529743261
##
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java:
##
@@ -155,12 +157,18 @@ public void readFromStreamWithContainer(VectorContainer
myContainer, InputStream
for (SerializedField metaData : fieldList) {
final int dataLength = metaData.getBufferLength();
final MaterializedField field = MaterializedField.create(metaData);
- final DrillBuf buf = allocator.buffer(dataLength);
- final ValueVector vector;
+ DrillBuf buf = null;
+ ValueVector vector = null;
try {
+buf = allocator.buffer(dataLength);
buf.writeBytes(input, dataLength);
vector = TypeHelper.getNewVector(field, allocator);
vector.load(metaData, buf);
+ } catch (OutOfMemoryException oom) {
+for (ValueVector valueVector : vectorList) {
+ valueVector.clear();
+}
+throw UserException.memoryError(oom).message("Allocator memory
failed").build(logger);
Review Comment:
when we prepare to allocator memory using "allocator.buffer(dataLength)"
for hashjoinPop allocator, if actual memory > maxAllocation(The parameter is
calculated by call computeOperatorMemory) ,then it throw exception, like
following my test。
user can adjust directMemory parameters (DRILL_MAX_DIRECT_MEMORY) or
reduce concurrency based on actual conditions.
**throw exception code**
```
public DrillBuf buffer(final int initialRequestSize, BufferManager manager) {
assertOpen();
AllocationOutcome outcome = allocateBytes(actualRequestSize);
if (!outcome.isOk()) {
throw new OutOfMemoryException(createErrorMsg(this, actualRequestSize,
initialRequestSize));
}
```
**my test scenario**
```
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
allocate buffer of size 16384 (rounded from 14359) due to memory limit
(41943040). Current allocation: 22583616
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
at
org.apache.drill.exec.cache.VectorAccessibleSerializable.readFromStreamWithContainer(VectorAccessibleSerializable.java:172)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] DRILL-8484: HashJoinPOP memory leak is caused by an oom exception wh… (drill)
shfshihuafeng commented on code in PR #2889:
URL: https://github.com/apache/drill/pull/2889#discussion_r1529743261
##
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java:
##
@@ -155,12 +157,18 @@ public void readFromStreamWithContainer(VectorContainer
myContainer, InputStream
for (SerializedField metaData : fieldList) {
final int dataLength = metaData.getBufferLength();
final MaterializedField field = MaterializedField.create(metaData);
- final DrillBuf buf = allocator.buffer(dataLength);
- final ValueVector vector;
+ DrillBuf buf = null;
+ ValueVector vector = null;
try {
+buf = allocator.buffer(dataLength);
buf.writeBytes(input, dataLength);
vector = TypeHelper.getNewVector(field, allocator);
vector.load(metaData, buf);
+ } catch (OutOfMemoryException oom) {
+for (ValueVector valueVector : vectorList) {
+ valueVector.clear();
+}
+throw UserException.memoryError(oom).message("Allocator memory
failed").build(logger);
Review Comment:
when we prepare to allocator memory using "allocator.buffer(dataLength)"
for hashjoinPop allocator, if actualmemory > maxAllocation(The parameter is
calculated by call computeOperatorMemory) ,then it throw exception, like
following my test。
user can adjust directMemory parameters (DRILL_MAX_DIRECT_MEMORY) or
reduce concurrency based on actual conditions.
```
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
allocate buffer of size 16384 (rounded from 14359) due to memory limit
(41943040). Current allocation: 22583616
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
at
org.apache.drill.exec.cache.VectorAccessibleSerializable.readFromStreamWithContainer(VectorAccessibleSerializable.java:172)
```
##
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java:
##
@@ -155,12 +157,18 @@ public void readFromStreamWithContainer(VectorContainer
myContainer, InputStream
for (SerializedField metaData : fieldList) {
final int dataLength = metaData.getBufferLength();
final MaterializedField field = MaterializedField.create(metaData);
- final DrillBuf buf = allocator.buffer(dataLength);
- final ValueVector vector;
+ DrillBuf buf = null;
+ ValueVector vector = null;
try {
+buf = allocator.buffer(dataLength);
buf.writeBytes(input, dataLength);
vector = TypeHelper.getNewVector(field, allocator);
vector.load(metaData, buf);
+ } catch (OutOfMemoryException oom) {
+for (ValueVector valueVector : vectorList) {
+ valueVector.clear();
+}
+throw UserException.memoryError(oom).message("Allocator memory
failed").build(logger);
Review Comment:
when we prepare to allocator memory using "allocator.buffer(dataLength)"
for hashjoinPop allocator, if actual memory > maxAllocation(The parameter is
calculated by call computeOperatorMemory) ,then it throw exception, like
following my test。
user can adjust directMemory parameters (DRILL_MAX_DIRECT_MEMORY) or
reduce concurrency based on actual conditions.
```
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
allocate buffer of size 16384 (rounded from 14359) due to memory limit
(41943040). Current allocation: 22583616
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
at
org.apache.drill.exec.cache.VectorAccessibleSerializable.readFromStreamWithContainer(VectorAccessibleSerializable.java:172)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] DRILL-8484: HashJoinPOP memory leak is caused by an oom exception wh… (drill)
shfshihuafeng commented on code in PR #2889:
URL: https://github.com/apache/drill/pull/2889#discussion_r1529743261
##
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java:
##
@@ -155,12 +157,18 @@ public void readFromStreamWithContainer(VectorContainer
myContainer, InputStream
for (SerializedField metaData : fieldList) {
final int dataLength = metaData.getBufferLength();
final MaterializedField field = MaterializedField.create(metaData);
- final DrillBuf buf = allocator.buffer(dataLength);
- final ValueVector vector;
+ DrillBuf buf = null;
+ ValueVector vector = null;
try {
+buf = allocator.buffer(dataLength);
buf.writeBytes(input, dataLength);
vector = TypeHelper.getNewVector(field, allocator);
vector.load(metaData, buf);
+ } catch (OutOfMemoryException oom) {
+for (ValueVector valueVector : vectorList) {
+ valueVector.clear();
+}
+throw UserException.memoryError(oom).message("Allocator memory
failed").build(logger);
Review Comment:
when we prepare to allocator memory using "allocator.buffer(dataLength)"
for hashjoinPop allocator, if memory > maxAllocation(The parameter is
calculated by call computeOperatorMemory) ,then it throw exception, like
following my test。
user can adjust directMemory parameters (DRILL_MAX_DIRECT_MEMORY) or
reduce concurrency based on actual conditions.
```
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
allocate buffer of size 16384 (rounded from 14359) due to memory limit
(41943040). Current allocation: 22583616
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
at
org.apache.drill.exec.cache.VectorAccessibleSerializable.readFromStreamWithContainer(VectorAccessibleSerializable.java:172)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] DRILL-8484: HashJoinPOP memory leak is caused by an oom exception wh… (drill)
shfshihuafeng commented on code in PR #2889:
URL: https://github.com/apache/drill/pull/2889#discussion_r1529743261
##
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java:
##
@@ -155,12 +157,18 @@ public void readFromStreamWithContainer(VectorContainer
myContainer, InputStream
for (SerializedField metaData : fieldList) {
final int dataLength = metaData.getBufferLength();
final MaterializedField field = MaterializedField.create(metaData);
- final DrillBuf buf = allocator.buffer(dataLength);
- final ValueVector vector;
+ DrillBuf buf = null;
+ ValueVector vector = null;
try {
+buf = allocator.buffer(dataLength);
buf.writeBytes(input, dataLength);
vector = TypeHelper.getNewVector(field, allocator);
vector.load(metaData, buf);
+ } catch (OutOfMemoryException oom) {
+for (ValueVector valueVector : vectorList) {
+ valueVector.clear();
+}
+throw UserException.memoryError(oom).message("Allocator memory
failed").build(logger);
Review Comment:
when we prepare to allocator memory using "allocator.buffer(dataLength)"
for hashjoinPop allocator, if memory > maxAllocation(The parameter is
calculated by call computeOperatorMemory) then it throw exception, like
following my test。user can adjust directMemory parameters
(DRILL_MAX_DIRECT_MEMORY) or reduce concurrency based on actual conditions.
```
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
allocate buffer of size 16384 (rounded from 14359) due to memory limit
(41943040). Current allocation: 22583616
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
at
org.apache.drill.exec.cache.VectorAccessibleSerializable.readFromStreamWithContainer(VectorAccessibleSerializable.java:172)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] DRILL-8484: HashJoinPOP memory leak is caused by an oom exception wh… (drill)
cgivre commented on code in PR #2889:
URL: https://github.com/apache/drill/pull/2889#discussion_r1528802856
##
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java:
##
@@ -155,12 +157,18 @@ public void readFromStreamWithContainer(VectorContainer
myContainer, InputStream
for (SerializedField metaData : fieldList) {
final int dataLength = metaData.getBufferLength();
final MaterializedField field = MaterializedField.create(metaData);
- final DrillBuf buf = allocator.buffer(dataLength);
- final ValueVector vector;
+ DrillBuf buf = null;
+ ValueVector vector = null;
try {
+buf = allocator.buffer(dataLength);
buf.writeBytes(input, dataLength);
vector = TypeHelper.getNewVector(field, allocator);
vector.load(metaData, buf);
+ } catch (OutOfMemoryException oom) {
+for (ValueVector valueVector : vectorList) {
+ valueVector.clear();
+}
+throw UserException.memoryError(oom).message("Allocator memory
failed").build(logger);
Review Comment:
Do we know what would cause an error like this? If so what would the user
need to do to fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
