Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Anon. E. Moose (Code Review)]

Anon. E. Moose #1000171 has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1: Contrib+1

Analytics Compatibility Tests Successful
https://goo.gl/kta3Dz : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1: Integration-Tests+1

Integration Tests Successful

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/6266/ 
: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1: Contrib-2

BAD Compatibility Tests Failed

https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/3086/ : UNSTABLE

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

BAD Compatibility Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/3086/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/6266/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/3848/ 
(12/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/3828/
 (11/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/4112/ (10/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Anon. E. Moose (Code Review)]

Anon. E. Moose #1000171 has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Analytics Compatibility Compilation Successful
https://goo.gl/BY3bbt : SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-ensure-ancestor/1404/ 
(9/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/3612/ 
(7/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/3397/
 (8/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/3928/ (5/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/3309/ 
(6/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/8860/ (2/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-format/3356/ 
(3/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/3385/ 
(4/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/7337/ (1/12)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2604
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Luo Chen 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Avoid memory leak after delete file from Buf...

[Luo Chen (Code Review)]

Luo Chen has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/2604

Change subject: [NO ISSUE][STO] Avoid memory leak after delete file from 
BufferCache
..

[NO ISSUE][STO] Avoid memory leak after delete file from BufferCache

- user model changes: no
- storage format changes: no
- interface changes: no

Details:
- Currently whwn we delete a file from a buffer cache, we didn't
remove the file info immediately after the FileInfoMap. However,
typically we often set max open files as a max integer number.
After the system is running, it's possible that the FileInfoMap could
contain a huge number of entries. This patch deletes the file info
immediately when the caller wants to delete a file.

Change-Id: I8a199d3e83592425ab5055bd12bde519e80deb13
---
M 
hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java
M 
hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/file/BufferedFileHandle.java
2 files changed, 31 insertions(+), 48 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/04/2604/1

diff --git 
a/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java
 
b/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java
index 1443bbc..63acb54 100644
--- 
a/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java
+++ 
b/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java
@@ -77,7 +77,7 @@
 private final Queue headerPageCache = new 
ConcurrentLinkedQueue<>();
 
 //DEBUG
-private Level fileOpsLevel = Level.DEBUG;
+private final Level fileOpsLevel = Level.DEBUG;
 private ArrayList confiscatedPages;
 private Lock confiscateLock;
 private HashMap confiscatedPagesOwner;
@@ -607,34 +607,34 @@
 
 void write(CachedPage cPage) throws HyracksDataException {
 BufferedFileHandle fInfo = getFileInfo(cPage);
+if (fInfo == null) {
+throw HyracksDataException.create(ErrorCode.FILE_DOES_NOT_EXIST);
+}
 // synchronize on fInfo to prevent the file handle from being deleted 
until the page is written.
 synchronized (fInfo) {
-if (!fInfo.fileHasBeenDeleted()) {
-ByteBuffer buf = cPage.buffer.duplicate();
-final int totalPages = cPage.getFrameSizeMultiplier();
-final int extraBlockPageId = cPage.getExtraBlockPageId();
-final boolean contiguousLargePages = 
(BufferedFileHandle.getPageId(cPage.dpid) + 1) == extraBlockPageId;
-BufferCacheHeaderHelper header = checkoutHeaderHelper();
-try {
-buf.limit(contiguousLargePages ? pageSize * totalPages : 
pageSize);
-buf.position(0);
-long bytesWritten = 
ioManager.syncWrite(fInfo.getFileHandle(),
-
getOffsetForPage(BufferedFileHandle.getPageId(cPage.dpid)),
-header.prepareWrite(cPage, buf));
+ByteBuffer buf = cPage.buffer.duplicate();
+final int totalPages = cPage.getFrameSizeMultiplier();
+final int extraBlockPageId = cPage.getExtraBlockPageId();
+final boolean contiguousLargePages = 
(BufferedFileHandle.getPageId(cPage.dpid) + 1) == extraBlockPageId;
+BufferCacheHeaderHelper header = checkoutHeaderHelper();
+try {
+buf.limit(contiguousLargePages ? pageSize * totalPages : 
pageSize);
+buf.position(0);
+long bytesWritten = ioManager.syncWrite(fInfo.getFileHandle(),
+
getOffsetForPage(BufferedFileHandle.getPageId(cPage.dpid)), 
header.prepareWrite(cPage, buf));
 
-if (bytesWritten != (contiguousLargePages ? pageSize * 
(totalPages - 1) : 0)
-+ getPageSizeWithHeader()) {
-throw new HyracksDataException("Failed to write 
completely: " + bytesWritten);
-}
-} finally {
-returnHeaderHelper(header);
+if (bytesWritten != (contiguousLargePages ? pageSize * 
(totalPages - 1) : 0)
++ getPageSizeWithHeader()) {
+throw new HyracksDataException("Failed to write 
completely: " + bytesWritten);
 }
-if (totalPages > 1 && !contiguousLargePages) {
-buf.limit(totalPages * pageSize);
-ioManager.syncWrite(fInfo.getFileHandle(), 
getOffsetForPage(extraBlockPageId), buf);
-