>From Murtadha Hubail <mhub...@apache.org>: Attention is currently required from: Ritik Raj. Murtadha Hubail has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249 )
Change subject: [ASTERIXDB-3636][STO] Introducing column buffer pool ...................................................................... Patch Set 6: (20 comments) File asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/046e1be1_42397d8f PS6, Line 332: 0.0 make a constant indicating that column buffer memory is unlimited File asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/StorageProperties.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/608519d6_c25275ad PS6, Line 77: STORAGE_COLUMN_BUFFER_MAX_SIZE_PERCENTAGE Maybe STORAGE_COLUMN_BUFFER_POOL_MEMORY_PERCENTAGE https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/d8491659_a7140f4b PS6, Line 78: STORAGE_COLUMN_BUFFER_POOL_MAX_SIZE make ranged (1, xxxxxxx) https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/866d2262_b43f0854 PS6, Line 258: double poolSizePercentage = accessor.getDouble(Option.STORAGE_COLUMN_BUFFER_MAX_SIZE_PERCENTAGE); : long maxMemoryForColumnBuffers = (long) (MAX_HEAP_BYTES * (poolSizePercentage / 100)); : long maxSpaceAcquiredByPool = (long) getColumnBufferPoolMaxSize() * getColumnBufferSize(); : if (maxSpaceAcquiredByPool > maxMemoryForColumnBuffers) { : throw new IllegalStateException( : "Invalid column buffer pool configuration, more memory budgeted than available in JVM. JVM max memory: " : + MAX_HEAP_BYTES + ", Column buffer size: " + getColumnBufferSize() : + ", Column buffer pool size: " + getColumnBufferPoolMaxSize() : + ", Column buffer pool percentage: " + poolSizePercentage); : } I don't think you will need this once you make the configuration accept only valid values https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/491c2ff0_e500ea65 PS6, Line 295: // Should we consider only the column buffer pool size? account for the reserved percentage File hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/api/AbstractColumnTupleWriter.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/4d2000b7_74906fe0 PS6, Line 102: closeOnAbort abort File hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/btree/ColumnBTreeBulkloader.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/50728793_831b47d9 PS6, Line 63: writersInitialized writerInitialized https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/9d116fda_768723ad PS6, Line 67: columnCredits just called reservedBufferCount or something like that https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/057a3a1f_6ca49f38 PS6, Line 155: if (columnCredits > 0) { : columnBufferPool.release(columnCredits); : columnCredits = 0; : } refactor File hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/ColumnBufferPool.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/9445716e_0e522049 PS6, Line 36: remove https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/39bbc745_eaca3e7f PS6, Line 46: numFailedRecycle remove if not too useful https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/61de1dd9_bed45aa1 PS6, Line 83: nitialized: poolSize={}, cappedMemoryInBytes={}, maxCredits={}, bufferGranuleInBytes={}, reserveTimeoutMillis={}", remember to update with the new APIs names https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/48651b1b_b147b46c PS6, Line 129: permits use same language as reserve https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/4cd60e30_9ae96cd4 PS6, Line 168: buffer.clear(); remove one of the clears https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/5621f651_c949de99 PS6, Line 192: LOGGER.info error https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/9fe12179_d5caa5a7 PS6, Line 194: columnBufferGranuleInBytes update wording https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/b3a0ccd3_5bc357d3 PS6, Line 200: memory limit reached be specific about which memory limit was reached File hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/IBufferCache.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/f3ca0de6_9f58f4ad PS6, Line 277: revert File hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/IColumnBufferPool.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/7e59c28d_5c424015 PS6, Line 25: remove https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/70834af6_f2e01009 PS6, Line 29: Credits unify the wording -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: ionic Gerrit-Change-Id: I62437839d89b11d950e6715a00e844aedd0dab8e Gerrit-Change-Number: 20249 Gerrit-PatchSet: 6 Gerrit-Owner: Ritik Raj <ritik....@couchbase.com> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Gerrit-CC: Murtadha Hubail <mhub...@apache.org> Gerrit-Attention: Ritik Raj <ritik....@couchbase.com> Gerrit-Comment-Date: Tue, 26 Aug 2025 09:03:46 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Gerrit-MessageType: comment
