pitrou commented on code in PR #48108:
URL: https://github.com/apache/arrow/pull/48108#discussion_r2522280467
##########
cpp/src/arrow/memory_pool.h:
##########
@@ -245,6 +246,64 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+/// EXPERIMENTAL MemoryPool wrapper with an upper limit
+class ARROW_EXPORT CappedMemoryPool : public MemoryPool {
+ public:
+ CappedMemoryPool(MemoryPool* wrapped_pool, int64_t bytes_allocated_limit)
+ : wrapped_(wrapped_pool), bytes_allocated_limit_(bytes_allocated_limit)
{}
+
+ using MemoryPool::Allocate;
+ using MemoryPool::Reallocate;
+
+ Status Allocate(int64_t size, int64_t alignment, uint8_t** out) override {
+ const auto attempted = size + wrapped_->bytes_allocated();
+ if (ARROW_PREDICT_FALSE(attempted > bytes_allocated_limit_)) {
+ return OutOfMemory(attempted);
+ }
Review Comment:
The MemoryPool stats (`bytes_allocated`) use atomic ops. But it's true that
the check as written does not fully guarantee that the stat will stay below the
limit.
I honestly don't know how to solve this in a simple and efficient way, but I
should add a comment about the issue.
##########
cpp/src/arrow/memory_pool.h:
##########
@@ -245,6 +246,64 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+/// EXPERIMENTAL MemoryPool wrapper with an upper limit
+class ARROW_EXPORT CappedMemoryPool : public MemoryPool {
+ public:
+ CappedMemoryPool(MemoryPool* wrapped_pool, int64_t bytes_allocated_limit)
+ : wrapped_(wrapped_pool), bytes_allocated_limit_(bytes_allocated_limit)
{}
+
+ using MemoryPool::Allocate;
+ using MemoryPool::Reallocate;
+
+ Status Allocate(int64_t size, int64_t alignment, uint8_t** out) override {
+ const auto attempted = size + wrapped_->bytes_allocated();
+ if (ARROW_PREDICT_FALSE(attempted > bytes_allocated_limit_)) {
+ return OutOfMemory(attempted);
Review Comment:
With the limitation that it probably won't account for all memory
allocations (for example STL structures used to represent metadata).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
