This is an automated email from the ASF dual-hosted git repository.
zeroshade pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new 71321841eb GH-40900: [Go] Fix Mallocator Weirdness (#40902)
71321841eb is described below
commit 71321841eb6d94946de43cccb7f04afe5cf2aa10
Author: Matt Topol <[email protected]>
AuthorDate: Mon Apr 1 11:15:59 2024 -0400
GH-40900: [Go] Fix Mallocator Weirdness (#40902)
### Rationale for this change
With help from @ lidavidm and @ bkietz digging into the linked issue, we
found the following:
* Using `mtrace` and `strace` didn't produce much enlightenment to what was
happening.
* If the python adbc_driver_manager was built so that the cython lib is
built using `CMAKE_BUILD_TYPE=Debug` then the crash/failure goes away
* If the env var `MALLOC_MMAP_THRESHOLD_` is set to 128MB, the
crash/failure goes away
* It is only reproducible when calling through python, I haven't been able
to reproduce it using pure Go
* Calling `calloc` again after it fails, still fails
* Calling `malloc` + `memset` immediately after the failing `calloc` works
perfectly and doesn't fail anymore
### What changes are included in this PR?
Adding a comment describing the situation and falling back to `malloc` +
`memset` if `calloc` returns an error. If the pointer returned from `malloc` is
`nil` then we surface the error.
* GitHub Issue: #40900
Authored-by: Matt Topol <[email protected]>
Signed-off-by: Matt Topol <[email protected]>
---
go/arrow/memory/mallocator/mallocator.go | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/go/arrow/memory/mallocator/mallocator.go
b/go/arrow/memory/mallocator/mallocator.go
index 59d240a106..9483bdfc2a 100644
--- a/go/arrow/memory/mallocator/mallocator.go
+++ b/go/arrow/memory/mallocator/mallocator.go
@@ -60,10 +60,19 @@ func (alloc *Mallocator) Allocate(size int) []byte {
}
ptr, err := C.calloc(C.size_t(size), 1)
if err != nil {
- panic(err)
+ // under some circumstances and allocation patterns, we can end
up in a scenario
+ // where for some reason calloc return ENOMEM even though there
is definitely memory
+ // available for use. So we attempt to fallback to simply doing
malloc + memset in
+ // this case. If malloc returns a nil pointer, then we know
we're out of memory
+ // and will surface the error.
+ if ptr = C.malloc(C.size_t(size)); ptr == nil {
+ panic(err)
+ }
+ C.memset(ptr, 0, C.size_t(size))
} else if ptr == nil {
panic("mallocator: out of memory")
}
+
atomic.AddUint64(&alloc.allocatedBytes, uint64(size))
return unsafe.Slice((*byte)(ptr), size)
}
