jorisvandenbossche commented on code in PR #14516:
URL: https://github.com/apache/arrow/pull/14516#discussion_r1005433867
##########
python/pyarrow/tests/test_dataset.py:
##########
@@ -491,6 +491,23 @@ def test_scanner(dataset, dataset_reader):
assert sorted_table['__last_in_fragment'].to_pylist() == [True] * 10
[email protected]
+def test_scanner_memory_pool(dataset):
+ # honor default pool - https://issues.apache.org/jira/browse/ARROW-18164
+ old_pool = pa.default_memory_pool()
+ # pool = pa.proxy_memory_pool(old_pool)
+ pool = pa.system_memory_pool()
Review Comment:
It seems that using the proxy memory pool with datasets crashes. I can't
reproduce this in an interactive session, but the following test segfaults:
```
@pytest.mark.parquet
def test_scanner_proxy_memory_pool(dataset):
proxy_pool = pa.proxy_memory_pool(pa.default_memory_pool())
_ = dataset.to_table(memory_pool=proxy_pool)
```
gdb backtrace:
```
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff37bec9e in arrow::PoolBuffer::~PoolBuffer (this=0x7fff9c001f30,
__in_chrg=<optimized out>) at
/home/joris/scipy/repos/arrow/cpp/src/arrow/memory_pool.cc:817
817 pool_->Free(ptr, capacity_);
(gdb) bt
#0 0x00007ffff37bec9e in arrow::PoolBuffer::~PoolBuffer
(this=0x7fff9c001f30, __in_chrg=<optimized out>) at
/home/joris/scipy/repos/arrow/cpp/src/arrow/memory_pool.cc:817
#1 0x00007ffff37becc8 in arrow::PoolBuffer::~PoolBuffer
(this=0x7fff9c001f30, __in_chrg=<optimized out>) at
/home/joris/scipy/repos/arrow/cpp/src/arrow/memory_pool.cc:819
#2 0x00007ffff5c2545a in
std::default_delete<arrow::ResizableBuffer>::operator() (this=0x7fff9c001fa0,
__ptr=0x7fff9c001f30)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
