abreis edited a comment on issue #9420: URL: https://github.com/apache/arrow/issues/9420#issuecomment-774810331
The missing insight here is that cloning a `MemTable` is actually cheap. `MemTable`s hold `RecordBatch`es, which themselves are built from `ArrayRef`s (i.e., `Arc<dyn Array>`). This seems to be an anti-pattern though, so much so that `MemTable` doesn't implement `Clone`. You'd probably want to store `RecordBatch`es in your `Library` instead, and build `MemTable`s from them whenever you want to add them to a context (`RecordBatch`es are `Clone`). This also means that adding record batches to the context is cheap, and that the context doesn't own those batches. If you want to refresh data in the context, however, you'll need to unregister the table and reregister it with the new data. If you don't do this, when you hold your batches behind an `RwLock` and replace them, the context will actually keep an `Arc` pointed to the old data and prevent it from being deallocated. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
