wesm commented on PR #13364: URL: https://github.com/apache/arrow/pull/13364#issuecomment-1156491212
> I think the best solution for performance is to remove shared_ptr to data types and have a canonical owner. I'm straining to see how it to implement this in a safe way. With the way the library is designed, if DataType instances were stored in some central place, absent some form of reference counting mechanism, it would be basically impossible to know if it is ever safe to garbage collect a data type instance. So one proposal would be something like: * Central type metadata registry where every data type has an associated atomic "reference count" * Any non-trivial DataType (e.g. if its definition is more complex than just a `Type::type`) has a back-reference to the central metadata registry, where deallocation decrements the refcount in the metadata registry. Trivial types (e.g. Int64Type) do nothing * The metadata registry performs a garbage collection pass every so often I can visualize how to implement this but it would be a truly chaotic refactor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
