This is only focusing on slices.
That is not going to work out well, because you should be able to decrease the ref count of a pointer to something arbitrary (void) without knowing the object type. And the ref count object should persist after destruction of the object in the case of weak pointers.
