It does not look like jl_array_del_end currently frees any memory https://github.com/JuliaLang/julia/blob/0dacb65c0587eab268362c4dece1b2c662bef15e/src/array.c#L616
I think you could gain something by storing pointers or object ids directly in a typed array, and avoid the safety features and garbage collector traverses that reduces performance.
