GGraziadei commented on PR #16827: URL: https://github.com/apache/iceberg/pull/16827#issuecomment-5361944128
Hi @gimgit, Thanks for your question helped to improve drastically the quality of this work. I would like to start from point 2. I measured and compared the memory allocation with a profiler, and I reported the results here: https://github.com/apache/iceberg/pull/16827#issuecomment-5361125612 (Claude helped me put together a quick report). While conducting this analysis, I was already aware that the main cost was the transpose operation and that it was not buffered. I then buffered the transpose, which significantly improved memory allocation: from 1.7 GB/op before buffering to only ~6.7 KB/op afterward as for ZOrder. The improvement is impressive, thanks a lot for focusing on this point! Regarding point 1, I want to clarify something. I really appreciate your PR effort here, because this is exactly what allows us to track the quality of the clustering strategy. The real metric for evaluating cluster quality is locality. In my mind, this can be understood as the variance in density across clusters. This is where I see the real advantage of Hilbert over Z-order. So your question is absolutely correct: how do we trigger this? I will open an issue and address it in a separate PR once this one is merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
