pzhdfy commented on PR #65883: URL: https://github.com/apache/doris/pull/65883#issuecomment-5088185311
> @pzhdfy Thank you for the contibution! Just to clarify, paimon-cpp has already been donated from the Alibaba repository [Alibaba repo](https://github.com/alibaba/paimon-cpp) to the [Apache repo](https://github.com/apache/paimon-cpp) and is currently in the release process. It is now the official Paimon standard-format implementation under the Apache community, and we expect it to continue tracking the upstream standard as the Paimon community evolves. So, I’m not sure “bespoke C++ port” is a accurate wording here. Also, paimon-cpp has been widely used in production within Alibaba for about 2 years, with both stability and performance validated in large-scale scenarios. Given that Doris’s backend is implemented in C++, I wonder whether the C++ path might be a more natural fit to prioritize here. 1. paimon-rust is maintained by paimon core team（who also maintains paimon-java), for example the paimon PMC @JingsongLi. However paimon-cpp is maintained by another team in Alibaba 2. paimon-cpp(opensource version) has poor performance，when doing merge-on-read or predicate pushdown paimon-cpp spends 4x~6x time than paimon-java. - paimon-cpp use arrow parquet, which has poor read performance, and predicate pushdown is not well implemented. - paimon-cpp merge on read is not well implemented in opensource version - all above is base on opensource version. we do not know what is different in paimon-cpp Alibaba internal implemention 3.paimon-rust is 2x~3x faster than paimon-java when read MOR ,and close to doris native reader when read DV（ only slow 10%，because arrow convert to doris block） - rust native parquet reader has nice performance - better Vectorization and predicate pushdown we has a wechat team ( together with doris and paimon community），which has more context <img width="693" height="403" alt="image" src="https://github.com/user-attachments/assets/dadbe461-d79a-4a0b-bed0-8c6667b8beca"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
