Z-SWEI opened a new issue, #59889: URL: https://github.com/apache/doris/issues/59889
High FE metadata memory usage with large number of tablets – any plan to further slim FE metadata in Doris 3.0 Cloud? We’ve noticed relatively high memory usage on the FE side caused by metadata, especially in clusters with a large number of tablets. As the cluster scales, FE memory pressure becomes quite noticeable. After a brief analysis, it seems that the main factor is the tablet count: Each tablet maintains certain metadata structures in FE With many tables / partitions / replicas, the total number of tablets grows rapidly Metadata-related objects take an increasing portion of FE JVM memory We’d like to ask the community a few questions: In Doris 3.0 Cloud mode, are there any plans to further slim down or refactor FE metadata, especially tablet-related metadata? Will the Cloud architecture introduce mechanisms such as metadata offloading, layering, or lazy loading to reduce FE resident memory usage? If there is already an optimization direction, is there a rough timeline or version plan available? We understand that FE needs to keep global metadata for consistency. Our goal here is mainly to understand the future optimization roadmap for large-scale tablet scenarios, so we can better plan our architecture and capacity. Thanks in advance for your insights and discussion 🙏 FE 元数据内存占用较高，tablet 数量多场景下压力明显，3.0 Cloud 模式是否有进一步瘦身计划？ 目前在使用 Doris 的过程中，发现 FE 端元数据占用内存较高，在集群规模和 tablet 数量较大的情况下，FE 内存压力比较明显。FULL GC 会导致 BDB 集群崩溃。 我们简单分析了一下内存构成，发现主要与 tablet 数量偏多 有关： 每个 tablet 在 FE 中都会维护一定量的元数据信息 当表数量 / 分区数 / 副本数较多时，tablet 总数快速膨胀 FE JVM 内存中元数据对象占比持续上升 想请教下社区： 在 Doris 3.0 Cloud 模式 下，是否有针对 FE 元数据（尤其是 tablet 相关元数据）进一步瘦身或重构 的规划？ 是否会通过 Cloud 架构（如元数据分层、下沉、惰性加载等）来降低 FE 常驻内存占用？ 如果已经有相关优化方向，是否方便透露一个 大致排期或版本规划？ 我们理解 FE 需要维护全局元数据来保证一致性，这里更多是想了解 未来在大规模 tablet 场景下的优化路线，以便提前做架构和容量规划。 感谢社区的解答与讨论 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
