Hey Hudi friends, I’ve heard several of you wrestle with choosing Hudi configs, building metrics/observability for Hudi pipelines, or sometimes just gathering logs for debugging. I am working on early designs for a lightweight, free Hudi tool that can give out-of-the box metrics, dashboards, logs, and debug aids for your Hudi pipelines. It may help you observe the basics of your pipeline from things like records written, bytes/commit, partition skew distribution, clean frequency, write operation failures, and many more details about your writers and table services.
If this topic is interesting, or if you struggle with monitoring and tuning your Hudi pipelines, I would love to learn about your needs and brainstorm the designs with you. I want to explore if we can build something that is useful for the whole community so your opinions and ideas are valuable no matter if you are brand new or a long time Hudi expert! Use any of these 3 ways to chime in: 1. Reply on this thread and I will reach out to you for a chat. 2. Chime in on Hudi Slack, I cross-posted there 3. If you don’t like talking to people, but want to share some ideas/feedback, use this survey link: https://forms.gle/uYLr8dMaLjfxxey69 <https://forms.gle/uYLr8dMaLjfxxey69> Thanks, Kyle Weller
