walterddr opened a new issue #7652: URL: https://github.com/apache/pinot/issues/7652
Background === Currently Pinot doesn't have a way to store system metadata (other than looking through the logs) This makes it super hard to debug, for example when I debug with @dongxiaoman in issue #7612 . It is super hard to figure out exactly which segment causes the AIOOB exception other than doing a log search. Pitch === I was wondering if we can create some sort of system logging mechanism such as: https://dev.mysql.com/doc/refman/5.7/en/server-logs.html. Draft Design Ideas === comparing with the mysql server logs. we can start with several subcategory of system metadata tables such as - server error logs ( indexed by server ID / segment ID ?) - query logs (controller / broker / server, indexed by requestID ?) - minion task logs (taskID?) I am not sure if it is a good idea to add another step to write to a system metadata table in broker/server/controller/minion, especially if it potentially delays the query response time or minion task runtime. I was wondering if we can create a pinot-metadata SPI similar to pinot-metrics SPI so that we can easily plug in different metadata store backends and metadata writer (agent-based, direct push, JDBC, ... ) Thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
