gengliangwang commented on PR #46947: URL: https://github.com/apache/spark/pull/46947#issuecomment-2163943962
@pan3793 I appreciate the feedback and concerns raised regarding the migration of variables in our new structured logging framework for Apache Spark. I'd like to address these concerns and provide some clarity on our approach. ## Why Migrate All Variables? The primary reason for migrating all variables in our log messages is to avoid ongoing debates about which specific keys should be included as log keys. By disallowing the inlining of variables in log messages and ensuring all variables are explicitly named and migrated, we achieve a consistent and comprehensive logging structure. This approach ensures uniformity across all log entries. ## Current Status of Migration The migration process is nearly complete, which means we've already invested significant effort into ensuring that our log entries adhere to the new framework. Moving forward, as long as we follow the new logging rules, we maintain control over all the variables in our logs. This control is crucial for effective log analysis and troubleshooting. ## Managing Variable Overload To address concerns about the potential overload of variables for users, we have a couple of solutions: * Configuration Allow List: We can build a configuration option to pass an allow list of log keys, enabling users to specify which variables should appear in their logs. * Partitioned Log Keys: We can partition the log keys so that only essential ones appear by default, while others show up in verbose logging mode. This way, users can choose the level of detail they want to see based on their needs. Our goal with these changes is to enhance the clarity, consistency, and utility of our logs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
