HeartSaVioR edited a comment on issue #23776: [SPARK-26650][core] Demote noisy HBase-related log message. URL: https://github.com/apache/spark/pull/23776#issuecomment-463430775 > And now you get an exception that is a couple of pages long every time you start Spark against a secure cluster, whether you're using HBase or not. That's worse. In my case the query crashed after 7 days because HBase token was not renewed due to classpath issue (the issue was in AM side so driver/executor worked for these days) but surprisingly further error messages make me hard to guess there was classpath issue on HBase token provider. Without enabling debug log level I wouldn't find the root reason. But I also agree there're more users to leverage HDFS so my change affected more users to be annoyed. Agree to roll back the log level: let's leave the reason of log level set to debug so no one would try to promote it again. > Unless you're saying this should be moved to HBase or something external that is not part of the Spark distribution at all. To be honest, once Spark doesn't support HBase in main repo and token provider is dynamically loadable as of now (I guess it didn't), I'm not 100% sure this is the right way to have HBase token provider in Spark repo/distribution, with dealing with reflection because we don't know which HBase version end users will link. So basically I agree that should not be a part of Spark distribution: HBase connector can have it as of Spark 3.0.0. Don't have idea how Spark community has been dealing with ecosystem though.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
