gaogaotiantian commented on PR #58429: URL: https://github.com/apache/spark/pull/58429#issuecomment-5627169678
> [It pins all loggers at WARN](https://github.com/apache/spark/blob/master/python/pyspark/logger/logger.py#L144), which would prevent us from controlling logger levels hierarchically. Probably hard to change now without breaking backward compatibility? It does not "pin" loggers at `WARN`, it initializes them at `WARN`. You can easily change them with `setLevel`. > [It always attaches a handler](https://github.com/apache/spark/blob/1b8ef5fa621e4959063626ecc1ad328206c0d61f/python/pyspark/logger/logger.py#L156) -- so similar issue: if we have a hierarchy of loggers, a message would get logged N times, once at each level. I think this behavior is just wrong. We should not by default add a handler for each logger - we should add the handler on root logger. The only place we used this in our code base, we set `logger.propagate = False`. I think it's safe to change the current behavior - it's a relatively new feature and it's for logging. We should make the behavior reasonable. We don't need to backport it but we can fix it in `branch-4.x` and `master`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
