First of all thanks for raising this ticket and important discussion Alexander! See answers inline below:
> It's a tiny bit unusual to turn on debug logging for all users by default > though, and there should be occasions to turn it on when facing issues that > you want to debug (if they can be easily reproduced). > This is not supposed to be "debugging" log in the strict sense of the word used for development purposes, but rather a "verbose system log" - which is verbose enough to difficult human reading of "system.log" but not enough to affect performance and could aid in troubleshooting. Perhaps this is not so clear so we should definitely improve communication in that front for committers and reviewers, but this has been very useful to troubleshoot and investigate hard to reproduce issues in production environments. Advanced operators can still disable it if they have automated log analysis, but it's useful to the beginner operator to understand system.log and "escalate" to debug.log if/when they are facing problems. > There's a danger that contributors might add debug loggings in the future > though, with the idea that it comes for free since the appender is > asynchronous (I would think so), and create new performance issues. > It's up to the committer and reviewer to ensure the log statements do not affect performance, as per our review guidelines (http://cassandra.apache.org/doc/latest/development/how_to_review.html#review-checklist): Logging - Are logging statements logged at the correct level? - Are there logs in the critical path that could affect performance? - Is there any log that could be added to communicate status or troubleshoot potential problems in this feature? - Can any unnecessary logging statement be removed? This is a subjective area and we can definitely try to quantify this and make this guideline more explicit if we start seeing more problems like this in the future. > Can someone assign the ticket to me? Assigned! > > Thanks! > > Le dim. 18 mars 2018 à 04:33, Paulo Motta <pauloricard...@gmail.com> a > écrit : > >> The reasoning to have debug.log enabled by default after >> CASSANDRA-10241 is to have a "safety log" that you can use when you >> want to troubleshoot issues after the fact, that is verbose enough >> that you can reconstruct events that may have caused a problem, but >> should have negligible performance impact. The way we found to >> implement this was the following: >> - INFO: Human-readable operator log. >> - DEBUG: Safety-log to be used for troubleshooting and bug reporting. >> - TRACE: C* Programmer Debugging. >> >> If debug.log is causing performance problems we should definitely fix >> it, but this has been very useful to troubleshoot complex production >> issues since it has been added. >> >> 2018-03-17 16:35 GMT-03:00 Michael Kjellman <kjell...@apple.com>: >> > ive never understood this change. and it’s been explained to me multiple >> times. >> > >> > DEBUG shouldn’t run by default in prod. and it certainly shouldn’t be >> enabled by default for users. >> > >> > but hey, what do i know! just my 2 cents. >> > >> >> On Mar 17, 2018, at 10:55 AM, J. D. Jordan <jeremiah.jor...@gmail.com> >> wrote: >> >> >> >> We went through an exercise of setting things up so that DEBUG logging >> was asynchronous would give people a “production” debug log. >> https://issues.apache.org/jira/browse/CASSANDRA-10241 >> >> If there are some things going out at DEBUG that cause performance >> issues then most likely those should be moved to TRACE so that debug >> logging can stay enabled for all the useful information found there. >> >> >> >> -Jeremiah >> >> >> >>> On Mar 17, 2018, at 1:49 PM, Alexander Dejanovski < >> a...@thelastpickle.com> wrote: >> >>> >> >>> Hi folks, >> >>> >> >>> we've been upgrading clusters from 2.0 to 2.2 recently and we've >> noticed >> >>> that debug logging was causing serious performance issues in some >> cases, >> >>> specifically because of its use in the query pager. >> >>> >> >>> I've opened a ticket with some benchmarks and flame graphs : >> >>> https://issues.apache.org/jira/browse/CASSANDRA-14318 >> >>> >> >>> The problem should be less serious in the read path with Cassandra 3.0 >> and >> >>> above as the query pager code has been reworked and doesn't log at >> debug >> >>> level. >> >>> I think that debug logging shouldn't be turned on by default though, >> since >> >>> we see it doesn't come for free and that it lowers read performance in >> 2.2. >> >>> >> >>> Was there any specific reason why it was enabled by default in 2.2 ? >> >>> >> >>> Is anyone opposed to disabling debug logging by default in all >> branches ? >> >>> >> >>> -- >> >>> ----------------- >> >>> Alexander Dejanovski >> >>> France >> >>> @alexanderdeja >> >>> >> >>> Consultant >> >>> Apache Cassandra Consulting >> >>> http://www.thelastpickle.com >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> > For additional commands, e-mail: dev-h...@cassandra.apache.org >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> >> -- > ----------------- > Alexander Dejanovski > France > @alexanderdeja > > Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
