Re: Debug logging enabled by default since 2.2

Ariel can we move discussion to the ticket now that we have one open? That way we can keep all thoughts in one place. Thanks. https://issues.apache.org/jira/browse/CASSANDRA-14326 > On Mar 20, 2018, at 12:42 PM, Ariel Weisberg wrote: > > Hi, > > That's good to hear. > >

Re: Debug logging enabled by default since 2.2

That's a good question. At this point ¯\_(ツ)_/¯ On Tue, Mar 20, 2018 at 4:42 PM Ariel Weisberg wrote: > Hi, > > That's good to hear. > > What's the difference between DEBUG and TRACE? Obviously we decide > ourselves. I don't have a good answer because right now we are in the

Re: Debug logging enabled by default since 2.2

Hi, That's good to hear. What's the difference between DEBUG and TRACE? Obviously we decide ourselves. I don't have a good answer because right now we are in the process of eliminating the distinction we used to make which is that DEBUG is safe to turn on in production and TRACE is not.

Re: Debug logging enabled by default since 2.2

Ariel, the current plan that's discussed on the ticket ( https://issues.apache.org/jira/browse/CASSANDRA-14326) is to maintain the separation and keep the debug.log for "real" DEBUG level logging, which would be disabled by default. A new intermediate marker would be created to have VERBOSE_INFO

Re: Debug logging enabled by default since 2.2

Hi, Signal to noise ratio matters for logs. Things that we log at DEBUG aren't at all bound by constraints of human readability or being particularly relevant most of the time. I don't want to see most of this stuff unless I have already not found what I am looking for at INFO and above. Can

Re: Debug logging enabled by default since 2.2

That sounds like a good plan, Alexander! Thanks! Stefan, someone needs to go through all messages being logged at DEBUG and reclassify important ones as INFO. I suggest continuing discussion on specifics on CASSANDRA-14326. 2018-03-20 6:46 GMT-03:00 Stefan Podkowinski : > Are

Re: Debug logging enabled by default since 2.2

Are you suggesting to move all messages currently logged via debug() to info() with the additional marker set, or only particular messages? On 19.03.2018 19:51, Paulo Motta wrote: > Thanks for the constructive input and feedback! From this discussion > it seems like overloading the DEBUG level

Re: Debug logging enabled by default since 2.2

Hi Paulo, I agree that CASSANDRA-10241 prevents from simply turning off debug logging and there's more work to do (we need compaction and flush traces among others). My 2 cents on the other points : - I was happy with a single log file for 99.5% of my ops needs and verbosity was never an

Re: Debug logging enabled by default since 2.2

> For this reason I'm more in favor of converting the debug.log into async/verbose_system.log as suggested by Jeremiah and use a marker to direct these logs (former DEBUG level logs) to that log instead. +1 Thanks for the inclusive and considerate summary Paulo, and for the suggestion that

Re: Debug logging enabled by default since 2.2

People seem hung up on DEBUG here. The goal of CASSANDRA-10241 was to clean up the system.log so that it a very high “signal” in terms of what was logged to it synchronously, but without reducing the ability of the logs to allow people to solve problems and perform post mortem analysis of

Re: Debug logging enabled by default since 2.2

I'd agree that INFO should be the default. Turning on the DEBUG logging can cause notable performance issues and I would not enable it on production systems unless I really have to. That's why I created  CASSANDRA-12696 for 4.0, so you'll be able to at least only partially enable DEBUG based on

Re: Debug logging enabled by default since 2.2

> > In a way the real issue might be that we don’t have nightly performance > runs that would make an accidentally introduced debug statement obvious. First off, very much this. Secondly, IMO it's inconsistent to leave / use assertions in a code-base as a mix of preconditions and assertions but

Re: Debug logging enabled by default since 2.2

Hello, I am not developing Cassandra, but I am using it actively and helping people to work with it. My perspective might be missing some code considerations and history as I did not go through the ticket where this 'debug' level was added by default. But here is a feedback after upgrading a few

RE: Debug logging enabled by default since 2.2

: Re: Debug logging enabled by default since 2.2 On the same page as Michael here. We disable debug logs in production due to the performance impact. Personally I think if debug logging is necessary for users to use the software we're doing something wrong. Also in my experience, if something

Re: Debug logging enabled by default since 2.2

On the same page as Michael here. We disable debug logs in production due to the performance impact. Personally I think if debug logging is necessary for users to use the software we're doing something wrong. Also in my experience, if something doesn't reproduce it will not get fixed. Debug

Re: Debug logging enabled by default since 2.2

Hi, In a way the real issue might be that we don’t have nightly performance runs that would make an accidentally introduced debug statement obvious. A log statement that runs once or more per read or write should be easy to spot. I haven’t measured the impact though. And as a bonus by having

Re: Debug logging enabled by default since 2.2

In Cassandra-10241 I said I was torn on this whole ticket, since most people would end up turning it off if it had a negative impact. You said: “I'd like to emphasize that we're not talking about turning debug or trace on for client-generated request paths. There's way too much data generated

Re: Debug logging enabled by default since 2.2

It seems to me the power user who wants to eek out that last bit of performance can do so. It also seems like the user who doesn't even know about that is the exact kind of user that's going to need the debug log in the future. On Sun, Mar 18, 2018 at 1:45 PM, Michael Kjellman

Re: Debug logging enabled by default since 2.2

First of all thanks for raising this ticket and important discussion Alexander! See answers inline below: > It's a tiny bit unusual to turn on debug logging for all users by default > though, and there should be occasions to turn it on when facing issues that > you want to debug (if they can be

Re: Debug logging enabled by default since 2.2

i’m not trying to get into a fight here jeremiah. and this will be my last reply on this as i’ve made my opinion pretty clear. but ask yourself: would you run c* in idea debugger and then do performance testing? no. because it’s a DEBUGger. > On Mar 18, 2018, at 11:43 AM, J. D. Jordan

Re: Debug logging enabled by default since 2.2

If there are some log messages you think should be improved to make them more useful please do so. Saying things are “crap” is not productive. I have seen having the extra information from the debug.log be very helpful in debugging production issues after the fact on operational clusters many

Re: Debug logging enabled by default since 2.2

what really baffles me with this entire thing is as a project we don’t even log things like partition keys along with the tombstone overwhelming or batch to large log messages.. this would immediately be helpful to thousands and thousands of people... yet somehow we think it’s okay to log tons

Re: Debug logging enabled by default since 2.2

it’s too easy to make a regression there. and does anyone even have a splunk (or equivalent) infrastructure to actually keep debug logs around for a long enough retention period to even have them be helpful? again: this is something engineers for the project want. it’s not in the best interest

Re: Debug logging enabled by default since 2.2

That really depends on whether you're judicious in deciding what to log at debug, doesn't it? On Sun, Mar 18, 2018 at 12:57 PM, Michael Kjellman wrote: > +1. this is how it works. > > your computer doesn’t run at debug logging by default. your phone doesn’t > either. neither

Re: Debug logging enabled by default since 2.2

+1. this is how it works. your computer doesn’t run at debug logging by default. your phone doesn’t either. neither does your smart tv. your database can’t be running at debug just because it makes our lives as engineers easier. > On Mar 18, 2018, at 5:14 AM, Alexander Dejanovski

Re: Debug logging enabled by default since 2.2

Hi Paulo, I understand the intent and Jeremiah suggested on the ticket that logging causing performance issues should be switched to TRACE level. It's a tiny bit unusual to turn on debug logging for all users by default though, and there should be occasions to turn it on when facing issues that

Re: Debug logging enabled by default since 2.2

The reasoning to have debug.log enabled by default after CASSANDRA-10241 is to have a "safety log" that you can use when you want to troubleshoot issues after the fact, that is verbose enough that you can reconstruct events that may have caused a problem, but should have negligible performance

Re: Debug logging enabled by default since 2.2

ive never understood this change. and it’s been explained to me multiple times. DEBUG shouldn’t run by default in prod. and it certainly shouldn’t be enabled by default for users. but hey, what do i know! just my 2 cents. > On Mar 17, 2018, at 10:55 AM, J. D. Jordan

Re: Debug logging enabled by default since 2.2

We went through an exercise of setting things up so that DEBUG logging was asynchronous would give people a “production” debug log. https://issues.apache.org/jira/browse/CASSANDRA-10241 If there are some things going out at DEBUG that cause performance issues then most likely those should be

Debug logging enabled by default since 2.2

Hi folks, we've been upgrading clusters from 2.0 to 2.2 recently and we've noticed that debug logging was causing serious performance issues in some cases, specifically because of its use in the query pager. I've opened a ticket with some benchmarks and flame graphs :