Ariel can we move discussion to the ticket now that we have one open? That way
we can keep all thoughts in one place. Thanks.
> On Mar 20, 2018, at 12:42 PM, Ariel Weisberg wrote:
> That's good to hear.
That's a good question.
At this point ¯\_(ツ)_/¯
On Tue, Mar 20, 2018 at 4:42 PM Ariel Weisberg wrote:
> That's good to hear.
> What's the difference between DEBUG and TRACE? Obviously we decide
> ourselves. I don't have a good answer because right now we are in the
That's good to hear.
What's the difference between DEBUG and TRACE? Obviously we decide ourselves. I
don't have a good answer because right now we are in the process of eliminating
the distinction we used to make which is that DEBUG is safe to turn on in
production and TRACE is not.
the current plan that's discussed on the ticket (
https://issues.apache.org/jira/browse/CASSANDRA-14326) is to maintain the
separation and keep the debug.log for "real" DEBUG level logging, which
would be disabled by default.
A new intermediate marker would be created to have VERBOSE_INFO
Another point I want to mention. Another reason we separate the files is that
they can then use separate log rolling policies. A verbose debug log no longer
rolls the INFO level system.log.
On Tue, Mar 20, 2018, at 12:19 PM, Ariel Weisberg wrote:
> Signal to noise ratio
Signal to noise ratio matters for logs. Things that we log at DEBUG aren't at
all bound by constraints of human readability or being particularly relevant
most of the time. I don't want to see most of this stuff unless I have already
not found what I am looking for at INFO and above.
That sounds like a good plan, Alexander! Thanks!
Stefan, someone needs to go through all messages being logged at DEBUG
and reclassify important ones as INFO. I suggest continuing discussion
on specifics on CASSANDRA-14326.
2018-03-20 6:46 GMT-03:00 Stefan Podkowinski :
Are you suggesting to move all messages currently logged via debug() to
info() with the additional marker set, or only particular messages?
On 19.03.2018 19:51, Paulo Motta wrote:
> Thanks for the constructive input and feedback! From this discussion
> it seems like overloading the DEBUG level
I agree that CASSANDRA-10241 prevents from simply turning off debug logging
and there's more work to do (we need compaction and flush traces among
My 2 cents on the other points :
- I was happy with a single log file for 99.5% of my ops needs and
verbosity was never an
> For this reason I'm more in favor of converting the debug.log into
async/verbose_system.log as suggested by Jeremiah and use a marker to
direct these logs (former DEBUG level logs) to that log instead.
Thanks for the inclusive and considerate summary Paulo, and for the
Thanks for the constructive input and feedback! From this discussion
it seems like overloading the DEBUG level to signify
async-verbose-INFO on CASSANDRA-10241 is leading to some confusion and
we should fix this.
However, we cannot simply turn debug.log off as during CASSANDRA-10241
People seem hung up on DEBUG here. The goal of CASSANDRA-10241 was
to clean up the system.log so that it a very high “signal” in terms of what was
to it synchronously, but without reducing the ability of the logs to allow
solve problems and perform post mortem analysis of
I'd agree that INFO should be the default. Turning on the DEBUG logging
can cause notable performance issues and I would not enable it on
production systems unless I really have to. That's why I created
CASSANDRA-12696 for 4.0, so you'll be able to at least only partially
enable DEBUG based on
> In a way the real issue might be that we don’t have nightly performance
> runs that would make an accidentally introduced debug statement obvious.
First off, very much this.
Secondly, IMO it's inconsistent to leave / use assertions in a code-base as
a mix of preconditions and assertions but
I am not developing Cassandra, but I am using it actively and helping
people to work with it. My perspective might be missing some code
considerations and history as I did not go through the ticket where this
'debug' level was added by default. But here is a feedback after upgrading
: Re: Debug logging enabled by default since 2.2
On the same page as Michael here. We disable debug logs in production due to
the performance impact. Personally I think if debug logging is necessary for
users to use the software we're doing something wrong. Also in my experience,
On the same page as Michael here. We disable debug logs in production due
to the performance impact. Personally I think if debug logging is necessary
for users to use the software we're doing something wrong. Also in my
experience, if something doesn't reproduce it will not get fixed. Debug
In a way the real issue might be that we don’t have nightly performance runs
that would make an accidentally introduced debug statement obvious.
A log statement that runs once or more per read or write should be easy to
spot. I haven’t measured the impact though. And as a bonus by having
In Cassandra-10241 I said I was torn on this whole ticket, since most people
would end up turning it off if it had a negative impact. You said:
“I'd like to emphasize that we're not talking about turning debug or trace on
for client-generated request paths. There's way too much data generated
It seems to me the power user who wants to eek out that last bit of
performance can do so. It also seems like the user who doesn't even know
about that is the exact kind of user that's going to need the debug log in
On Sun, Mar 18, 2018 at 1:45 PM, Michael Kjellman
First of all thanks for raising this ticket and important discussion
Alexander! See answers inline below:
> It's a tiny bit unusual to turn on debug logging for all users by default
> though, and there should be occasions to turn it on when facing issues that
> you want to debug (if they can be
i’m not trying to get into a fight here jeremiah. and this will be my last
reply on this as i’ve made my opinion pretty clear. but ask yourself: would you
run c* in idea debugger and then do performance testing? no. because it’s a
> On Mar 18, 2018, at 11:43 AM, J. D. Jordan
If there are some log messages you think should be improved to make them more
useful please do so. Saying things are “crap” is not productive.
I have seen having the extra information from the debug.log be very helpful in
debugging production issues after the fact on operational clusters many
what really baffles me with this entire thing is as a project we don’t even log
things like partition keys along with the tombstone overwhelming or batch to
large log messages.. this would immediately be helpful to thousands and
thousands of people... yet somehow we think it’s okay to log tons
it’s too easy to make a regression there. and does anyone even have a splunk
(or equivalent) infrastructure to actually keep debug logs around for a long
enough retention period to even have them be helpful?
again: this is something engineers for the project want. it’s not in the best
That really depends on whether you're judicious in deciding what to log at
debug, doesn't it?
On Sun, Mar 18, 2018 at 12:57 PM, Michael Kjellman
> +1. this is how it works.
> your computer doesn’t run at debug logging by default. your phone doesn’t
> either. neither
+1. this is how it works.
your computer doesn’t run at debug logging by default. your phone doesn’t
either. neither does your smart tv. your database can’t be running at debug
just because it makes our lives as engineers easier.
> On Mar 18, 2018, at 5:14 AM, Alexander Dejanovski
I understand the intent and Jeremiah suggested on the ticket that logging
causing performance issues should be switched to TRACE level.
It's a tiny bit unusual to turn on debug logging for all users by default
though, and there should be occasions to turn it on when facing issues that
The reasoning to have debug.log enabled by default after
CASSANDRA-10241 is to have a "safety log" that you can use when you
want to troubleshoot issues after the fact, that is verbose enough
that you can reconstruct events that may have caused a problem, but
should have negligible performance
ive never understood this change. and it’s been explained to me multiple times.
DEBUG shouldn’t run by default in prod. and it certainly shouldn’t be enabled
by default for users.
but hey, what do i know! just my 2 cents.
> On Mar 17, 2018, at 10:55 AM, J. D. Jordan
We went through an exercise of setting things up so that DEBUG logging was
asynchronous would give people a “production” debug log.
If there are some things going out at DEBUG that cause performance issues then
most likely those should be
we've been upgrading clusters from 2.0 to 2.2 recently and we've noticed
that debug logging was causing serious performance issues in some cases,
specifically because of its use in the query pager.
I've opened a ticket with some benchmarks and flame graphs :
Mail list logo