[
https://issues.apache.org/jira/browse/CASSANDRA-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613331#comment-17613331
]
Stefan Miklosovic commented on CASSANDRA-17948:
-----------------------------------------------
Thanks for the field report, [~MichielSA]. It is always interesting to read
about the issues users get into so we can make it easier for them. There are
multiple issues I want to go through as your comment mixes few things together.
Firstly, if we are already logging the information about what the large
partitions are but then you are forced to go through the logs to actually see
what they are so you can block them via "denylist" mechanism, correct? If that
is true, we should expose this information about large partitions directly into
some cql virtual table so you do not need to go through them. Parsing logs so
you can deny it is quite uncomfortable anyway, is not it?
Secondly, why aret we automatically denying partitions when they are detected
to be large? Ideally, you should not do anything. If Cassandra evaluates that
the partition you are going to insert to is too large, why we have to deny it
manually? Why does not Cassandra block that partition in denylist automatically
for us so you do not have to deal with all this stuff in the first place? Of
course, sometimes you do want to insert regardless of the size of partition
even they are bigger than recommended because you just do not have any other
option, but automatic denying might be configurable per table so if it occurs,
that partition will be automatically denylisted.
When it comes to logs, I think coding a custom slf4j appender is the most
natural thing to implement. It is super easy. One caveat is that if we are
selecting from that virtual table, the rows you get should be in reversed
order. Basically, the newest log messages are last in the file but they should
be first in the table because we can not expect people to "scroll down" to the
end of cql output every time to see new messages. They should be at the very
top.
When it comes reading a file, we should then read it in a reversed order.
Unfortunately, there is (1) which is able to do that and we depend on
apache-commons already so we do not need to add any new dependency. The problem
with this approach is - what should be the primary key? Parsing the timestamp
from a line of log message seems to be error-prone because users might have
different formatting of timestamps set in logger configuration (message
format).
My idea of doing this to use CircularFifoBuffer (2) - we again already depend
on this library. The basic idea behind this is that buffer would have a
constant size, e.g. 10k lines, messages would be added on top and if the buffer
is full, the oldest message would be dropped so new one can be inserted. By
doing this, we would have constant size of the vtable and the log messages
would be "rolled up" automatically. With compressing the messages, I think the
cost of having 10k of messages in memory is peanuts. Even 100k is just fine.
What do you think [~rustyrazorblade]? [~brandon.williams]
(1)
https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/input/ReversedLinesFileReader.html
(2)
https://commons.apache.org/proper/commons-collections/javadocs/api-3.2.2/org/apache/commons/collections/buffer/CircularFifoBuffer.html
> Get warning and errors through virtual tables
> ---------------------------------------------
>
> Key: CASSANDRA-17948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17948
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Michiel Saelen
> Priority: Normal
>
> The warnings and errors are currently only accessible through Cassandra logs.
> Automating the monitoring of the nodes would be much easier/secure if we can
> make use of virtual tables to get the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
