Fair points. Kafka doesn't really have slow queries. I was thinking about this kind of log in response to a request processing slowdown we had during an internal release.. it's unlikely a slow query log would have really helped since it slowed down requests from all entities (see KAFKA-2664 for more).
I suppose one example of a "slow" query is produce with "ack=-1" in case the replicas aren't catching up. However, we do have other metrics to catch this. Aditya On Thu, Oct 15, 2015 at 12:43 AM, Ewen Cheslack-Postava <e...@confluent.io> wrote: > Kafka doesn't have the same type of queries that RDBMS systems have. What > "slow queries" would we be trying to capture info about? > > -Ewen > > On Wed, Oct 14, 2015 at 4:27 PM, Gwen Shapira <g...@confluent.io> wrote: > > > I had some experience with the feature in MySQL. > > > > Its main good use is to identify queries that are obviously bad (full > scans > > on OLTP system) and need optimization. You can't infer from it anything > > about the system as a whole because it lacks context and information > about > > what the rest of the system was doing at the same time. > > > > I'd like to hear how you see yourself using it in Apache Kafka to better > > understand its usefulness. Can you share some details about how you would > > have used it in the recent issue you mentioned? > > > > What I see as helpful: > > 1. Ability to enable/disable trace/debug level logging of request > handling > > for specific request types and clients without restarting the broker > (i.e. > > through JMX, protocol or ZK) > > 2. Publish histograms of the existing request time metrics > > 3. Capture detailed timing of a random sample of the requests and log it > > (i.e sample metrics rather than avgs). Note that clients that send more > > requests and longer requests are more likely to get sampled. I've found > > this super useful in the past. > > > > Gwen > > > > On Wed, Oct 14, 2015 at 3:39 PM, Aditya Auradkar < > > aaurad...@linkedin.com.invalid> wrote: > > > > > Hey everyone, > > > > > > We were recently discussing a small logging improvement for Kafka. > > > Basically, add a request log for queries that took longer than a > certain > > > configurable time to execute. This can be quite useful for debugging > > > purposes, in fact it would have proven handy while investigating a > recent > > > issue during one of our deployments at LinkedIn. > > > > > > There is also supported in several other projects. For example: MySQL > and > > > Postgres both have slow request logs. > > > https://dev.mysql.com/doc/refman/5.0/en/slow-query-log.html > > > https://wiki.postgresql.org/wiki/Logging_Difficult_Queries > > > > > > Thoughts? > > > > > > Thanks, > > > Aditya > > > > > > > > > -- > Thanks, > Ewen >
