techdocsmith commented on a change in pull request #12363:
URL: https://github.com/apache/druid/pull/12363#discussion_r836763347
##########
File path: docs/configuration/index.md
##########
@@ -279,24 +280,24 @@ For SQL query request, the `native_query` field is empty.
Example
2019-01-14T10:00:00.000Z 127.0.0.1
{"sqlQuery/time":100,"sqlQuery/bytes":600,"success":true,"identity":"user1"}
{"query":"SELECT page, COUNT(*) AS Edits FROM wikiticker WHERE __time BETWEEN
TIMESTAMP '2015-09-12 00:00:00' AND TIMESTAMP '2015-09-13 00:00:00' GROUP BY
page ORDER BY Edits DESC LIMIT
10","context":{"sqlQueryId":"c9d035a0-5ffd-4a79-a865-3ffdadbb5fdd","nativeQueryIds":"[490978e4-f5c7-4cf6-b174-346e63cf8863]"}}
```
-#### Emitter Request Logging
+#### Emitter request logging
-Every request is emitted to some external location.
+The `emitter` request logger emits every request to some external location,
which is configured through an [emitter](#enabling-metrics).
|Property|Description|Default|
|--------|-----------|-------|
|`druid.request.logging.feed`|Feed name for requests.|none|
-#### SLF4J Request Logging
+#### SLF4J request logging
-Every request is logged via SLF4J. Native queries are serialized into JSON in
the log message regardless of the SLF4J format specification. They will be
logged under the class `org.apache.druid.server.log.LoggingRequestLogger`.
+The `slf4j` request logger logs every request via SLF4J. Native queries are
serialized into JSON in the log message regardless of the SLF4J format
specification. Requests are logged under the class
`org.apache.druid.server.log.LoggingRequestLogger`.
|Property|Description|Default|
|--------|-----------|-------|
-|`druid.request.logging.setMDC`|If MDC entries should be set in the log entry.
Your logging setup still has to be configured to handle MDC to format this
data|false|
-|`druid.request.logging.setContextMDC`|If the druid query `context` should be
added to the MDC entries. Has no effect unless `setMDC` is `true`|false|
+|`druid.request.logging.setMDC`|If MDC entries should be set in the log entry.
Your logging setup still has to be configured to handle MDC to format this
data.|false|
Review comment:
```suggestion
|`druid.request.logging.setMDC`|If you want to set MDC entries within the
log entry, set this value to "true" to handle the MDC data format.|false|
```
not !00% sure what this was trying to say, verify before accepting change.
##########
File path: docs/configuration/index.md
##########
@@ -279,24 +280,24 @@ For SQL query request, the `native_query` field is empty.
Example
2019-01-14T10:00:00.000Z 127.0.0.1
{"sqlQuery/time":100,"sqlQuery/bytes":600,"success":true,"identity":"user1"}
{"query":"SELECT page, COUNT(*) AS Edits FROM wikiticker WHERE __time BETWEEN
TIMESTAMP '2015-09-12 00:00:00' AND TIMESTAMP '2015-09-13 00:00:00' GROUP BY
page ORDER BY Edits DESC LIMIT
10","context":{"sqlQueryId":"c9d035a0-5ffd-4a79-a865-3ffdadbb5fdd","nativeQueryIds":"[490978e4-f5c7-4cf6-b174-346e63cf8863]"}}
```
-#### Emitter Request Logging
+#### Emitter request logging
-Every request is emitted to some external location.
+The `emitter` request logger emits every request to some external location,
which is configured through an [emitter](#enabling-metrics).
Review comment:
```suggestion
The `emitter` request logger emits every request to the external location
specified in the [emitter](#enabling-metrics) configuration.
```
##########
File path: docs/configuration/index.md
##########
@@ -279,24 +280,24 @@ For SQL query request, the `native_query` field is empty.
Example
2019-01-14T10:00:00.000Z 127.0.0.1
{"sqlQuery/time":100,"sqlQuery/bytes":600,"success":true,"identity":"user1"}
{"query":"SELECT page, COUNT(*) AS Edits FROM wikiticker WHERE __time BETWEEN
TIMESTAMP '2015-09-12 00:00:00' AND TIMESTAMP '2015-09-13 00:00:00' GROUP BY
page ORDER BY Edits DESC LIMIT
10","context":{"sqlQueryId":"c9d035a0-5ffd-4a79-a865-3ffdadbb5fdd","nativeQueryIds":"[490978e4-f5c7-4cf6-b174-346e63cf8863]"}}
```
-#### Emitter Request Logging
+#### Emitter request logging
-Every request is emitted to some external location.
+The `emitter` request logger emits every request to some external location,
which is configured through an [emitter](#enabling-metrics).
|Property|Description|Default|
|--------|-----------|-------|
|`druid.request.logging.feed`|Feed name for requests.|none|
-#### SLF4J Request Logging
+#### SLF4J request logging
-Every request is logged via SLF4J. Native queries are serialized into JSON in
the log message regardless of the SLF4J format specification. They will be
logged under the class `org.apache.druid.server.log.LoggingRequestLogger`.
+The `slf4j` request logger logs every request via SLF4J. Native queries are
serialized into JSON in the log message regardless of the SLF4J format
specification. Requests are logged under the class
`org.apache.druid.server.log.LoggingRequestLogger`.
|Property|Description|Default|
|--------|-----------|-------|
-|`druid.request.logging.setMDC`|If MDC entries should be set in the log entry.
Your logging setup still has to be configured to handle MDC to format this
data|false|
-|`druid.request.logging.setContextMDC`|If the druid query `context` should be
added to the MDC entries. Has no effect unless `setMDC` is `true`|false|
+|`druid.request.logging.setMDC`|If MDC entries should be set in the log entry.
Your logging setup still has to be configured to handle MDC to format this
data.|false|
+|`druid.request.logging.setContextMDC`|If the Druid query `context` should be
added to the MDC entries. Has no effect unless `setMDC` is `true`.|false|
Review comment:
```suggestion
|`druid.request.logging.setContextMDC`|Set to "true" to add the Druid query
`context` to the MDC entries. Only applies when `setMDC` is `true`.|false|
```
##########
File path: docs/operations/metrics.md
##########
@@ -72,11 +69,11 @@ Available Metrics
|`segment/scan/pending`|Number of segments in queue waiting to be
scanned.||Close to 0|
|`query/segmentAndCache/time`|Milliseconds taken to query individual segment
or hit the cache (if it is enabled on the Historical process).|id,
segment.|several hundred milliseconds|
|`query/cpu/time`|Microseconds of CPU time taken to complete a query|Common:
dataSource, type, interval, hasFilters, duration, context, remoteAddress, id.
Aggregation Queries: numMetrics, numComplexMetrics. GroupBy: numDimensions.
TopN: threshold, dimension.|Varies|
-|`query/count`|number of total queries|This metric is only available if the
QueryCountStatsMonitor module is included.||
-|`query/success/count`|number of queries successfully processed|This metric is
only available if the QueryCountStatsMonitor module is included.||
-|`query/failed/count`|number of failed queries|This metric is only available
if the QueryCountStatsMonitor module is included.||
-|`query/interrupted/count`|number of queries interrupted due to
cancellation.|This metric is only available if the QueryCountStatsMonitor
module is included.||
-|`query/timeout/count`|number of timed out queries.|This metric is only
available if the QueryCountStatsMonitor module is included.||
+|`query/count`|Number of total queries|This metric is only available if the
QueryCountStatsMonitor module is included.||
Review comment:
Number of total reads awkwardly is this the "total number of queries"?
##########
File path: docs/operations/request-logging.md
##########
@@ -0,0 +1,249 @@
+---
+id: request-logging
+title: Request logging
+sidebar_label: Request logging
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+All Apache Druid services that can serve queries can also log the query
requests they see.
+Request logs contain information on query metrics, including execution time
and memory usage.
+You can use information in the request logs to monitor query performance,
determine bottlenecks, and analyze and improve slow queries.
+
+Request logging is disabled by default.
+This topic describes how to configure Druid to generate request logs to track
query metrics.
+
+## Configure request logging
+
+To enable request logging, determine the type of request logger to use, then
set the configurations specific to the request logger type.
+
+The following request logger types are available:
+
+- `noop`: Disables request logging, the default behavior.
+- [`file`](../configuration/index.md#file-request-logging): Stores logs to
disk.
+- [`emitter`](../configuration/index.md#emitter-request-logging): Logs request
to an external location, which is configured through an emitter.
+- [`slf4j`](../configuration/index.md#slf4j-request-logging): Logs queries via
the SLF4J Java logging API.
+- [`filtered`](../configuration/index.md#filtered-request-logging): Filters
requests by query type or execution time before logging the filtered queries by
the delegated request logger.
+- [`composing`](../configuration/index.md#composing-request-logging): Logs all
requests to multiple request loggers.
+- [`switching`](../configuration/index.md#switching-request-logging): Logs
native queries and SQL queries to separate request loggers.
+
+Define the type of request logger in `druid.request.logging.type`.
+See the [Request logging
configuration](../configuration/index.md#request-logging) for properties to set
for each type of request logger.
+Specify these properties in the `common.runtime.properties` file.
+You must restart Druid for the changes to take effect.
+
+Druid stores the results in the Broker logs, unless the request logging type
is `emitter`.
+If you use emitter request logging, you must also configure metrics emission.
+
+## Configure metrics emission
+
+Druid emits metrics and alerts via an emitter.
Review comment:
```suggestion
Druid includes various emitters to send metrics and alerts.
```
##########
File path: docs/configuration/index.md
##########
@@ -310,31 +311,36 @@ For native query, the following MDC fields are populated
with `setMDC`:
|`resultOrdering`|The ordering of results|
|`descending`|If the query is a descending query|
-#### Filtered Request Logging
-Filtered Request Logger filters requests based on a configurable query/time
threshold (for native query) and sqlQuery/time threshold (for SQL query).
-For native query, only request logs where query/time is above the threshold
are emitted. For SQL query, only request logs where sqlQuery/time is above the
threshold are emitted.
+#### Filtered request logging
+
+The `filtered` request logger filters requests based on the query type or how
long a query takes to complete.
+For native queries, requests are only logged when the `query/time` metric is
above the user-provided threshold.
Review comment:
```suggestion
For native queries, the logger only logs requests when the `query/time`
metric exceeds the user-provided threshold.
```
##########
File path: docs/configuration/index.md
##########
@@ -310,31 +311,36 @@ For native query, the following MDC fields are populated
with `setMDC`:
|`resultOrdering`|The ordering of results|
|`descending`|If the query is a descending query|
-#### Filtered Request Logging
-Filtered Request Logger filters requests based on a configurable query/time
threshold (for native query) and sqlQuery/time threshold (for SQL query).
-For native query, only request logs where query/time is above the threshold
are emitted. For SQL query, only request logs where sqlQuery/time is above the
threshold are emitted.
+#### Filtered request logging
+
+The `filtered` request logger filters requests based on the query type or how
long a query takes to complete.
+For native queries, requests are only logged when the `query/time` metric is
above the user-provided threshold.
+For SQL queries, requests are only logged when the `sqlQuery/time` metric is
above the user-provided threshold.
Review comment:
```suggestion
For SQL queries, it only logs requests when the `sqlQuery/time` metric
exceeds the user-provided threshold.
```
for line 317 too, is this a specific threshold we can mention. For example
when sqlQuery/time exceeds the threshold set in <config>
##########
File path: docs/configuration/index.md
##########
@@ -454,7 +465,7 @@ The additional configs are:
##### Graphite Emitter
-To use graphite as emitter set `druid.emitter=graphite`. For configuration
details please follow this
[link](../development/extensions-contrib/graphite.md).
+To use graphite as emitter set `druid.emitter=graphite`. For configuration
details, see the [documentation](../development/extensions-contrib/graphite.md)
for the Graphite emitter Druid extension.
Review comment:
```suggestion
To use graphite as emitter set `druid.emitter=graphite`. For configuration
details, see [Graphite emitter](../development/extensions-contrib/graphite.md)
for the Graphite emitter Druid extension.
```
##########
File path: docs/configuration/index.md
##########
@@ -279,24 +280,24 @@ For SQL query request, the `native_query` field is empty.
Example
2019-01-14T10:00:00.000Z 127.0.0.1
{"sqlQuery/time":100,"sqlQuery/bytes":600,"success":true,"identity":"user1"}
{"query":"SELECT page, COUNT(*) AS Edits FROM wikiticker WHERE __time BETWEEN
TIMESTAMP '2015-09-12 00:00:00' AND TIMESTAMP '2015-09-13 00:00:00' GROUP BY
page ORDER BY Edits DESC LIMIT
10","context":{"sqlQueryId":"c9d035a0-5ffd-4a79-a865-3ffdadbb5fdd","nativeQueryIds":"[490978e4-f5c7-4cf6-b174-346e63cf8863]"}}
```
-#### Emitter Request Logging
+#### Emitter request logging
-Every request is emitted to some external location.
+The `emitter` request logger emits every request to some external location,
which is configured through an [emitter](#enabling-metrics).
|Property|Description|Default|
|--------|-----------|-------|
|`druid.request.logging.feed`|Feed name for requests.|none|
-#### SLF4J Request Logging
+#### SLF4J request logging
-Every request is logged via SLF4J. Native queries are serialized into JSON in
the log message regardless of the SLF4J format specification. They will be
logged under the class `org.apache.druid.server.log.LoggingRequestLogger`.
+The `slf4j` request logger logs every request via SLF4J. Native queries are
serialized into JSON in the log message regardless of the SLF4J format
specification. Requests are logged under the class
`org.apache.druid.server.log.LoggingRequestLogger`.
Review comment:
```suggestion
The `slf4j` request logger logs every request using SLF4J. It serializes
native queries into JSON in the log message regardless of the SLF4J format
specification. Requests are logged under the class
`org.apache.druid.server.log.LoggingRequestLogger`.
```
##########
File path: docs/operations/request-logging.md
##########
@@ -0,0 +1,249 @@
+---
+id: request-logging
+title: Request logging
+sidebar_label: Request logging
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+All Apache Druid services that can serve queries can also log the query
requests they see.
Review comment:
```suggestion
All Apache Druid services that can serve queries can also log the query
requests they process.
```
Not sure how sight figures in.
##########
File path: docs/operations/request-logging.md
##########
@@ -0,0 +1,249 @@
+---
+id: request-logging
+title: Request logging
+sidebar_label: Request logging
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+All Apache Druid services that can serve queries can also log the query
requests they see.
+Request logs contain information on query metrics, including execution time
and memory usage.
+You can use information in the request logs to monitor query performance,
determine bottlenecks, and analyze and improve slow queries.
+
+Request logging is disabled by default.
+This topic describes how to configure Druid to generate request logs to track
query metrics.
+
+## Configure request logging
+
+To enable request logging, determine the type of request logger to use, then
set the configurations specific to the request logger type.
+
+The following request logger types are available:
+
+- `noop`: Disables request logging, the default behavior.
+- [`file`](../configuration/index.md#file-request-logging): Stores logs to
disk.
+- [`emitter`](../configuration/index.md#emitter-request-logging): Logs request
to an external location, which is configured through an emitter.
+- [`slf4j`](../configuration/index.md#slf4j-request-logging): Logs queries via
the SLF4J Java logging API.
+- [`filtered`](../configuration/index.md#filtered-request-logging): Filters
requests by query type or execution time before logging the filtered queries by
the delegated request logger.
+- [`composing`](../configuration/index.md#composing-request-logging): Logs all
requests to multiple request loggers.
+- [`switching`](../configuration/index.md#switching-request-logging): Logs
native queries and SQL queries to separate request loggers.
+
+Define the type of request logger in `druid.request.logging.type`.
+See the [Request logging
configuration](../configuration/index.md#request-logging) for properties to set
for each type of request logger.
+Specify these properties in the `common.runtime.properties` file.
+You must restart Druid for the changes to take effect.
+
+Druid stores the results in the Broker logs, unless the request logging type
is `emitter`.
+If you use emitter request logging, you must also configure metrics emission.
+
+## Configure metrics emission
+
+Druid emits metrics and alerts via an emitter.
+To emit query metrics, set `druid.request.logging.feed=emitter`, and define
the emitter type in the `druid.emitter` property.
+
+You can use any of the following emitters in Druid:
+
+- `noop`: Disables metric emission, the default behavior.
+- [`logging`](../configuration/index.md#logging-emitter-module): Emits metrics
to Log4j 2. See [Logging](../configuration/logging.md) to configure Log4j 2 for
use with Druid.
+- [`http`](../configuration/index.md#http-emitter-module): Sends HTTP `POST`
requests containing the metrics in JSON format to a user-defined endpoint.
+-
[`parametrized`](../configuration/index.md#parametrized-http-emitter-module):
Operates like the `http` emitter but fine-tunes the recipient URL based on the
event feed.
+- [`composing`](../configuration/index.md#composing-emitter-module): Emits
metrics to multiple emitter types.
+- [`graphite`](../configuration/index.md#graphite-emitter): Emits metrics to a
[Graphite](https://graphiteapp.org/) Carbon service.
+
+Specify these properties in the `common.runtime.properties` file.
+See the [Metrics emitters
configuration](../configuration/index.md#metrics-emitters) for properties to
set for each type of metrics emitter.
+You must restart Druid for the changes to take effect.
+
+
+## Example
+
+The following configuration shows how to enable request logging and post query
metrics to the endpoint `http://example.com:8080/path`.
+```
+# Enable request logging and configure the emitter request logger
+druid.request.logging.type=emitter
+druid.request.logging.feed=myRequestLogFeed
+
+# Enable metrics emission and tell Druid where to emit messages
+druid.emitter=http
+druid.emitter.http.recipientBaseUrl=http://example.com:8080/path
+
+# Authenticate to the base URL, if needed
+druid.emitter.http.basicAuthentication=username:password
+```
+
+The following output shows an example of the emitted information:
Review comment:
```suggestion
The following shows an example log emitter output:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
