This is an automated email from the ASF dual-hosted git repository.
mitchell852 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficcontrol.git
The following commit(s) were added to refs/heads/master by this push:
new f2aeec7 Rewrote deliveryservice_stats to Go (#3758)
f2aeec7 is described below
commit f2aeec7a92d8e88ec2df7114e8ad5e1ef2d63b98
Author: ocket8888 <[email protected]>
AuthorDate: Mon Oct 14 07:45:32 2019 -0600
Rewrote deliveryservice_stats to Go (#3758)
* Rewrote deliveryservice_stats to Go
* Fixed api.CreateInfluxClient always attempting to use SSL
* Added descriptions of the totalBytes and totalTransactions fields
* fixed influx client builder considering servers that are not ONLINE
* increased traffic_stats log verbosity
* Better error reporting
* added appropriate deprecation notices
* Fixed config key to conform to standard behavior
* fixed GoDoc comments
* Added sane default for 'Secure' flag to influx configs
* Moved MimeType into a dedicated RFC package/library
* returns now explicit
* fixed rfc test compilation
* fixed more implicit returns, match return type of db helper funcs
* s/Equal/Satisfy/g
* match snake_case for configuration variables
* consistent casing for initialisms
* use Ptr util funcs
* var/const blocks
* Move config object into lib/go-tc
* lowercase local variables
* fixed compilation errors introduced by earlier changes
* Split some functionality out into utilities
* Collapse conditionals
* Made checking for excludes more similar to checking for orderables
* All internal errors now straight-up 500s, details to client omitted
* Add test for config parsing
* Switched back to a regular expression for intervals
* Column name value->sum_count
* Added missing struct tag
* Fix erroneous warning log message
---
CHANGELOG.md | 1 +
docs/source/api/deliveryservice_stats.rst | 284 +++++-----
infrastructure/cdn-in-a-box/traffic_ops/config.sh | 11 +-
infrastructure/cdn-in-a-box/traffic_ops/run-go.sh | 2 +-
infrastructure/cdn-in-a-box/traffic_stats/run.sh | 9 +
lib/go-rfc/mimetype.go | 259 +++++++++
lib/go-rfc/mimetype_test.go | 359 +++++++++++++
lib/go-tc/traffic_stats.go | 247 +++++++++
traffic_ops/app/conf/development/influxdb.conf | 3 +-
traffic_ops/app/conf/integration/influxdb.conf | 3 +-
traffic_ops/app/conf/production/influxdb.conf | 3 +-
traffic_ops/app/conf/test/influxdb.conf | 3 +-
traffic_ops/traffic_ops_golang/api/api.go | 73 +++
traffic_ops/traffic_ops_golang/config/config.go | 81 ++-
traffic_ops/traffic_ops_golang/routing/routes.go | 4 +
.../traffic_ops_golang/traffic_ops_golang.go | 6 +-
.../trafficstats/trafficstats.go | 595 +++++++++++++++++++++
.../trafficstats/trafficstats_test.go | 106 ++++
.../src/common/api/DeliveryServiceStatsService.js | 6 +-
19 files changed, 1904 insertions(+), 151 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f4b27f2..cb57aa8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -26,6 +26,7 @@ The format is based on [Keep a
Changelog](http://keepachangelog.com/en/1.0.0/).
- /api/1.1/servers/:name/configfiles/ats/parent.config
- /api/1.1/servers/:name/configfiles/ats/remap.config
- /api/1.1/user/login/token `POST`
+ - /api/1.4/deliveryservice_stats `GET`
- To support reusing a single riak cluster connection, an optional parameter
is added to riak.conf: "HealthCheckInterval". This options takes a 'Duration'
value (ie: 10s, 5m) which affects how often the riak cluster is health checked.
Default is currently set to: "HealthCheckInterval": "5s".
- Added a new Go db/admin binary to replace the Perl db/admin.pl script which
is now deprecated and will be removed in a future release. The new db/admin
binary is essentially a drop-in replacement for db/admin.pl since it supports
all of the same commands and options; therefore, it should be used in place of
db/admin.pl for all the same tasks.
diff --git a/docs/source/api/deliveryservice_stats.rst
b/docs/source/api/deliveryservice_stats.rst
index 1bb265b..f920da1 100644
--- a/docs/source/api/deliveryservice_stats.rst
+++ b/docs/source/api/deliveryservice_stats.rst
@@ -19,12 +19,8 @@
*************************
``deliveryservice_stats``
*************************
-.. caution:: This page is a stub! Much of it may be missing or just downright
wrong - it needs a lot of love from people with the domain knowledge required
to update it.
-
.. versionadded:: 1.2
-.. warning:: This endpoint does **NOT** respect tenancy permissions! The bug
is tracked by `GitHub Issue #3187
<https://github.com/apache/trafficcontrol/issues/3187>`_.
-
``GET``
=======
Retrieves time-aggregated statistics on a specific :term:`Delivery Service`.
@@ -37,140 +33,168 @@ Request Structure
-----------------
.. table:: Request Query Parameters
-
+----------------------+----------+--------------------------------------------------------------------------------------------------------------------+
- | Name | Required | Description
|
-
+======================+==========+====================================================================================================================+
- | deliveryServiceName | yes | The name of the :term:`Delivery
Service` for which statistics will be aggregated
|
-
+----------------------+----------+--------------------------------------------------------------------------------------------------------------------+
- | metricType | yes | The metric type being reported -
one of:
|
- | | |
|
- | | | kbps
|
- | | | The total traffic rate in
kilobytes per second served by the :term:`Delivery Service`
|
- | | | out_bytes
|
- | | | The total number of bytes sent
out to clients through the :term:`Delivery Service`
|
- | | | status_4xx
|
- | | | The amount of requests that were
serviced with 400-499 HTTP status codes
|
- | | | status_5xx
|
- | | | The amount of requests that were
serviced with 500-599 HTTP status codes
|
- | | | tps_total
|
- | | | The total traffic rate in
transactions per second served by the :term:`Delivery Service`
|
- | | | tps_2xx
|
- | | | The total traffic rate in
transactions per second serviced with 200-299 HTTP status codes
|
- | | | tps_3xx
|
- | | | The total traffic rate in
transactions per second serviced with 300-399 HTTP status codes
|
- | | | tps_4xx
|
- | | | The total traffic rate in
transactions per second serviced with 400-499 HTTP status codes
|
- | | | tps_5xx
|
- | | | The total traffic rate in
transactions per second serviced with 500-599 HTTP status codes
|
- | | |
|
-
+----------------------+----------+--------------------------------------------------------------------------------------------------------------------+
- | startDate | yes | The date and time from which
statistics shall be aggregated in ISO8601 format, e.g.
``2018-08-11T12:30:00-07:00`` |
-
+----------------------+----------+--------------------------------------------------------------------------------------------------------------------+
- | endDate | yes | The date and time until which
statistics shall be aggregated in ISO8601 format, e.g.
``2018-08-12T12:30:00-07:00`` |
-
+----------------------+----------+--------------------------------------------------------------------------------------------------------------------+
+
+---------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | Name | Required | Description
|
+
+=====================+===================+===========================================================================================================================================================================================+
+ | deliveryService | yes\ [#ds-param]_ | Either the :ref:`ds-xmlid`
of a :term:`Delivery Service` for which statistics will be aggregated or the
integral, unique identifier of said :term:`Delivery Service`
|
+
+---------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | deliveryServiceName | yes\ [#ds-param]_ | The :ref:`ds-xmlid` of the
:term:`Delivery Service` for which statistics will be aggregated
|
+
+---------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | endDate | yes | The date and time until
which statistics shall be aggregated in :rfc:`3339` format (with or without
sub-second precision), the number of nanoseconds since the Unix
|
+ | | | Epoch, or in the same,
proprietary format as the ``lastUpdated`` fields prevalent throughout the
Traffic Ops API
|
+
+---------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | exclude | no | Either "series" to omit the
data series from the result, or "summary" to omit the summary data from the
result - directly corresponds to fields in the
|
+ | | | `Response Structure`_
|
+
+---------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | interval | no | Specifies the interval
within which data will be "bucketed"; e.g. when requesting data from
2019-07-25T00:00:00Z to 2019-07-25T23:59:59Z with an interval of "1m",
|
+ | | | the resulting data series
(assuming it is not excluded) should contain
|
+ | | |
:math:`24\frac{\mathrm{hours}}{\mathrm{day}}\times60\frac{\mathrm{minutes}}{\mathrm{hour}}\times1\mathrm{day}\times1\frac{\mathrm{minute}}{\mathrm{data
point}}=1440\mathrm{data\;points}`|
+ | | | The allowed values for this
parameter are valid InfluxQL duration literal strings matching
:regexp:`^\d+[mhdw]$`
|
+ | | |
|
+
+---------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | limit | no | A natural number indicating
the maximum amount of data points should be returned in the ``series`` object
|
+
+---------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | metricType | yes | The metric type being
reported - one of:
|
+ | | |
|
+ | | | kbps
|
+ | | | The total traffic rate in
kilobytes per second served by the :term:`Delivery Service`
|
+ | | | out_bytes
|
+ | | | The total number of bytes
sent out to clients through the :term:`Delivery Service`
|
+ | | | status_4xx
|
+ | | | The amount of requests
that were serviced with 400-499 HTTP status codes
|
+ | | | status_5xx
|
+ | | | The amount of requests
that were serviced with 500-599 HTTP status codes
|
+ | | | tps_total
|
+ | | | The total traffic rate in
transactions per second served by the :term:`Delivery Service`
|
+ | | | tps_2xx
|
+ | | | The total traffic rate in
transactions per second serviced with 200-299 HTTP status codes
|
+ | | | tps_3xx
|
+ | | | The total traffic rate in
transactions per second serviced with 300-399 HTTP status codes
|
+ | | | tps_4xx
|
+ | | | The total traffic rate in
transactions per second serviced with 400-499 HTTP status codes
|
+ | | | tps_5xx
|
+ | | | The total traffic rate in
transactions per second serviced with 500-599 HTTP status codes
|
+ | | |
|
+
+---------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | offset | no | A natural number of data
points to drop from the beginning of the returned data set
|
+
+---------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | orderby | no | Though one struggles to
imagine why, this can be used to specify "time" to sort data points by their
"time" (which is the default behavior), or "value" to sort them by their
"value" |
+
+---------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | startDate | yes | The date and time from
which statistics shall be aggregated in :rfc:`3339` format (with or without
sub-second precision), the number of nanoseconds since the Unix
|
+ | | | Epoch, or in the same,
proprietary format as the ``lastUpdated`` fields prevalent throughout the
Traffic Ops API
|
+
+---------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+.. _deliveryservice_stats-get-request-example:
+.. code-block:: http
+ :caption: Request Example
+
+ GET
/api/1.3/deliveryservice_stats?deliveryServiceName=demo1&startDate=2019-07-22T17:55:00Z&endDate=2019-07-22T17:56:00.000Z&metricType=tps_total
HTTP/1.1
+ User-Agent: python-requests/2.20.1
+ Accept-Encoding: gzip, deflate
+ Accept: application/json;timestamp=unix,
application/json;timestamp=rfc;q=0.9, application/json;q=0.8, */*;q=0.7
+ Connection: keep-alive
+ Cookie: mojolicious=...
+
+Content Format
+""""""""""""""
+It's important to note in :ref:`deliveryservice_stats-get-request-example` the
use of a complex "Accept" header. This endpoint accepts two special media types
in the "Accept" header that instruct it on how to format the timestamps
associated with the returned data. Specifically, Traffic Ops will recognize the
special, optional, non-standard parameter of :mimetype:`application/json`:
``timestamp``. The values of this parameter are restricted to one of
+
+rfc
+ Returned timestamps will be formatted according to :rfc:`3339` (no
sub-second precision).
+unix
+ Returned timestamps will be formatted as the number of nanoseconds
since the Unix Epoch (midnight on January 1\ :sup:`st` 1970 UTC).
+
+ .. impl-detail:: The endpoint passes back nanoseconds, specifically,
because that is the form used both by InfluxDB, which is used to store the data
being served, and Go's standard library. Clients may need to convert the value
to match their own standard libraries - e.g. the :js:class:`Date` class in
Javascript expects milliseconds.
+
+The default behavior - when only e.g. :mimetype:`application/json` or
:mimetype:`*/*` is given - is to use :rfc:`3339` formatting. It will, however,
respect quality parameters. It is suggested that clients request timestamps
they can handle specifically, rather than relying on this default behavior, as
it **is subject to change** and is in fact **expected to invert in the next
major release** as string-based time formats become deprecated.
+
+.. seealso:: For more information on the "Accept" HTTP header, consult `its
dedicated page on MDN
<https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept>`_.
Response Structure
------------------
-.. table:: Response Keys
-
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- | Parameter | Type | Description
|
-
+============================+===============+=========================================================================================+
- |``source`` | string | The source of the data
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``summary`` | hash | Summary data
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>totalBytes`` | float |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>count`` | int |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>min`` | float |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>max`` | float |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>fifthPercentile`` | float |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>ninetyEighthPercentile`` | float |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>ninetyFifthPercentile`` | float |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>average`` | float |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>totalTransactions`` | int |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``series`` | hash | Series data
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>count`` | int |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>columns`` | array |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>name`` | string |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>values`` | array |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>>time`` | string |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
- |``>>value`` | float |
|
-
+----------------------------+---------------+-----------------------------------------------------------------------------------------+
-
-.. code-block:: json
+:series: An object containing the actual data series and information necessary
for working with it.
+
+ :columns: This is an array of names of the columns of the data
contained in the "values" array - should always be ``["time", "sum_count"]``
+ :count: The number of data points contained in the "values" array
+ :name: The name of the data set. Should always match
:samp:`{metric}.ds.1min` where ``metric`` is the requested ``metricType``
+ :values: The actual array of data points. Each represents a length of
time specified by the ``interval`` query parameter
+
+ :time: The time at which the measurement was taken. This
corresponds to the *beginning* of the interval. This time comes in the format
of either an :rfc:`3339`-formatted string, or a number containing the number of
nanoseconds since the Unix Epoch depending on the "Accept" header sent by the
client, according to the rules outlined in `Content Format`_.
+ :value: The value of the requested ``metricType`` at the time
given by ``time``. This will always be a floating point number, unless no data
is available for the data interval, in which case it will be ``null``
+
+:source: A legacy field meant only for plugins that override this endpoint to
name themselves. Should always be "TrafficStats".
+
+ .. deprecated:: 1.4
+ As this has no known purpose, developers are advised it will be
removed in the future.
+
+:summary: An object containing summary statistics describing the data series
+
+ :average: The arithmetic mean of the data's values
+ :count: The number of measurements taken within the
requested timespan. This is, in general, **not** the same as the ``count``
field of the ``series`` object, as it reflects the number of underlying,
un-"bucketed" data points, and is therefore dependent on the implementation of
Traffic Stats.
+ :fifthPercentile: Data points with values less than or equal to
this number constitute the "bottom" 5% of the data set
+ :max: The maximum value that can be found in the
requested data set
+ :min: The minimum value that can be found in the
requested data set
+ :ninetyEighthPercentile: Data points with values greater than or equal
to this number constitute the "top" 2% of the data set
+ :ninetyFifthPercentile: Data points with values greater than or equal
to this number constitute the "top" 5% of the data set
+ :totalBytes: When the ``metricType`` requested is ``kbps``,
this will contain the total number of bytes transferred by the :term:`Delivery
Service` within the requested time window. Note that fractional amounts are
possible, as the data transfer rate will almost certainly not be cleanly
divided by the requested time range.
+ :totalTransactions: When the ``metricType`` requested is **not**
``kbps``, this will contain the total number of transactions completed by the
:term:`Delivery Service` within the requested time window. Note that fractional
amounts are possible, as the transaction rate will almost certainly not be
cleanly divided by the requested time range.
+
+:version: A legacy field that seems to have been meant to indicate the API
version used. Will always be "1.2"
+
+ .. deprecated:: 1.4
+ As this has no known purpose, developers are advised it will be
removed in the future.
+
+.. code-block:: http
:caption: Response Example
+ HTTP/1.1 200 OK
+ Access-Control-Allow-Credentials: true
+ Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type,
Accept, Set-Cookie, Cookie
+ Access-Control-Allow-Methods: POST,GET,OPTIONS,PUT,DELETE
+ Access-Control-Allow-Origin: *
+ Content-Encoding: gzip
+ Content-Type: application/json
+ Set-Cookie: mojolicious=; Path=/; HttpOnly
+ Whole-Content-Sha512:
zXJGjcYuu6HxWINVp8HA1gL31J3ukry5wCsTDNxtP/waC6rSD8h10KJ9jEAtRzJ9owOSVPvKaA/2bRu/QeuCpQ==
+ X-Server-Name: traffic_ops_golang/
+ Date: Mon, 22 Jul 2019 17:57:14 GMT
+ Transfer-Encoding: chunked
+
{ "response": {
- "source": "TrafficStats",
- "summary": {
- "average": 1081172.785,
- "count": 28,
- "fifthPercentile": 888827.26,
- "max": 1326680.31,
- "min": 888827.26,
- "ninetyEighthPercentile": 1324785.47,
- "ninetyFifthPercentile": 1324785.47,
- "totalBytes": 37841047.475,
- "totalTransactions": 1020202030101
- },
"series": {
"columns": [
"time",
- ""
- ],
- "count": 60,
- "name": "kbps",
- "tags": {
- "cachegroup": "total"
- },
- "values": [
- [
- "2015-08-11T11:36:00Z",
- 888827.26
- ],
- [
- "2015-08-11T11:37:00Z",
- 980336.563333333
- ],
- [
- "2015-08-11T11:38:00Z",
- 952111.975
- ],
- [
- "2015-08-11T11:39:00Z",
- null
+ "sum_count"
],
- [
- "2015-08-11T11:43:00Z",
- null
- ],
- [
- "2015-08-11T11:44:00Z",
- 934682.943333333
- ],
- [
- "2015-08-11T11:45:00Z",
- 1251121.28
- ],
- [
- "2015-08-11T11:46:00Z",
- 1111012.99
+ "count": 2,
+ "name": "tps_total.ds.1min",
+ "tags": {
+ "cachegroup": "total"
+ },
+ "values": [
+ [
+ 1563818100000000000,
+ 0
+ ],
+ [
+ 1563818160000000000,
+ 0
+ ]
]
- ]
- }}}
+ },
+ "source": "TrafficStats",
+ "summary": {
+ "average": 0,
+ "count": 2,
+ "fifthPercentile": 0,
+ "max": 0,
+ "min": 0,
+ "ninetyEighthPercentile": 0,
+ "ninetyFifthPercentile": 0,
+ "totalBytes": null,
+ "totalTransactions": 0
+ },
+ "version": "1.2"
+ }}
+
+.. [#ds-param] Either ``deliveryServiceName`` or ``deliveryService`` *must* be
present, but if both are ``deliveryServiceName`` will be used and
``deliveryService`` will be ignored.
diff --git a/infrastructure/cdn-in-a-box/traffic_ops/config.sh
b/infrastructure/cdn-in-a-box/traffic_ops/config.sh
index ab0dc41..77b9f1c 100755
--- a/infrastructure/cdn-in-a-box/traffic_ops/config.sh
+++ b/infrastructure/cdn-in-a-box/traffic_ops/config.sh
@@ -128,7 +128,8 @@ cat <<-EOF >/opt/traffic_ops/app/conf/cdn.conf
"user" : "",
"password" : "",
"address" : ""
- }
+ },
+ "influxdb_conf_location":
"/opt/traffic_ops/app/conf/production/influx.conf"
}
EOF
@@ -162,3 +163,11 @@ cat <<-EOF >/opt/traffic_ops/app/conf/production/riak.conf
"password": "$TV_RIAK_PASSWORD"
}
EOF
+
+cat <<-EOF >/opt/traffic_ops/app/conf/production/influx.conf
+{
+ "user": "$INFLUXDB_ADMIN_USER",
+ "password": "$INFLUXDB_ADMIN_PASSWORD",
+ "secure": false
+}
+EOF
diff --git a/infrastructure/cdn-in-a-box/traffic_ops/run-go.sh
b/infrastructure/cdn-in-a-box/traffic_ops/run-go.sh
index fffe774..1b3d960 100755
--- a/infrastructure/cdn-in-a-box/traffic_ops/run-go.sh
+++ b/infrastructure/cdn-in-a-box/traffic_ops/run-go.sh
@@ -52,7 +52,7 @@ source /to-access.sh
. /config.sh
while ! nc "$TO_PERL_FQDN" $TO_PERL_PORT </dev/null 2>/dev/null; do
- echo "waiting for $TO_PERL_FQDN:$TO_PERL_PORT"
+ echo "waiting for $TO_PERL_FQDN:$TO_PERL_PORT"
sleep 3
done
diff --git a/infrastructure/cdn-in-a-box/traffic_stats/run.sh
b/infrastructure/cdn-in-a-box/traffic_stats/run.sh
index cb580f4..31374b2 100755
--- a/infrastructure/cdn-in-a-box/traffic_stats/run.sh
+++ b/infrastructure/cdn-in-a-box/traffic_stats/run.sh
@@ -90,6 +90,15 @@ cat <<-EOF >$TSCONF
}
EOF
+cat <<-EOF >/opt/traffic_stats/conf/traffic_stats_seelog.xml
+<?xml version='1.0'?>
+<seelog minlevel="debug">
+ <outputs formatid="std:debug-short">
+ <file
path="/opt/traffic_stats/var/log/traffic_stats/traffic_stats.log" />
+ </outputs>
+</seelog>
+EOF
+
touch /opt/traffic_stats/var/log/traffic_stats/traffic_stats.log
# Wait for influxdb
diff --git a/lib/go-rfc/mimetype.go b/lib/go-rfc/mimetype.go
new file mode 100644
index 0000000..ad9d056
--- /dev/null
+++ b/lib/go-rfc/mimetype.go
@@ -0,0 +1,259 @@
+package rfc
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import "mime"
+import "sort"
+import "strings"
+import "strconv"
+
+/*
+MimeType represents a "Media Type" as defined by RFC6838, along with some
ease-of-use functionality.
+
+Note that this structure is in no way guaranteed to represent a *real* MIME
Type, only one that
+is *syntactically valid*. The hope is that it will make content negotiation
easier for developers,
+and should not be considered a security measure by any standard.
+*/
+type MimeType struct {
+ // Name is the full name of the MIME Type, e.g. 'application/json'.
+ // Usually for printing, it's better to call MimeType.String
+ Name string
+ // Parameters contains a map of provided parameter names to
corresponding values. Note that for
+ // MimeTypes constructed with NewMimeType, this will always be
initialized, even when empty.
+ Parameters map[string]string
+}
+
+/*
+Quality retrieves and parses the "quality" parameter of a MimeType.
+
+As specified in RFC7231, the quality parameter's name is "q", not actually
"quality". To obtain
+a literal "quality" parameter value, access MimeType.Parameters directly.
+MimeTypes with no "q" parameter implicitly have a "quality" of 1.0.
+*/
+func (m MimeType) Quality() float64 {
+ if m.Parameters == nil {
+ return 1
+ }
+
+ fs, ok := m.Parameters["q"]
+ if !ok {
+ return 1
+ }
+
+ ret, err := strconv.ParseFloat(fs, 64)
+ if err != nil {
+ return 1
+ }
+ return ret
+}
+
+/*
+Charset retrieves the "charset" parameter of a MimeType.
+
+Returns an empty string if no charset exists in the parameters, or if the
parameters themselves are
+not initialized.
+*/
+func (m MimeType) Charset() string {
+ if m.Parameters == nil {
+ return ""
+ }
+
+ c, ok := m.Parameters["charset"]
+ if !ok {
+ return ""
+ }
+ return c
+}
+
+// Type returns only the "main" type of a MimeType.
+func (m MimeType) Type() string {
+ return strings.SplitN(m.Name, "/", 2)[0]
+}
+
+// SubType returns only the "sub" type of a MimeType.
+func (m MimeType) SubType() string {
+ s := strings.SplitN(m.Name, "/", 2)
+ if len(s) != 2 {
+ return ""
+ }
+ return s[1]
+}
+
+// Facet returns the MimeType's "facet" if one exists, otherwise an empty
string.
+func (m MimeType) Facet() string {
+ s := m.SubType()
+ if fx := strings.SplitN(s, ".", 2); len(fx) == 2 {
+ return fx[0]
+ }
+ return ""
+}
+
+// Syntax returns the MimeType's "syntax suffix" if one exists, otherwise an
empty string.
+func (m MimeType) Syntax() string {
+ s := m.SubType()
+ if fx := strings.Split(s, "+"); len(fx) > 1 {
+ return fx[len(fx)-1]
+ }
+ return ""
+}
+
+// String implements the Stringer interface using mime.FormatMediaType.
+func (m MimeType) String() string {
+ return mime.FormatMediaType(m.Name, m.Parameters)
+}
+
+// Satisfy checks whether or not the MimeType "satifies" some other MimeType,
o.
+//
+// Note that this does not check if the two are literally the *same*.
Specifically, if the Type or
+// SubType of the given MimeType o is the special '*' name, then this will
instead check whether or
+// not this MimeType can *satisfy* the other according to RFC7231. This means
that this satisfaction
+// check is NOT associative - that is a.Satisfy(b) does not imply b.Satisfy(a).
+//
+// See Also: The MDN documentation on the Accept Header:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept
+func (m MimeType) Satisfy(o MimeType) bool {
+ // literally anything will satisfy this
+ if o.Type() == "*" && o.SubType() == "*" {
+ return true
+ }
+
+ // it's not syntactically valid to have a "*" type and a non-"*"
subtype, e.g. "*/foo", so we're
+ // done here
+ if o.Type() != "*" && o.SubType() == "*" {
+ return o.Type() == m.Type()
+ }
+
+ if o.Type() != m.Type() || o.SubType() != m.SubType() {
+ return false
+ }
+
+ for k,v := range o.Parameters {
+ if k == "q" {
+ continue
+ }
+
+ if mv, ok := m.Parameters[k]; !ok || mv != v {
+ return false
+ }
+ }
+
+ return true
+}
+
+/*
+Less checks whether or not this MimeType is "less than" some other MimeType, o.
+
+This is done using a comparison of "quality value" of the two MimeTypes, as
specified in RFC7231.
+
+See Also: The MDN documentation on "quality value" comparisons:
https://developer.mozilla.org/en-US/docs/Glossary/Quality_values
+*/
+func (m MimeType) Less(o MimeType) bool {
+ mq := m.Quality()
+ oq := o.Quality()
+
+ if mq < oq {
+ return true
+ } else if oq < mq {
+ return false
+ }
+
+ mType := m.Type()
+ mSub := m.SubType()
+ oType := o.Type()
+ oSub := o.SubType()
+
+ if mSub == "*" {
+ if oSub != "*" {
+ return true
+ }
+
+ if mType == "*" {
+ if oType != "*" {
+ return true
+ }
+ } else if oType == "*" {
+ return false
+ }
+
+ } else if oSub == "*" {
+ return false
+ }
+
+ return len(m.Parameters) < len(o.Parameters)
+}
+
+/*
+NewMimeType creates a new MimeType, initializing its Parameters map.
+
+If v cannot be parsed as a valid MIME (by mime.ParseMediaType), returns an
error.
+*/
+func NewMimeType(v string) (m MimeType, err error) {
+ m.Name, m.Parameters, err = mime.ParseMediaType(v)
+ return m, err
+}
+
+/*
+MimeTypesFromAccept constructs a *sorted* list of MimeTypes from the provided
text, which is assumed
+to be from an HTTP 'Accept' header. The list is sorted using to SortMimeTypes.
+
+If a is an empty string, this will return an empty slice and no error.
+*/
+func MimeTypesFromAccept(a string) ([]MimeType, error) {
+ mimes := []MimeType{}
+ if a == "" {
+ return mimes, nil
+ }
+
+ for _, raw := range strings.Split(a, ",") {
+ m, err := NewMimeType(raw)
+ if err != nil {
+ return mimes, err
+ }
+ mimes = append(mimes, m)
+ }
+ SortMimeTypes(mimes)
+ return mimes, nil
+}
+
+/*
+SortMimeTypes sorts the passed MimeTypes according to their "quality value".
See MimeType.Less for
+more information on how MimeTypes are compared.
+*/
+func SortMimeTypes(m []MimeType) {
+ // using !Less because default sort order is ascending
+ sort.SliceStable(m, func(i, j int) bool {return !m[i].Less(m[j])})
+}
+
+// MIME_JSON is a pre-defined MimeType for JSON data.
+var MIME_JSON = MimeType {
+ Name: "application/json",
+ Parameters: map[string]string{},
+}
+
+// MIME_PLAINTEXT is a pre-defined MimeType for plain text data.
+var MIME_PLAINTEXT = MimeType {
+ Name: "text/plain",
+ Parameters: map[string]string{"charset": "utf-8"},
+}
+
+// MIME_HTML is a pre-defined MimeType for HTML data.
+var MIME_HTML = MimeType {
+ Name: "text/html",
+ Parameters: map[string]string{"charset": "utf-8"},
+}
diff --git a/lib/go-rfc/mimetype_test.go b/lib/go-rfc/mimetype_test.go
new file mode 100644
index 0000000..afd5737
--- /dev/null
+++ b/lib/go-rfc/mimetype_test.go
@@ -0,0 +1,359 @@
+package rfc
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import "fmt"
+import "testing"
+
+const TEST_MIME = "tEXt/vND.plaIN+cRLf; charset=utf-8; q=2.2;foo=bar"
+const PRETTY_MIME = "text/vnd.plain+crlf; charset=utf-8; foo=bar; q=2.2"
+
+func ExampleNewMimeType() {
+ m, err := NewMimeType("text/plain;charset=utf-8")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+
+ fmt.Println("Name:", m.Name, "Parameters:", m.Parameters)
+ // Output: Name: text/plain Parameters: map[charset:utf-8]
+}
+
+func ExampleMimeType_Quality() {
+ m, err := NewMimeType("text/plain;charset=utf-8")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+ q := m.Quality()
+ fmt.Print(q)
+
+ m.Parameters["q"] = "0.9"
+ q = m.Quality()
+ fmt.Println("", q)
+
+ // Output: 1 0.9
+}
+
+func ExampleMimeType_Charset() {
+ m, err := NewMimeType("text/plain;charset=utf-8")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+ c,_ := m.Charset() // it's okay to ignore this error, because I was a
good boy and used NewMimeType
+ fmt.Println(c)
+ // Output: utf-8
+}
+
+func ExampleMimeType_Type() {
+ m, err := NewMimeType("text/plain;charset=utf-8")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+ fmt.Println(m.Type())
+ // Output: text
+}
+
+func ExampleMimeType_SubType() {
+ m, err := NewMimeType("text/vnd.plain+crlf;charset=utf-8")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+ fmt.Println(m.SubType())
+ // Output: vnd.plain+crlf
+}
+
+func ExampleMimeType_Facet() {
+ m, err := NewMimeType("text/vnd.plain+crlf;charset=utf-8")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+ fmt.Println(m.Facet())
+ // Output: vnd
+}
+
+func ExampleMimeType_Syntax() {
+ m, err := NewMimeType("text/vnd.plain+crlf;charset=utf-8")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+ fmt.Println(m.Syntax())
+ // Output: crlf
+}
+
+func ExampleMimeType_String() {
+ m, err := NewMimeType("text/plain;foo=bar;charset=utf-8")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+ fmt.Println(m)
+ // Output: text/plain; charset=utf-8; foo=bar
+}
+
+func ExampleMimeType_Satisfy() {
+ m, err := NewMimeType("text/plain")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+
+ o, err := NewMimeType("text/*")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+
+ fmt.Println(m.Satisfy(o), o.Satisfy(m))
+ // Output: true false
+}
+
+func ExampleMimeType_Less() {
+ one, err := NewMimeType("text/plain")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+
+ two, err := NewMimeType("text/*")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+
+ three, err := NewMimeType("text/plain;q=0.9")
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+
+
+ fmt.Println(one.Less(two), two.Less(three), one.Less(three))
+ // Output: false false false
+}
+
+func ExampleMimeTypesFromAccept() {
+ const acceptLine = "text/html,text/xml;q=0.9,text/*;q=0.9,*/*"
+ mimes, err := MimeTypesFromAccept(acceptLine)
+ if err != nil {
+ fmt.Println(err.Error())
+ return
+ }
+
+ for _,m := range mimes {
+ fmt.Printf("%s, ", m)
+ }
+ fmt.Println()
+ // Output: text/html, */*, text/xml; q=0.9, text/*; q=0.9,
+}
+
+func ExampleSortMimeTypes() {
+ // Normally don't do this, but for the sake of brevity in an example I
will
+ mimes := []MimeType{
+ MimeType {
+ "text/html",
+ map[string]string{},
+ },
+ MimeType {
+ "text/xml",
+ map[string]string{"q": "0.9"},
+ },
+ MimeType {
+ "text/*",
+ map[string]string{"q": "0.9"},
+ },
+ MimeType {
+ "*/*",
+ map[string]string{},
+ },
+ }
+
+ SortMimeTypes(mimes)
+ for _,m := range mimes {
+ fmt.Printf("%s, ", m)
+ }
+ fmt.Println()
+ // Output: text/html, */*, text/xml; q=0.9, text/*; q=0.9,
+}
+
+func TestMimeType(t *testing.T) {
+ m, err := NewMimeType(TEST_MIME)
+ if err != nil {
+ t.Fatalf("Failed to construct a MimeType from TEST_MIME: %v",
err)
+ }
+
+ if m.Name != "text/vnd.plain+crlf" {
+ t.Errorf("Incorrect MIME name, expected 'text/vnd.plain+crld'
but got '%s'", m.Name)
+ }
+
+ if m.Type() != "text" {
+ t.Errorf("Incorrect 'Type', expected 'text' but got '%s'",
m.Type())
+ }
+
+ if m.SubType() != "vnd.plain+crlf" {
+ t.Errorf("Incorrect 'SubType', expected 'vnd.plain+crlf' but
got '%s'", m.SubType())
+ }
+
+ if m.Facet() != "vnd" {
+ t.Errorf("Incorrect 'Facet', expected 'vnd' but got '%s'",
m.Facet())
+ }
+
+ if m.Syntax() != "crlf" {
+ t.Errorf("Incorrect 'syntax suffix', expected 'crlf' but got
'%s'", m.Syntax())
+ }
+
+ if m.String() != PRETTY_MIME {
+ t.Errorf("Incorrect string representation, expected '%s' but
got '%s'", PRETTY_MIME, m.String())
+ }
+
+ if len(m.Parameters) != 3 {
+ t.Errorf("Incorrect number of Parameters, expected 3 but got
%d", len(m.Parameters))
+ }
+
+ if q := m.Quality(); q != 2.2 {
+ t.Errorf("Incorrect quality, expected 2.2 but got %g", q)
+ }
+
+ if c := m.Charset(); c != "utf-8" {
+ t.Errorf("Incorrect charset, expected 'utf-8', but got '%s'", c)
+ }
+
+}
+
+func TestMimeType_Satisfy(t *testing.T) {
+ m, err := NewMimeType(TEST_MIME)
+ if err != nil {
+ t.Fatalf("Failed to construct a MimeType from TEST_MIME: %v",
err)
+ }
+
+ o, err := NewMimeType("*/*")
+ if err != nil {
+ t.Fatalf("Failed to construct a MimeType from '*/*': %v", err)
+ }
+
+ if !m.Satisfy(o) {
+ t.Errorf("Expected %s to satisfy %s, but it did not", m, o)
+ }
+
+ if o.Satisfy(m) {
+ t.Errorf("Expected %s to not satisfy %s, but it did", o, m)
+ }
+
+ if o, err = NewMimeType("text/*"); err != nil {
+ t.Fatalf("Failed to construct a MimeType from 'text/*': %v",
err)
+ }
+
+ if !m.Satisfy(o) {
+ t.Errorf("Expected %s to satisfy %s, but it did not", m, o)
+ }
+
+ if o.Satisfy(m) {
+ t.Errorf("Expected %s to not satisfy %s, but it did", o, m)
+ }
+
+ if o, err = NewMimeType("text/vnd.plain+crlf;q=2.1"); err != nil {
+ t.Fatalf("Failed to construct a MimeType from
'text/vnd.plain+crlf;q=2.1': %v", err)
+ }
+
+ if !m.Satisfy(o) {
+ t.Errorf("Expected %s to satisfy %s, but it did not", m, o)
+ }
+
+ if o.Satisfy(m) {
+ t.Errorf("Expected %s to not satisfy %s, but it did", o, m)
+ }
+}
+
+func TestMimeType_Less(t *testing.T) {
+ m, err := NewMimeType("text/*")
+ if err != nil {
+ t.Fatalf("Failed to construct MimeType from 'text/*': %v", err)
+ }
+
+ o, err := NewMimeType("*/*")
+ if err != nil {
+ t.Fatalf("Failed to construct MimeType from '*/*': %v", err)
+ }
+
+ const less = "Expected %s to be less than %s, but it was not"
+ const notLess = "Expected %s to not be less than %s, but it was"
+
+ if m.Less(o) {
+ t.Errorf(notLess, m, o)
+ }
+
+ if !o.Less(m) {
+ t.Errorf(less, o, m)
+ }
+
+ if o, err = NewMimeType("text/plain"); err != nil {
+ t.Fatalf("Failed to construct MimeType from 'text/plain': %v",
err)
+ }
+
+ if !m.Less(o) {
+ t.Errorf(less, m, o)
+ }
+
+ if o.Less(m) {
+ t.Errorf(notLess, o, m)
+ }
+
+ if m, err = NewMimeType("text/plain;foo=bar;q=1.0"); err != nil {
+ t.Fatalf("Failed to construct MimeType from
'text/plain;foo=bar;q=1.0': %v", err)
+ }
+
+ if m.Less(o) {
+ t.Errorf(notLess, m, o)
+ }
+
+ if !o.Less(m) {
+ t.Errorf(less, o, m)
+ }
+
+ if o, err = NewMimeType("text/plain;q=1.1"); err != nil {
+ t.Fatalf("Failed to construct MimeType from 'text/plain;q=1.1':
%v", err)
+ }
+
+ if !m.Less(o) {
+ t.Errorf(less, m, o)
+ }
+
+ if o.Less(m) {
+ t.Errorf(notLess, o, m)
+ }
+
+ if o, err = NewMimeType("text/plain;fizz=buzz;q=1.0"); err != nil {
+ t.Fatalf("Failed to construct MimeType from
'text/plain;fizz=buzz;q=1.0': %v", err)
+ }
+
+ if m.Less(o) {
+ t.Errorf(notLess, m, o)
+ }
+
+ if o.Less(m) {
+ t.Errorf(notLess, o, m)
+ }
+
+}
diff --git a/lib/go-tc/traffic_stats.go b/lib/go-tc/traffic_stats.go
new file mode 100644
index 0000000..a4f0573
--- /dev/null
+++ b/lib/go-tc/traffic_stats.go
@@ -0,0 +1,247 @@
+package tc
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import "encoding/json"
+import "errors"
+import "fmt"
+import "regexp"
+import "strconv"
+import "strings"
+import "time"
+
+import "github.com/apache/trafficcontrol/lib/go-log"
+
+import influx "github.com/influxdata/influxdb1-client/v2"
+
+// TRAFFIC_STATS_VERSION was supposed to be the "API version", but actually
the plugin (this route
+// used to be a plugin in Perl) always returned this static value
+const TRAFFIC_STATS_VERSION = "1.2"
+
+// TRAFFIC_STATS_SOURCE is the value of the "source" field in an API response.
Perl always returned
+// source="TrafficStats", so we do too
+const TRAFFIC_STATS_SOURCE = "TrafficStats"
+
+// TrafficStatsDurationPattern reflects all the possible durations that can be
requested via the
+// deliveryservice_stats endpoint
+var TrafficStatsDurationPattern = regexp.MustCompile(`^\d+[mhdw]$`)
+
+// DurationLiteralToSeconds returns the number of seconds to which an InfluxQL
duration literal is
+// equivalent. For invalid objects, it returns -1 - otherwise it will always,
of course be > 0.
+func DurationLiteralToSeconds(d string) (int64, error) {
+ if strings.HasSuffix(d, "m") {
+ v,err := strconv.ParseInt(strings.Split(d, "m")[0], 10, 64)
+ return v*60, err
+ }
+
+ if strings.HasSuffix(d, "h") {
+ v,err := strconv.ParseInt(strings.Split(d, "h")[0], 10, 64)
+ return v*3600, err
+ }
+
+ if strings.HasSuffix(d, "d") {
+ v,err := strconv.ParseInt(strings.Split(d, "d")[0], 10, 64)
+ return v*86400, err
+ }
+
+ if strings.HasSuffix(d, "w") {
+ v,err := strconv.ParseInt(strings.Split(d, "w")[0], 10, 64)
+ return v*604800, err
+ }
+
+ return -1, errors.New("Invalid duration literal, no recognized suffix")
+}
+
+// TrafficStatsOrderable encodes what columns by which the data returned from
a Traffic Stats query
+// may be ordered.
+type TrafficStatsOrderable string
+const (
+ // TimeOrder indicates an ordering by time at which the measurement was
taken
+ TimeOrder TrafficStatsOrderable = "time"
+ // ValueOrder indicates an ordering by the actual value of the
measurement
+ ValueOrder TrafficStatsOrderable = "sum_count"
+)
+
+// OrderableFromString parses the passed string and returns the corresponding
value as a pointer to
+// a TrafficStatsOrderable - or nil if the value was invalid.
+func OrderableFromString(v string) *TrafficStatsOrderable {
+ var o TrafficStatsOrderable
+ switch v {
+ case "time":
+ o = TimeOrder
+ case "value":
+ o = ValueOrder
+ default:
+ return nil
+ }
+ return &o
+}
+
+// TrafficStatsExclude encodes what parts of a response to a request to a
"Traffic Stats" endpoint
+// of the TO API may be omitted.
+type TrafficStatsExclude string
+const (
+ ExcludeSeries TrafficStatsExclude = "series"
+ ExcludeSummary TrafficStatsExclude = "summary"
+ ExcludeInvalid TrafficStatsExclude = "INVALID"
+)
+
+// ExcludeFromString parses the passed string and returns the corresponding
value as a TrafficStatsExclude.
+func ExcludeFromString(v string) TrafficStatsExclude {
+ switch v {
+ case "series":
+ return ExcludeSeries
+ case "summary":
+ return ExcludeSummary
+ default:
+ return ExcludeInvalid
+ }
+}
+
+
+// TrafficStatsConfig represents the configuration of a request made to
Traffic Stats. This is
+// typically constructed by parsing a request body submitted to Traffic Ops.
+type TrafficStatsConfig struct {
+ DeliveryService string
+ End time.Time
+ ExcludeSeries bool
+ ExcludeSummary bool
+ Interval string
+ Limit *uint64
+ MetricType string
+ Offset *uint64
+ OrderBy *TrafficStatsOrderable
+ Start time.Time
+ Unix bool
+}
+
+// This is a stupid, dirty hack to try to convince Influx to not give back
data that's outside of the
+// range in a WHERE clause. It doesn't work, but it helps.
+// (https://github.com/influxdata/influxdb/issues/8010)
+func (c *TrafficStatsConfig) OffsetString() string {
+ iSecs, err := DurationLiteralToSeconds(c.Interval)
+ if err != nil {
+ log.Errorf("Parsing duration literal: %v", err)
+ return "0s"
+ }
+ return fmt.Sprintf("%ds", int64(c.Start.Sub(time.Unix(0,
0))/time.Second)%iSecs)
+}
+
+// TrafficStatsResponse represents a response from one of the "Traffic Stats
endpoints" of the
+// Traffic Ops API, e.g. `/deliveryservice_stats`.
+type TrafficStatsResponse struct {
+ // Series holds the actual data - it is NOT in general the same as a
github.com/influxdata/influxdb1-client/models.Row
+ Series *TrafficStatsSeries `json:"series,omitempty"`
+ // Source has an unknown purpose. I believe this is supposed to name
the "plugin" that provided
+ // the data - kept for compatibility with the Perl version(s) of the
"Traffic Stats endpoints".
+ // Deprecated: this'll be removed or reworked to make more sense in the
future
+ Source string `json:"source"`
+ // Summary contains summary statistics of the data in Series
+ Summary *TrafficStatsSummary `json:"summary,omitempty"`
+ // Version is supposed to represent the API version - but actually the
API just reports a static
+ // number (TRAFFIC_STATS_VERSION).
+ // Deprecated: this'll be removed or reworked to make more sense in the
future
+ Version string `json:"version"`
+}
+
+// TrafficStatsSummary contains summary statistics for a data series.
+type TrafficStatsSummary struct {
+ // Average is calculated as an arithmetic mean
+ Average float64 `json:"average"`
+ // Count is the total number of data points _except_ for any values
that would appear as 'nil'
+ // in the corresponding series
+ Count uint `json:"count"`
+ FifthPercentile float64 `json:"fifthPercentile"`
+ Max float64 `json:"max"`
+ Min float64 `json:"min"`
+ NinetyEighthPercentile float64 `json:"ninetyEighthPercentile"`
+ NinetyFifthPercentile float64 `json:"ninetyFifthPercentile"`
+ // TotalBytes is the total number of bytes served when the "metric
type" requested is "kbps"
+ // (or actually just contains "kbps"). If this is not nil,
TotalTransactions *should* always be
+ // nil.
+ TotalBytes *float64 `json:"totalBytes"`
+ // Totaltransactions is the total number of transactions within the
requested window. Whenever
+ // the requested metric doesn't contain "kbps", it assumed to be some
kind of transactions
+ // measurement. In that case, this will not be nil - otherwise it will
be nil. If this not nil,
+ // TotalBytes *should* always be nil.
+ TotalTransactions *float64 `json:"totalTransactions"`
+}
+
+// TrafficStatsSeries is the actual data returned by a request to a "Traffic
Stats endpoint".
+type TrafficStatsSeries struct {
+ // Columns is a list of column names. Each "row" in Values is ordered
to match up with these
+ // column names.
+ Columns []string `json:"columns"`
+ // Count is the total number of returned data points. Should be the
same as len(Values)
+ Count uint `json:"count"`
+ // Name is the name of the InfluxDB database from which the data was
retrieved
+ Name string `json:"name"`
+ // Tags is a set of InfluxDB tags associated with the requested
database.
+ Tags map[string]string `json:"tags"`
+ // Values is an array of rows of arbitrary data that can only really be
interpreted by
+ // inspecting Columns, in general. In practice, however, each element
is nearly always a
+ // slice where the first element is an RFC3339 timestamp (as a string)
and the second/final
+ // element is a floating point number (or nil) indicating the value at
that time.
+ Values [][]interface{} `json:"values"`
+}
+
+// FormatTimestamps formats the timestamps contained in the Values array as
RFC3339 strings.
+// This returns an error if the data is not in the expected format.
+func (s TrafficStatsSeries) FormatTimestamps() error {
+ for i, v := range s.Values {
+ if len(v) != 2 {
+ return fmt.Errorf("Datapoint %d (%v) malformed", i, v)
+ }
+
+ switch v[0].(type) {
+ case int64:
+ s.Values[i][0] = time.Unix(0,
v[0].(int64)).Format(time.RFC3339)
+ case float64:
+ s.Values[i][0] = time.Unix(0,
int64(v[0].(float64))).Format(time.RFC3339)
+ case json.Number:
+ val, err := v[0].(json.Number).Int64()
+ if err != nil {
+ return fmt.Errorf("Datapoint %d (%v) malformed:
%v", i, v, err)
+ }
+ s.Values[i][0] = time.Unix(0, val).Format(time.RFC3339)
+ default:
+ return fmt.Errorf("Invalid type %T for datapoint %d
(%v)", v[0], i, v)
+ }
+ }
+ return nil
+}
+
+// MessagesToString converts a set of messages from an InfluxDB node into a
single, print-able string
+func MessagesToString (msgs []influx.Message) string {
+ if msgs == nil || len(msgs) == 0 {
+ return ""
+ }
+
+ b := strings.Builder{}
+ b.Write([]byte("Messages: ["))
+ for _, m := range msgs {
+ b.WriteString(m.Level)
+ b.WriteRune(':')
+ b.WriteString(m.Text)
+ b.Write([]byte(", "))
+ }
+ b.WriteRune(']')
+ return b.String()
+}
diff --git a/traffic_ops/app/conf/development/influxdb.conf
b/traffic_ops/app/conf/development/influxdb.conf
index 4e1f78a..4bd5a91 100644
--- a/traffic_ops/app/conf/development/influxdb.conf
+++ b/traffic_ops/app/conf/development/influxdb.conf
@@ -2,5 +2,6 @@
"user" : "influxuser",
"password" : "password",
"deliveryservice_stats_db_name":"deliveryservice_stats",
- "cache_stats_db_name":"cache_stats"
+ "cache_stats_db_name":"cache_stats",
+ "secure": false
}
diff --git a/traffic_ops/app/conf/integration/influxdb.conf
b/traffic_ops/app/conf/integration/influxdb.conf
index 43c076e..3091d3a 100644
--- a/traffic_ops/app/conf/integration/influxdb.conf
+++ b/traffic_ops/app/conf/integration/influxdb.conf
@@ -1,4 +1,5 @@
{
"user": "influxuser",
- "password": "password"
+ "password": "password",
+ "secure": false
}
diff --git a/traffic_ops/app/conf/production/influxdb.conf
b/traffic_ops/app/conf/production/influxdb.conf
index 9137c75..ad991d6 100644
--- a/traffic_ops/app/conf/production/influxdb.conf
+++ b/traffic_ops/app/conf/production/influxdb.conf
@@ -2,5 +2,6 @@
"user": "influxuser",
"password": "password",
"deliveryservice_stats_db_name": "deliveryservice_stats",
- "cache_stats_db_name": "cache_stats"
+ "cache_stats_db_name": "cache_stats",
+ "secure": false
}
diff --git a/traffic_ops/app/conf/test/influxdb.conf
b/traffic_ops/app/conf/test/influxdb.conf
index 4e1f78a..4bd5a91 100644
--- a/traffic_ops/app/conf/test/influxdb.conf
+++ b/traffic_ops/app/conf/test/influxdb.conf
@@ -2,5 +2,6 @@
"user" : "influxuser",
"password" : "password",
"deliveryservice_stats_db_name":"deliveryservice_stats",
- "cache_stats_db_name":"cache_stats"
+ "cache_stats_db_name":"cache_stats",
+ "secure": false
}
diff --git a/traffic_ops/traffic_ops_golang/api/api.go
b/traffic_ops/traffic_ops_golang/api/api.go
index 1e91890..83958cb 100644
--- a/traffic_ops/traffic_ops_golang/api/api.go
+++ b/traffic_ops/traffic_ops_golang/api/api.go
@@ -41,6 +41,7 @@ import (
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/config"
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/tocookie"
+ influx "github.com/influxdata/influxdb1-client/v2"
"github.com/jmoiron/sqlx"
"github.com/lib/pq"
)
@@ -50,6 +51,18 @@ const ConfigContextKey = "context"
const ReqIDContextKey = "reqid"
const APIRespWrittenKey = "respwritten"
+const influxServersQuery = `
+SELECT (host_name||'.'||domain_name) as fqdn,
+ tcp_port,
+ https_port
+FROM server
+WHERE type in ( SELECT id
+ FROM type
+ WHERE name='INFLUXDB'
+ )
+AND status=(SELECT id FROM status WHERE name='ONLINE')
+`
+
// WriteResp takes any object, serializes it as JSON, and writes that to w.
Any errors are logged and written to w via tc.GetHandleErrorsFunc.
// This is a helper for the common case; not using this in unusual cases is
perfectly acceptable.
func WriteResp(w http.ResponseWriter, r *http.Request, v interface{}) {
@@ -406,6 +419,66 @@ func SendMail(to rfc.EmailAddress, msg []byte, cfg
*config.Config) (int, error,
return http.StatusOK, nil, nil
}
+// CreateInfluxClient onstructs and returns an InfluxDB HTTP client, if
enabled and when possible.
+// The error this returns should not be exposed to the user; it's for logging
purposes only.
+//
+// If Influx connections are not enabled, this will return `nil` - but also no
error. It is expected
+// that the caller will handle this situation appropriately.
+func (inf *APIInfo) CreateInfluxClient() (*influx.Client, error) {
+ if !inf.Config.InfluxEnabled {
+ return nil, nil
+ }
+
+ var fqdn string
+ var tcpPort uint
+ var httpsPort sql.NullInt64 // this is the only one that's optional
+
+ row := inf.Tx.Tx.QueryRow(influxServersQuery)
+ if e := row.Scan(&fqdn, &tcpPort, &httpsPort); e != nil {
+ return nil, fmt.Errorf("Failed to create influx client: %v", e)
+ }
+
+ host := "http%s://%s:%d"
+ if inf.Config.ConfigInflux != nil && *inf.Config.ConfigInflux.Secure {
+ if !httpsPort.Valid {
+ log.Warnf("INFLUXDB Server %s has no secure ports,
assuming default of 8086!", fqdn)
+ httpsPort = sql.NullInt64{8086, true}
+ }
+ port, err := httpsPort.Value()
+ if err != nil {
+ return nil, fmt.Errorf("Failed to create influx client:
%v", err)
+ }
+
+ p := port.(int64)
+ if p <= 0 || p > 65535 {
+ log.Warnf("INFLUXDB Server %s has invalid port,
assuming default of 8086!", fqdn)
+ p = 8086
+ }
+
+ host = fmt.Sprintf(host, "s", fqdn, p)
+ } else if tcpPort > 0 && tcpPort <= 65535 {
+ host = fmt.Sprintf(host, "", fqdn, tcpPort)
+ } else {
+ log.Warnf("INFLUXDB Server %s has invalid port, assuming
default of 8086!", fqdn)
+ host = fmt.Sprintf(host, "", fqdn, 8086)
+ }
+
+ config := influx.HTTPConfig{
+ Addr: host,
+ Username: inf.Config.ConfigInflux.User,
+ Password: inf.Config.ConfigInflux.Password,
+ UserAgent: fmt.Sprintf("TrafficOps/%s (Go)",
inf.Config.Version),
+ Timeout: time.Duration(float64(inf.Config.ReadTimeout)/2.1) *
time.Second,
+ }
+
+ var client influx.Client
+ client, e := influx.NewHTTPClient(config)
+ if client == nil {
+ return nil, fmt.Errorf("Failed to create influx client (client
was nil): %v", e)
+ }
+ return &client, e
+}
+
// APIInfoImpl implements APIInfo via the APIInfoer interface
type APIInfoImpl struct {
ReqInfo *APIInfo
diff --git a/traffic_ops/traffic_ops_golang/config/config.go
b/traffic_ops/traffic_ops_golang/config/config.go
index 14dae49..d5d254a 100644
--- a/traffic_ops/traffic_ops_golang/config/config.go
+++ b/traffic_ops/traffic_ops_golang/config/config.go
@@ -21,16 +21,17 @@ package config
import (
"encoding/json"
+ "errors"
"fmt"
"io/ioutil"
"net/url"
"os"
"path/filepath"
-
"strings"
"github.com/apache/trafficcontrol/lib/go-log"
"github.com/apache/trafficcontrol/lib/go-rfc"
+ "github.com/apache/trafficcontrol/lib/go-util"
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/riaksvc"
"github.com/basho/riak-go-client"
)
@@ -47,12 +48,15 @@ type Config struct {
DB ConfigDatabase `json:"db"`
Secrets []string `json:"secrets"`
// NOTE: don't care about any other fields for now..
- RiakAuthOptions *riak.AuthOptions
- RiakEnabled bool
- ConfigLDAP *ConfigLDAP
- LDAPEnabled bool
- LDAPConfPath string `json:"ldap_conf_location"`
- Version string
+ RiakAuthOptions *riak.AuthOptions
+ RiakEnabled bool
+ ConfigLDAP *ConfigLDAP
+ LDAPEnabled bool
+ LDAPConfPath string `json:"ldap_conf_location"`
+ ConfigInflux *ConfigInflux
+ InfluxEnabled bool
+ InfluxDBConfPath string `json:"influxdb_conf_path"`
+ Version string
}
// ConfigHypnotoad carries http setting for hypnotoad (mojolicious) server
@@ -141,6 +145,14 @@ type ConfigLDAP struct {
LDAPTimeoutSecs int `json:"ldap_timeout_secs"`
}
+type ConfigInflux struct {
+ User string `json:"user"`
+ Password string `json:"password"`
+ DSDBName string `json:"deliveryservice_stats_db_name"`
+ CacheDBName string `json:"cache_stats_db_name"`
+ Secure *bool `json:"secure"`
+}
+
const DefaultLDAPTimeoutSecs = 60
const DefaultDBQueryTimeoutSecs = 20
@@ -235,11 +247,25 @@ func LoadConfig(cdnConfPath string, dbConfPath string,
riakConfPath string, appV
return cfg, []error{err}, BlockStartup
}
} else {
- cfg.LDAPEnabled = false
- return cfg, []error{}, AllowStartup // no ldap.conf,
disable and allow startup
+ cfg.LDAPEnabled = false // no ldap.conf, disable and
allow startup
}
}
+ idbPath := cfg.InfluxDBConfPath
+ if idbPath == "" {
+ idbPath = filepath.Join(filepath.Dir(cdnConfPath),
"influxdb.conf")
+ }
+
+ if _, err = os.Stat(idbPath); err != nil {
+ if os.IsNotExist(err) {
+ cfg.InfluxEnabled = false
+ } else {
+ return cfg, []error{err}, BlockStartup
+ }
+ } else if cfg.InfluxEnabled, cfg.ConfigInflux, err =
GetInfluxConfig(idbPath); err != nil {
+ return cfg, []error{err}, BlockStartup
+ }
+
return cfg, []error{}, AllowStartup
}
@@ -381,6 +407,43 @@ func GetLDAPConfig(LDAPConfPath string) (bool,
*ConfigLDAP, error) {
return true, LDAPconf, nil
}
+func GetInfluxConfig(path string) (bool, *ConfigInflux, error) {
+ raw, err := ioutil.ReadFile(path)
+ if err != nil {
+ return false, nil, fmt.Errorf("reading InfluxDB configuration
from '%s': %v", path, err)
+ }
+
+ c := ConfigInflux{}
+ if err := json.Unmarshal(raw, &c); err != nil {
+ return false, nil, fmt.Errorf("parsing InfluxDB configuration
from '%s': %v", path, err)
+ }
+
+ if c.User == "" {
+ return false, &c, errors.New("InfluxDB configuration missing a
username")
+ }
+
+ if c.Password == "" {
+ return false, &c, errors.New("InfluxDB configuration missing a
password")
+ }
+
+ if c.DSDBName == "" {
+ log.Warnln("InfluxDB configuration does not specify a DS stats
DB name - falling back on 'deliveryservice_stats'")
+ c.DSDBName = "deliveryservice_stats"
+ }
+
+ if c.CacheDBName == "" {
+ log.Warnln("InfluxDB configuration does not specify a Cache
Stats DB name - falling back on 'cache_stats'")
+ c.CacheDBName = "cache_stats"
+ }
+
+ if c.Secure == nil {
+ log.Warnln("InfluxDB configuration does not specify 'secure',
defaulting to 'true'")
+ c.Secure = util.BoolPtr(true)
+ }
+
+ return true, &c, nil
+}
+
func getLDAPConf(s string) (*ConfigLDAP, error) {
ldapConf := ConfigLDAP{LDAPTimeoutSecs: DefaultLDAPTimeoutSecs} //if
the field is not set in the config we use the default instead of 0
err := json.Unmarshal([]byte(s), &ldapConf)
diff --git a/traffic_ops/traffic_ops_golang/routing/routes.go
b/traffic_ops/traffic_ops_golang/routing/routes.go
index 28a47ae..2ec9a5b 100644
--- a/traffic_ops/traffic_ops_golang/routing/routes.go
+++ b/traffic_ops/traffic_ops_golang/routing/routes.go
@@ -75,6 +75,7 @@ import (
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/steering"
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/steeringtargets"
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/systeminfo"
+
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/trafficstats"
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/types"
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/urisigning"
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/user"
@@ -111,6 +112,9 @@ func Routes(d ServerData) ([]Route, []RawRoute,
http.Handler, error) {
{1.1, http.MethodPost, `asns/?$`,
api.CreateHandler(&asn.TOASNV11{}), auth.PrivLevelOperations, Authenticated,
nil},
{1.1, http.MethodDelete, `asns/{id}$`,
api.DeleteHandler(&asn.TOASNV11{}), auth.PrivLevelOperations, Authenticated,
nil},
+ // Traffic Stats access
+ {1.2, http.MethodGet, `deliveryservice_stats`,
trafficstats.GetDSStats, auth.PrivLevelOperations, Authenticated, nil},
+
{1.1, http.MethodGet, `caches/stats/?(\.json)?$`,
cachesstats.Get, auth.PrivLevelReadOnly, Authenticated, nil},
//CacheGroup: CRUD
diff --git a/traffic_ops/traffic_ops_golang/traffic_ops_golang.go
b/traffic_ops/traffic_ops_golang/traffic_ops_golang.go
index 7b1f834..6392c42 100644
--- a/traffic_ops/traffic_ops_golang/traffic_ops_golang.go
+++ b/traffic_ops/traffic_ops_golang/traffic_ops_golang.go
@@ -74,10 +74,10 @@ func main() {
}
cfg, errsToLog, blockStart := config.LoadConfig(*configFileName,
*dbConfigFileName, *riakConfigFileName, version)
+ for _, err := range errsToLog {
+ log.Errorln(err)
+ }
if blockStart {
- for _, err := range errsToLog {
- fmt.Println(err)
- }
os.Exit(1)
}
diff --git a/traffic_ops/traffic_ops_golang/trafficstats/trafficstats.go
b/traffic_ops/traffic_ops_golang/trafficstats/trafficstats.go
new file mode 100644
index 0000000..b1b2fb4
--- /dev/null
+++ b/traffic_ops/traffic_ops_golang/trafficstats/trafficstats.go
@@ -0,0 +1,595 @@
+package trafficstats
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import "database/sql"
+import "encoding/json"
+import "errors"
+import "fmt"
+import "net/http"
+import "strconv"
+import "strings"
+import "time"
+
+import "github.com/apache/trafficcontrol/lib/go-tc"
+import "github.com/apache/trafficcontrol/lib/go-rfc"
+import "github.com/apache/trafficcontrol/lib/go-log"
+import "github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/api"
+import "github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/tenant"
+
+import influx "github.com/influxdata/influxdb1-client/v2"
+
+var (
+ metricTypes = map[string]interface{}{
+ "kbps": struct{}{},
+ "tps_total": struct{}{},
+ "tps_2xx": struct{}{},
+ "tps_3xx": struct{}{},
+ "tps_4xx": struct{}{},
+ "tps_5xx": struct{}{},
+ }
+
+ jsonWithRFCTimestamps = rfc.MimeType{
+ "application/json",
+ map[string]string{"timestamp": "rfc"},
+ }
+
+ jsonWithUnixTimestamps = rfc.MimeType{
+ "application/json",
+ map[string]string{"timestamp": "unix"},
+ }
+)
+
+const (
+ DEFAULT_INTERVAL = "1m"
+ dsTenantIDFromXMLIDQuery = `
+ SELECT tenant_id
+ FROM deliveryservice
+ WHERE xml_id = $1`
+
+ xmlidFromIDQuery = `
+ SELECT xml_id
+ FROM deliveryservice
+ WHERE id = $1`
+
+ // TODO: Pretty sure all of this could actually be calculated using the
fetched data (assuming an
+ // interval is given). Check to see if that's faster than doing another
synchronous HTTP request.
+ summaryQuery = `
+ SELECT mean(value) AS "average",
+ percentile(value, 5) AS "fifthPercentile",
+ percentile(value, 95) AS "ninetyFifthPercentile",
+ percentile(value, 98) AS "ninetyEighthPercentile",
+ min(value) AS "min",
+ max(value) AS "max",
+ count(value) AS "count"
+ FROM "%s"."monthly"."%s.ds.1min"
+ WHERE time >= $start
+ AND time <= $end
+ AND cachegroup = 'total'
+ AND deliveryservice = $xmlid`
+
+ seriesQuery = `
+ SELECT mean(value)
+ FROM "%s"."monthly"."%s.ds.1min"
+ WHERE cachegroup = 'total'
+ AND deliveryservice = $xmlid
+ AND time >= $start
+ AND time <= $end
+ GROUP BY time(%s, %s), cachegroup%s`
+)
+
+func ConfigFromRequest(r *http.Request, i *api.APIInfo)
(tc.TrafficStatsConfig, error, int) {
+ var c tc.TrafficStatsConfig
+ var e error
+ if accept := r.Header.Get("Accept"); accept != "" {
+
+ mimes, err := rfc.MimeTypesFromAccept(accept)
+ if err != nil {
+ log.Warnf("Failed to negotiate content, Accept line
'%s', error: %v", accept, err)
+ } else {
+
+ found := false
+ for _, m := range mimes {
+ if jsonWithRFCTimestamps.Satisfy(m) {
+ found = true
+ break
+ }
+
+ if jsonWithUnixTimestamps.Satisfy(m) {
+ found = true
+ c.Unix = true
+ break
+ }
+ }
+
+ if !found {
+ e = fmt.Errorf("Failed to negotiate content;
cannot produce output satisfying %s", accept)
+ return c, e, http.StatusNotAcceptable
+ }
+ }
+ }
+
+ if limit, ok := i.Params["limit"]; ok {
+ lim, err := strconv.ParseUint(limit, 10, 64)
+ if err != nil {
+ e = errors.New("Invalid limit!")
+ return c, e, http.StatusBadRequest
+ }
+ c.Limit = &lim
+ }
+
+ if offset, ok := i.Params["offset"]; ok {
+ off, err := strconv.ParseUint(offset, 10, 64)
+ if err != nil {
+ e = errors.New("Invalid offset!")
+ return c, e, http.StatusBadRequest
+ }
+ c.Offset = &off
+ }
+
+ if orderby, ok := i.Params["orderby"]; ok {
+ if c.OrderBy = tc.OrderableFromString(orderby); c.OrderBy ==
nil {
+ e = errors.New("Invalid orderby! Must be 'time' or
'value'")
+ return c, e, http.StatusBadRequest
+ }
+ }
+
+ if c.Start, e = parseTime(i.Params["startDate"]); e != nil {
+ log.Errorf("Parsing startDate: %v", e)
+ e = errors.New("Invalid startDate!")
+ return c, e, http.StatusBadRequest
+ }
+
+ if c.End, e = parseTime(i.Params["endDate"]); e != nil {
+ log.Errorf("Parsing endDate: %v", e)
+ e = errors.New("Invalid endDate!")
+ return c, e, http.StatusBadRequest
+ }
+
+ if interval, ok := i.Params["interval"]; !ok {
+ c.Interval = DEFAULT_INTERVAL
+ } else if !tc.TrafficStatsDurationPattern.MatchString(interval) {
+ e = errors.New("interval: must be a valid InfluxQL duration
literal (resolution no less than minute)")
+ return c, e, http.StatusBadRequest
+ } else {
+ c.Interval = interval
+ }
+
+ if ex, ok := i.Params["exclude"]; ok {
+ switch tc.ExcludeFromString(ex) {
+ case tc.ExcludeSummary:
+ c.ExcludeSummary = true
+ case tc.ExcludeSeries:
+ c.ExcludeSeries = true
+ default:
+ e = errors.New("Invalid exclude! Must be 'series' or
'summary'")
+ return c, e, http.StatusBadRequest
+ }
+ }
+
+ c.MetricType = i.Params["metricType"]
+ if _, ok := metricTypes[c.MetricType]; !ok {
+ e = fmt.Errorf("Unknown metric type: %s", c.MetricType)
+ return c, e, http.StatusBadRequest
+ }
+
+ var ok bool
+ if c.DeliveryService, ok = i.Params["deliveryServiceName"]; !ok {
+ if c.DeliveryService, ok = i.Params["deliveryService"]; !ok {
+ e = errors.New("You must specify deliveryService or
deliveryServiceName!")
+ return c, e, http.StatusBadRequest
+ }
+
+ if dsID, err := strconv.ParseUint(c.DeliveryService, 10, 64);
err == nil {
+ // sql.ErrNoRows does not *necessarily* mean the DS
doesn't exist - an XMLID can simply
+ // be numeric, and so it was wrong to treat it as an ID
in the first place.
+ xmlid := c.DeliveryService
+ var exists bool
+ if exists, c.DeliveryService, err =
getXMLIDFromID(dsID, i.Tx.Tx); err != nil {
+ log.Errorf("Converting DSID to XMLID: %v", err)
+ e = errors.New("Internal Server Error")
+ return c, e, http.StatusInternalServerError
+ } else if !exists {
+ c.DeliveryService = xmlid
+ }
+ }
+ }
+
+ return c, nil, http.StatusOK
+}
+
+func GetDSStats(w http.ResponseWriter, r *http.Request) {
+ // Perl didn't require "interval", but it would only return summary
data if it was not given
+ inf, userErr, sysErr, errCode := api.NewInfo(r, []string{"metricType",
"startDate", "endDate"}, nil)
+ tx := inf.Tx.Tx
+ if userErr != nil || sysErr != nil {
+ api.HandleErr(w, r, tx, errCode, userErr, sysErr)
+ return
+ }
+ defer inf.Close()
+
+ var c tc.TrafficStatsConfig
+ if c, userErr, errCode = ConfigFromRequest(r, inf); userErr != nil {
+ sysErr = fmt.Errorf("Unable to process deliveryservice_stats
request: %v", userErr)
+ api.HandleErr(w, r, tx, errCode, userErr, sysErr)
+ return
+ }
+
+ client, err := inf.CreateInfluxClient()
+ if err != nil {
+ errCode = http.StatusInternalServerError
+ sysErr = err
+ api.HandleErr(w, r, tx, errCode, nil, sysErr)
+ return
+ } else if client == nil {
+ sysErr = errors.New("Traffic Stats is not configured, but DS
stats were requested")
+ errCode = http.StatusInternalServerError
+ api.HandleErr(w, r, tx, errCode, nil, sysErr)
+ return
+ }
+ defer (*client).Close()
+
+ exists, dsTenant, err := dsTenantIDFromXMLID(c.DeliveryService, tx)
+ if err != nil {
+ sysErr = err
+ errCode = http.StatusInternalServerError
+ api.HandleErr(w, r, tx, errCode, nil, sysErr)
+ return
+ } else if !exists {
+ userErr = fmt.Errorf("No such Delivery Service: %s",
c.DeliveryService)
+ errCode = http.StatusNotFound
+ api.HandleErr(w, r, tx, errCode, userErr, nil)
+ return
+ }
+
+ authorized, err := tenant.IsResourceAuthorizedToUserTx(int(dsTenant),
inf.User, tx)
+ if err != nil {
+ api.HandleErr(w, r, tx, http.StatusInternalServerError, nil,
err)
+ return
+ } else if !authorized {
+ // If the Tenant is not authorized to use the resource, then we
DON'T tell them that.
+ // Instead, we don't disclose that such a Delivery Service
exists at all - in keeping with
+ // the behavior of /deliveryservices
+ // This is different from what Perl used to do, but then again
Perl didn't check tenancy at
+ // all.
+ userErr = fmt.Errorf("No such Delivery Service: %s",
c.DeliveryService)
+ sysErr = fmt.Errorf("GetDSStats: unauthorized Tenant (#%d)
access", inf.User.TenantID)
+ errCode = http.StatusNotFound
+ api.HandleErr(w, r, tx, errCode, userErr, sysErr)
+ return
+ }
+
+ resp := struct{ Response tc.TrafficStatsResponse `json:"response"` }{
+ Response: tc.TrafficStatsResponse{
+ Source: tc.TRAFFIC_STATS_SOURCE,
+ Version: tc.TRAFFIC_STATS_VERSION,
+ Series: nil,
+ Summary: nil,
+ },
+ }
+
+ // TODO: as above, this could be done on TO itself, thus sending only
one synchronous request
+ // per hit on this endpoint, rather than the current two. Not sure if
that's worth it for large
+ // data sets, though.
+ if !c.ExcludeSummary {
+ summary, messages, err := getSummary(client, &c,
inf.Config.ConfigInflux.DSDBName)
+ log.Debugf("Messages from summary query: %s",
tc.MessagesToString(messages))
+
+ if err != nil {
+ sysErr = fmt.Errorf("Getting summary response from
Influx: %v", err)
+ api.HandleErr(w, r, tx, http.StatusInternalServerError,
nil, sysErr)
+ return
+ }
+
+ resp.Response.Summary = &summary
+ }
+
+ if !c.ExcludeSeries {
+ series, messages, err := getSeries(client, &c,
inf.Config.ConfigInflux.DSDBName)
+ log.Debugf("Messages from series query: %s",
tc.MessagesToString(messages))
+
+ if err != nil {
+ sysErr = fmt.Errorf("Getting summary response from
Influx: %v", err)
+ api.HandleErr(w, r, tx, http.StatusInternalServerError,
nil, sysErr)
+ return
+ }
+
+ if !c.Unix {
+ series.FormatTimestamps()
+ }
+
+ resp.Response.Series = &series
+ }
+
+ respBts, err := json.Marshal(resp)
+ if err != nil {
+ sysErr = fmt.Errorf("Marshalling response: %v", err)
+ errCode = http.StatusInternalServerError
+ api.HandleErr(w, r, tx, errCode, nil, sysErr)
+ return
+ }
+
+ if c.Unix {
+ w.Header().Set(tc.ContentType, jsonWithUnixTimestamps.String())
+ } else {
+ w.Header().Set(tc.ContentType, jsonWithRFCTimestamps.String())
+ }
+ w.Header().Set(http.CanonicalHeaderKey("vary"),
http.CanonicalHeaderKey("Accept"))
+ w.Write(append(respBts, '\n'))
+}
+
+func getSummary(client *influx.Client, conf *tc.TrafficStatsConfig, db string)
(tc.TrafficStatsSummary, []influx.Message, error) {
+
+ msgs := []influx.Message{}
+ s := tc.TrafficStatsSummary{}
+ qStr := fmt.Sprintf(summaryQuery, db, conf.MetricType)
+ q := influx.NewQueryWithParameters(qStr,
+ db,
+ "rfc3339", //this doesn't actually seem to have any effect...
+ map[string]interface{}{
+ "xmlid": conf.DeliveryService,
+ "start": conf.Start,
+ "end": conf.End,
+ "interval": string(conf.Interval),
+ })
+ q.RetentionPolicy = "monthly"
+
+ log.Debugf("InfluxDB SummaryQuery: %+v", q)
+
+ resp, err := (*client).Query(q)
+ if err != nil {
+ return s, msgs, err
+ }
+
+ if resp.Results != nil && len(resp.Results) == 1 {
+ r := resp.Results[0]
+ if r.Messages != nil {
+ for _, m := range r.Messages {
+ if m != nil {
+ msgs = append(msgs, *m)
+ }
+ }
+ }
+
+ if len(r.Series) != 1 {
+ return s, msgs, fmt.Errorf("Improper number of series:
%d", len(r.Series))
+ }
+
+ series := r.Series[0]
+
+ if len(series.Values) != 1 {
+ return s, msgs, fmt.Errorf("Improper number of returned
rows: %d", len(r.Series[0].Values))
+ }
+
+ vals := series.Values[0]
+ if len(vals) != 8 || len(series.Columns) != 8 {
+ return s, msgs, fmt.Errorf("Improper number of returned
values in row: %d (%d cols)", len(vals), len(series.Columns))
+ }
+
+ mappedValues := map[string]interface{}{}
+ for i, v := range vals {
+ mappedValues[series.Columns[i]] = v
+ }
+
+ var err error
+ if s.Average, err = extractFloat64("average", mappedValues);
err != nil {
+ return s, msgs, err
+ }
+
+ if s.Count, err = extractUInt("count", mappedValues); err !=
nil {
+ return s, msgs, err
+ }
+
+ if s.FifthPercentile, err = extractFloat64("fifthPercentile",
mappedValues); err != nil {
+ return s, msgs, err
+ }
+
+ if s.Max, err = extractFloat64("max", mappedValues); err != nil
{
+ return s, msgs, err
+ }
+
+ if s.Min, err = extractFloat64("min", mappedValues); err != nil
{
+ return s, msgs, err
+ }
+
+ if s.Max, err = extractFloat64("max", mappedValues); err != nil
{
+ return s, msgs, err
+ }
+
+ if s.NinetyEighthPercentile, err =
extractFloat64("ninetyEighthPercentile", mappedValues); err != nil {
+ return s, msgs, err
+ }
+
+ if s.NinetyFifthPercentile, err =
extractFloat64("ninetyFifthPercentile", mappedValues); err != nil {
+ return s, msgs, err
+ }
+
+ } else {
+ log.Debugf("InfluxDB summary response: %+v", resp)
+ return s, msgs, errors.New("'results' missing or improper!")
+ }
+
+ if resp.Error() != nil {
+ log.Debugf("response error, summary object was: %+v", s)
+ return s, msgs, resp.Error()
+ }
+
+ value := float64(s.Count*60) * s.Average
+ if conf.MetricType == "kbps" {
+ value /= 1000
+ s.TotalBytes = &value
+ } else {
+ s.TotalTransactions = &value
+ }
+
+ return s, msgs, nil
+}
+
+func extractUInt(k string, m map[string]interface{}) (uint, error) {
+ tmp, ok := m[k]
+ if !ok {
+ return 0, fmt.Errorf("response has no value for column %s", k)
+ }
+ switch t := tmp.(type) {
+ case float64:
+ return uint(tmp.(float64)), nil
+ case json.Number:
+ ret, err := tmp.(json.Number).Int64()
+ if err != nil {
+ return 0, fmt.Errorf("Error parsing value for column
'%s' as an int64: %v", k, err)
+ }
+ return uint(ret), nil
+ default:
+ return 0, fmt.Errorf("invalid type for column '%s' - expected
unsigned integer, got %T (%v)", k, t, tmp)
+ }
+}
+
+func extractFloat64(k string, m map[string]interface{}) (float64, error) {
+ tmp, ok := m[k]
+ if !ok {
+ return 0, fmt.Errorf("response has no value for column %s", k)
+ }
+
+ switch t := tmp.(type) {
+ case float64:
+ return tmp.(float64), nil
+ case json.Number:
+ ret, err := tmp.(json.Number).Float64()
+ if err != nil {
+ return 0, fmt.Errorf("Error parsing value for column
'%s' as a float64: %v", k, err)
+ }
+ return ret, nil
+ case nil:
+ // This is the only field that can be nil - because sometimes
there isn't enough data for
+ // 5% of it to be below a value. Could probably coalesce this
in the query at some point, though.
+ if k == "fifthPercentile" {
+ return 0, nil
+ }
+ return 0, fmt.Errorf("column '%s' was null/blank", k)
+ default:
+ return 0, fmt.Errorf("invalid type for column '%s' - expected
'float64' or 'json.Number', got %T (%v)", k, t, tmp)
+ }
+}
+
+func parseTime(raw string) (time.Time, error) {
+ t, e := time.Parse(time.RFC3339Nano, raw)
+ if e == nil {
+ return t, nil
+ }
+
+ if i, err := strconv.ParseInt(raw, 10, 64); err == nil {
+ t = time.Unix(0, i)
+ return t, nil
+ }
+
+ return time.Parse(tc.TimeLayout, raw)
+}
+
+func dsTenantIDFromXMLID(xmlid string, tx *sql.Tx) (bool, uint, error) {
+ row := tx.QueryRow(dsTenantIDFromXMLIDQuery, xmlid)
+ var tid uint
+ err := row.Scan(&tid)
+ if err == sql.ErrNoRows {
+ return false, 0, nil
+ }
+ return true, tid, err
+}
+
+func getXMLIDFromID(id uint64, tx *sql.Tx) (bool, string, error) {
+ row := tx.QueryRow(xmlidFromIDQuery, id)
+ var xmlid string
+ err := row.Scan(&xmlid)
+ if err == sql.ErrNoRows {
+ return false, "", nil
+ }
+ return true, xmlid, err
+}
+
+func getSeries(client *influx.Client, conf *tc.TrafficStatsConfig, db string)
(tc.TrafficStatsSeries, []influx.Message, error) {
+ s := tc.TrafficStatsSeries{}
+ msgs := []influx.Message{}
+ extraClauses := strings.Builder{}
+ if conf.OrderBy != nil {
+ extraClauses.Write([]byte(" ORDER BY "))
+ extraClauses.WriteString(string(*conf.OrderBy))
+ }
+
+ if conf.Limit != nil {
+ extraClauses.WriteString(fmt.Sprintf(" LIMIT %d", *conf.Limit))
+ }
+
+ if conf.Offset != nil {
+ extraClauses.WriteString(fmt.Sprintf(" OFFSET %d",
*conf.Offset))
+ }
+
+ qStr := fmt.Sprintf(seriesQuery, db, conf.MetricType, conf.Interval,
conf.OffsetString(), extraClauses.String())
+ q := influx.NewQueryWithParameters(qStr,
+ db,
+ "rfc3339", // this doesn't seem to do anything...
+ map[string]interface{}{
+ "xmlid": conf.DeliveryService,
+ "start": conf.Start,
+ "end": conf.End,
+ })
+ q.RetentionPolicy = "monthly"
+
+ log.Debugf("InfluxDB series query: %+v", q)
+
+ resp, err := (*client).Query(q)
+ if err != nil {
+ return s, msgs, err
+ }
+
+ if resp.Results != nil && len(resp.Results) == 1 {
+ r := resp.Results[0]
+ if r.Messages != nil {
+ for _, m := range r.Messages {
+ if m != nil {
+ msgs = append(msgs, *m)
+ }
+ }
+ }
+
+ if len(r.Series) != 1 {
+ return s, msgs, fmt.Errorf("Improper number of series:
%d", len(r.Series))
+ }
+
+ series := r.Series[0]
+
+ s = tc.TrafficStatsSeries{
+ Name: series.Name,
+ Tags: series.Tags,
+ Values: series.Values,
+ Columns: series.Columns,
+ Count: uint(len(series.Values)),
+ }
+
+ } else {
+ log.Debugf("InfluxDB series response: %+v", resp)
+ return s, msgs, errors.New("'results' missing or improper!")
+ }
+
+ if resp.Error() != nil {
+ log.Debugf("response error, series object was %+v", s)
+ return s, msgs, resp.Error()
+ }
+
+ return s, msgs, nil
+}
diff --git a/traffic_ops/traffic_ops_golang/trafficstats/trafficstats_test.go
b/traffic_ops/traffic_ops_golang/trafficstats/trafficstats_test.go
new file mode 100644
index 0000000..e96a629
--- /dev/null
+++ b/traffic_ops/traffic_ops_golang/trafficstats/trafficstats_test.go
@@ -0,0 +1,106 @@
+package trafficstats
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import "net/http"
+import "testing"
+import "time"
+
+import "github.com/apache/trafficcontrol/lib/go-tc"
+import "github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/api"
+
+func TestConfigFromRequest(t *testing.T) {
+ start := "2019-09-30T00:00:00Z"
+ startTime, err := time.Parse(time.RFC3339, start)
+ if err != nil {
+ t.Fatalf("Failed to parse test start time: %v", err)
+ }
+ end := "2019-10-01 00:00:00-07"
+ endTime, err := time.Parse(tc.TimeLayout, end)
+ if err != nil {
+ t.Fatalf("Failed to parse test end time: %v", err)
+ }
+
+ inf := api.APIInfo{
+ Params: map[string]string{
+ "limit": "10",
+ "offset": "0",
+ "orderby": "time",
+ "startDate": start,
+ "endDate": end,
+ "interval": "1m",
+ "metricType": "tps_total",
+ "deliveryService": "test",
+ },
+ }
+
+ r, e := http.NewRequest(http.MethodGet,
"https://example.test/api/1.4/deliveryservice_stats";, nil)
+ if e != nil {
+ t.Fatalf("Failed to build test request: %v", e)
+ }
+ r.Header.Add(tc.ContentType, tc.ApplicationJson)
+
+ cfg, err, code := ConfigFromRequest(r, &inf)
+ if err != nil {
+ t.Errorf("Unexpected error: %v", err)
+ }
+ if code != http.StatusOK {
+ t.Errorf("Expected OK status, but was %d", code)
+ }
+ if cfg.DeliveryService != "test" {
+ t.Errorf("Expected config to have DS 'test', but was '%s'",
cfg.DeliveryService)
+ }
+ if !cfg.End.Equal(endTime) {
+ t.Errorf("Expected end time to be %v, but was %v", endTime,
cfg.End)
+ }
+ if !cfg.Start.Equal(startTime) {
+ t.Errorf("Expected start time to be %v, but was %v", startTime,
cfg.Start)
+ }
+ if cfg.ExcludeSeries {
+ t.Errorf("Expected series to not be excluded, but it was")
+ }
+ if cfg.ExcludeSummary {
+ t.Errorf("Expected summary to not be excluded, but it was")
+ }
+ if cfg.Interval != tc.OneMinute {
+ t.Errorf("Expected interval to be '1m', but it was %s",
cfg.Interval)
+ }
+ if cfg.Limit == nil {
+ t.Errorf("Expected limit to not be nil, but it was")
+ } else if *cfg.Limit != 10 {
+ t.Errorf("Expected limit to be 10, but it was %d", *cfg.Limit)
+ }
+ if cfg.MetricType != "tps_total" {
+ t.Errorf("Expected metric type to be tps_total, but it was %s",
cfg.MetricType)
+ }
+ if cfg.Offset == nil {
+ t.Errorf("Expected offset to not be nil, but it was")
+ } else if *cfg.Offset != 0 {
+ t.Errorf("Expected offset to be 0, but it was %d", *cfg.Offset)
+ }
+ if cfg.OrderBy == nil {
+ t.Errorf("Expected Order By to not be nil, but it was")
+ } else if *cfg.OrderBy != tc.TimeOrder {
+ t.Errorf("Expected Order by to be time, but it was %s",
*cfg.OrderBy)
+ }
+ if cfg.Unix {
+ t.Errorf("Expected Unix to not be set without MIME parameter,
but it was")
+ }
+}
diff --git a/traffic_portal/app/src/common/api/DeliveryServiceStatsService.js
b/traffic_portal/app/src/common/api/DeliveryServiceStatsService.js
index ea97fbe..1898c3d 100644
--- a/traffic_portal/app/src/common/api/DeliveryServiceStatsService.js
+++ b/traffic_portal/app/src/common/api/DeliveryServiceStatsService.js
@@ -23,7 +23,7 @@ var DeliveryServiceStatsService = function($http, $q, ENV,
messageModel) {
var request = $q.defer();
var url = ENV.api['root'] + "deliveryservice_stats",
- params = { deliveryServiceName: xmlId, metricType:
'kbps', serverType: 'edge', startDate: start.seconds(00).format(), endDate:
end.seconds(00).format(), interval: '60s' };
+ params = { deliveryServiceName: xmlId, metricType:
'kbps', serverType: 'edge', startDate: start.seconds(00).format(), endDate:
end.seconds(00).format(), interval: '1m' };
$http.get(url, { params: params })
.then(
@@ -43,7 +43,7 @@ var DeliveryServiceStatsService = function($http, $q, ENV,
messageModel) {
var request = $q.defer();
var url = ENV.api['root'] + "deliveryservice_stats",
- params = { deliveryServiceName: xmlId, metricType:
'tps_total', serverType: 'edge', startDate: start.seconds(00).format(),
endDate: end.seconds(00).format(), interval: '60s' };
+ params = { deliveryServiceName: xmlId, metricType:
'tps_total', serverType: 'edge', startDate: start.seconds(00).format(),
endDate: end.seconds(00).format(), interval: '1m' };
$http.get(url, { params: params })
.then(
@@ -63,7 +63,7 @@ var DeliveryServiceStatsService = function($http, $q, ENV,
messageModel) {
var request = $q.defer();
var url = ENV.api['root'] + "deliveryservice_stats",
- params = { deliveryServiceName: xmlId, metricType:
'tps_' + httpStatus, serverType: 'edge', startDate: start.seconds(00).format(),
endDate: end.seconds(00).format(), interval: '60s' };
+ params = { deliveryServiceName: xmlId, metricType:
'tps_' + httpStatus, serverType: 'edge', startDate: start.seconds(00).format(),
endDate: end.seconds(00).format(), interval: '1m' };
$http.get(url, { params: params })
.then(
