[trafficserver] branch 9.0.x updated: traffic_dump: Make client-request gathered in a global hook (#6500)

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git


The following commit(s) were added to refs/heads/9.0.x by this push:
 new 919b9fa  traffic_dump: Make client-request gathered in a global hook 
(#6500)
919b9fa is described below

commit 919b9fa6b3d7366cda181aab0172c32308945360
Author: Brian Neradt 
AuthorDate: Tue Mar 10 21:31:26 2020 -0500

traffic_dump: Make client-request gathered in a global hook (#6500)

This will make client-request dumps more accurate to what the client
sent since it will allow us to capture the data before other plugins
modify it if they register their handler with a global
TS_HTTP_READ_REQUEST_HDR_HOOK. (This assumes traffic_dump.so is placed
at the top of the plugin.config list.)

Co-authored-by: bneradt 
(cherry picked from commit 41b8ea001c496f92ae50977fb78f3aedc64892fa)
---
 plugins/experimental/traffic_dump/traffic_dump.cc | 21 +
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/plugins/experimental/traffic_dump/traffic_dump.cc 
b/plugins/experimental/traffic_dump/traffic_dump.cc
index b802c91..9241c2d 100644
--- a/plugins/experimental/traffic_dump/traffic_dump.cc
+++ b/plugins/experimental/traffic_dump/traffic_dump.cc
@@ -297,9 +297,9 @@ write_message_node_no_content(TSMBuffer , TSMLoc 
_loc)
   TSMLoc field_loc = TSMimeHdrFieldGet(buffer, hdr_loc, 0);
   while (field_loc) {
 TSMLoc next_field_loc;
-const char *name;
-const char *value;
-int name_len, value_len;
+const char *name  = nullptr;
+const char *value = nullptr;
+int name_len = 0, value_len = 0;
 // Append to "fields" list if valid value exists
 if ((name = TSMimeHdrFieldNameGet(buffer, hdr_loc, field_loc, _len)) 
&& name_len) {
   value = TSMimeHdrFieldValueStringGet(buffer, hdr_loc, field_loc, -1, 
_len);
@@ -415,6 +415,11 @@ session_txn_handler(TSCont contp, TSEvent event, void 
*edata)
   }
 
   case TS_EVENT_HTTP_READ_REQUEST_HDR: {
+// This hook is registered globally, not at TS_EVENT_HTTP_SSN_START in
+// global_ssn_handler(). As such, this handler will be called with every
+// transaction. However, we know that we are dumping this transaction
+// because there is a ssnData associated with it.
+
 // We must grab the client request information before remap happens because
 // the remap process modifies the request buffer.
 TSMBuffer buffer;
@@ -437,6 +442,8 @@ session_txn_handler(TSCont contp, TSEvent event, void 
*edata)
 TSMLoc hdr_loc;
 if (TS_SUCCESS == TSHttpTxnClientReqGet(txnp, , _loc)) {
   txn_info += write_content_node(TSHttpTxnClientReqBodyBytesGet(txnp)) + 
"}";
+  TSHandleMLocRelease(buffer, TS_NULL_MLOC, hdr_loc);
+  buffer = nullptr;
 }
 if (TS_SUCCESS == TSHttpTxnServerReqGet(txnp, , _loc)) {
   TSDebug(PLUGIN_NAME, "Found proxy request");
@@ -585,7 +592,6 @@ global_ssn_handler(TSCont contp, TSEvent event, void *edata)
 TSMutexUnlock(ssnData->disk_io_mutex);
 
 TSHttpSsnHookAdd(ssnp, TS_HTTP_TXN_START_HOOK, ssnData->txn_cont);
-TSHttpSsnHookAdd(ssnp, TS_HTTP_READ_REQUEST_HDR_HOOK, ssnData->txn_cont);
 TSHttpSsnHookAdd(ssnp, TS_HTTP_TXN_CLOSE_HOOK, ssnData->txn_cont);
 break;
   }
@@ -673,6 +679,13 @@ TSPluginInit(int argc, const char *argv[])
 TSCont ssncont = TSContCreate(global_ssn_handler, nullptr);
 TSHttpHookAdd(TS_HTTP_SSN_START_HOOK, ssncont);
 TSHttpHookAdd(TS_HTTP_SSN_CLOSE_HOOK, ssncont);
+
+// Register the collecting of client-request headers at the global level so
+// we can process requests before other plugins. (Global hooks are
+// processed before session and transaction ones.)
+TSCont txn_cont = TSContCreate(session_txn_handler, nullptr);
+TSHttpHookAdd(TS_HTTP_READ_REQUEST_HDR_HOOK, txn_cont);
+
 TSLifecycleHookAdd(TS_LIFECYCLE_MSG_HOOK, ssncont);
 TSDebug(PLUGIN_NAME, "Initialized with sample pool size %" PRId64 " bytes 
and disk limit %" PRId64 " bytes",
 sample_pool_size.load(), max_disk_usage.load());

[trafficserver] branch master updated (51b8e9f -> 41b8ea0)

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git.


from 51b8e9f  Remove method that does nothing. (#6501)
 add 41b8ea0  traffic_dump: Make client-request gathered in a global hook 
(#6500)

No new revisions were added by this update.

Summary of changes:
 plugins/experimental/traffic_dump/traffic_dump.cc | 21 +
 1 file changed, 17 insertions(+), 4 deletions(-)

[trafficserver] branch master updated (afa11d9 -> 51b8e9f)

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git.


from afa11d9  Doc: traffic_manager - Add documentation for exp backoff and 
configuration
 add 51b8e9f  Remove method that does nothing. (#6501)

No new revisions were added by this update.

Summary of changes:
 include/tscpp/api/Url.h | 8 
 src/tscpp/api/Url.cc| 5 -
 src/tscpp/api/utils_internal.cc | 1 -
 3 files changed, 14 deletions(-)

[trafficserver] branch 9.0.x updated: Remove method that does nothing. (#6501)

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git


The following commit(s) were added to refs/heads/9.0.x by this push:
 new 0c88810  Remove method that does nothing. (#6501)
0c88810 is described below

commit 0c88810436264807fd36284fd7aa61d4c8cac526
Author: David Calavera 
AuthorDate: Tue Mar 10 19:29:47 2020 -0700

Remove method that does nothing. (#6501)

Long time ago, this method used to do something, but not anymore.

Signed-off-by: David Calavera 
(cherry picked from commit 51b8e9fe4cbe7183329eefee18f4b101d058d2f8)
---
 include/tscpp/api/Url.h | 8 
 src/tscpp/api/Url.cc| 5 -
 src/tscpp/api/utils_internal.cc | 1 -
 3 files changed, 14 deletions(-)

diff --git a/include/tscpp/api/Url.h b/include/tscpp/api/Url.h
index cf4b859..c01b8da 100644
--- a/include/tscpp/api/Url.h
+++ b/include/tscpp/api/Url.h
@@ -124,14 +124,6 @@ public:
*/
   void setPort(const uint16_t);
 
-  /**
-   * This method allows you to reset the url, this will force the Url to fully 
re-read all cached values.
-   * If this method is used on a Detached Requests' Url object it will 
completely destroy the values.
-   *
-   * \note This method should rarely be used.
-   */
-  void reset();
-
 private:
   bool isInitialized() const;
   void init(void *hdr_buf, void *url_loc);
diff --git a/src/tscpp/api/Url.cc b/src/tscpp/api/Url.cc
index a8f7144..c8e0430 100644
--- a/src/tscpp/api/Url.cc
+++ b/src/tscpp/api/Url.cc
@@ -63,11 +63,6 @@ bool inline Url::isInitialized() const
   return state_->hdr_buf_ && state_->url_loc_;
 }
 
-void
-Url::reset()
-{
-}
-
 std::string
 Url::getUrlString() const
 {
diff --git a/src/tscpp/api/utils_internal.cc b/src/tscpp/api/utils_internal.cc
index 923c48d..af76669 100644
--- a/src/tscpp/api/utils_internal.cc
+++ b/src/tscpp/api/utils_internal.cc
@@ -61,7 +61,6 @@ handleTransactionEvents(TSCont cont, TSEvent event, void 
*edata)
   utils::internal::setTransactionEvent(transaction, event);
   switch (event) {
   case TS_EVENT_HTTP_POST_REMAP:
-transaction.getClientRequest().getUrl().reset();
 // This is here to force a refresh of the cached client request url
 TSMBuffer hdr_buf;
 TSMLoc hdr_loc;

[trafficserver] branch 9.0.x updated: Make all_headers test more resilient to timimng

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git


The following commit(s) were added to refs/heads/9.0.x by this push:
 new 09a1738  Make all_headers test more resilient to timimng
09a1738 is described below

commit 09a17385a81b8bfe0d512abd80e3c83d0fd79503
Author: Susan Hinrichs 
AuthorDate: Mon Mar 9 15:20:18 2020 +

Make all_headers test more resilient to timimng

(cherry picked from commit 4f0a6be7fd38a09eb407fd872f6a492fbfcbef3f)
---
 tests/gold_tests/logging/all_headers.test.py  |  3 +--
 tests/gold_tests/logging/all_headers_sanitizer.py | 26 +--
 2 files changed, 21 insertions(+), 8 deletions(-)

diff --git a/tests/gold_tests/logging/all_headers.test.py 
b/tests/gold_tests/logging/all_headers.test.py
index 12b81f0..7e0f47e 100644
--- a/tests/gold_tests/logging/all_headers.test.py
+++ b/tests/gold_tests/logging/all_headers.test.py
@@ -96,8 +96,7 @@ tr.Processes.Default.ReturnCode = 0
 # Delay to allow TS to flush report to disk, then "sanitize" generated log.
 #
 tr = Test.AddTestRun()
-tr.DelayStart = 10
-tr.Processes.Default.Command = 'python3 {0} {4} < {2} | sh {1} > {3}'.format(
+tr.Processes.Default.Command = 'python3 {0} {2} {4} | sh {1} > {3}'.format(
 os.path.join(Test.TestDirectory, 'all_headers_sanitizer.py'),
 os.path.join(Test.TestDirectory, 'all_headers_sanitizer.sh'),
 os.path.join(ts.Variables.LOGDIR, 'test_all_headers.log'),
diff --git a/tests/gold_tests/logging/all_headers_sanitizer.py 
b/tests/gold_tests/logging/all_headers_sanitizer.py
index 2206b5e..64c22fd 100644
--- a/tests/gold_tests/logging/all_headers_sanitizer.py
+++ b/tests/gold_tests/logging/all_headers_sanitizer.py
@@ -19,6 +19,8 @@ Sanitize the ATS-generated custom log file from the 
all_headers test.
 
 import sys
 import re
+from os import path
+import time
 
 rexl = []
 rexl.append((re.compile(r"\{\{Date\}\:\{[^}]*\}\}"), "({__DATE__}}"))
@@ -29,15 +31,27 @@ rexl.append((re.compile(r"\{\{Server\}\:\{ECS [^}]*\}\}"), 
"({__ECS_SERVER__}}")
 rexl.append((re.compile(r"\{\{Via\}\:\{[^}]*\}\}"), "({__VIA__}}"))
 rexl.append((re.compile(r"\{\{Server\}\:\{ApacheTrafficServer/[0-9.]*\}\}"), 
"({__ATS2_SERVER__}}"))
 rexl.append((re.compile(r"\{\{Age\}\:\{[0-9]*\}\}"), "({__AGE__}}"))
-rexl.append((re.compile(r"\:" + sys.argv[1]), "__TS_PORT__")) # 1st and only 
argument is TS client port
+rexl.append((re.compile(r"\:" + sys.argv[2]), "__TS_PORT__")) # 1st and only 
argument is TS client port
 
 # Handle inconsistencies which I think are caused by different revisions of 
the standard Python http.server.HTTPServer class.
 
 rexl.append((re.compile(r'\{"359670651[^"]*"\}'), '{"{359670651__WEIRD__}"}'))
 rexl.append((re.compile(r'\{\{Accept-Ranges\}:\{bytes\}\}'), ''))
 
-for line in sys.stdin:
-for rex, subStr in rexl:
-line = rex.sub(subStr, line)
-
-print(line)
+# Loop until the file specified in argv[1] becomes availble
+filename = sys.argv[1]
+processed = False
+# Give up looking for file after 2 minutes
+limit_count = 0
+limit_max = 120
+while not processed and limit_count < limit_max:
+  limit_count += 1
+  if not path.exists(filename):
+time.sleep(1);
+  else:
+with open(filename, "r") as f:
+  processed = True
+  for line in f:
+for rex, subStr in rexl:
+  line = rex.sub(subStr, line)
+print(line)

[trafficserver] branch 9.0.x updated: Updated Changelog

2020-03-10 Thread bcall 
This is an automated email from the ASF dual-hosted git repository.

bcall pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git


The following commit(s) were added to refs/heads/9.0.x by this push:
 new 2d789c5  Updated Changelog
2d789c5 is described below

commit 2d789c5b82bab946123d98f9d8de15b711682e1a
Author: Bryan Call 
AuthorDate: Tue Mar 10 14:28:55 2020 -0700

Updated Changelog
---
 CHANGELOG-9.0.0 | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/CHANGELOG-9.0.0 b/CHANGELOG-9.0.0
index 6f11d8c..b05a399 100644
--- a/CHANGELOG-9.0.0
+++ b/CHANGELOG-9.0.0
@@ -940,7 +940,11 @@ Changes with Apache Traffic Server 9.0.0
   #6441 - Cleaned up ssl-post
   #6446 - Avoid a weird name collision between HRW and tscore
   #6450 - Adjust consume logic in data frame read
+  #6454 - traffic_dump post_process.py
   #6457 - Fix port selection for ssl ipv6
+  #6459 - Add more flexible error handling when open a config file.
+  #6460 - Make traffic_manager be flexible when opening config files.
+  #6461 - Reduce minimum config files needed for golden_tests.
   #6462 - x-remap ignoring age in gold file
   #6463 - [CPPAPI] Provide access to TSRemapRequestInfo in RemapPlugins.
   #6464 - cache_range_requests plugin: detect and handle TSCacheUrlSet 
failures…
@@ -950,5 +954,7 @@ Changes with Apache Traffic Server 9.0.0
   #6471 - Removes some things deprecated from older versions
   #6474 - Remove --read_core option
   #6475 - Remove noisy mutex warning
+  #6477 - traffic_dump: Fixing content:size collection.
   #6480 - Remove some outdated files.
   #6488 - Bikeshedding some code structures for reloadable plugins config
+  #6492 - Add null check to fix error tunnel crash

[trafficserver] 02/02: traffic_dump: Fixing content:size collection.

2020-03-10 Thread bcall 
This is an automated email from the ASF dual-hosted git repository.

bcall pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git

commit 16efdabf738a80af7edbb7566a354e5076b089e1
Author: bneradt 
AuthorDate: Thu Feb 27 23:41:28 2020 +

traffic_dump: Fixing content:size collection.

traffic_dump was collecting the client-request body information too
early, resulting in an incorrect content:size value (it was always
zero). This holds off on collecting the body size until later so the
value is accurate. A traffic_dump AuTest is being added to verify
correct behavior with requests containing bodies.

Also, this updates the Traffic Dump post_process.py script to add
proxy-request and server-response nodes to transactions without them
because the proxy replied locally to the request. This is useful in
replay scenarios in which the test version of Traffic Server likely
won't have the cached responses.

(cherry picked from commit ddaf9e5f42930d01570904214cd6086c0f1de24b)
---
 plugins/experimental/traffic_dump/post_process.py  | 59 +-
 plugins/experimental/traffic_dump/traffic_dump.cc  | 51 +--
 .../traffic_dump/gold/post_with_body.gold  |  8 +++
 .../pluginTest/traffic_dump/traffic_dump.test.py   | 45 +
 .../pluginTest/traffic_dump/verify_replay.py   | 28 +-
 5 files changed, 162 insertions(+), 29 deletions(-)

diff --git a/plugins/experimental/traffic_dump/post_process.py 
b/plugins/experimental/traffic_dump/post_process.py
index dd4a7ea..d2dbe96 100755
--- a/plugins/experimental/traffic_dump/post_process.py
+++ b/plugins/experimental/traffic_dump/post_process.py
@@ -109,11 +109,14 @@ def verify_response(response):
 raise VerifyResponseError("Response did not have a status.")
 
 
-def verify_transaction(transaction):
+def verify_transaction(transaction, fabricate_proxy_requests=False):
 """ Function to verify that a transaction looks complete.
 
 Args:
 transaction (json object)
+fabricate_proxy_requests (bool) Whether the post-processor should
+  fabricate proxy requests if they don't exist because the proxy served
+  the response locally.
 
 Raises:
 VerifySessionError if there is no transaction.
@@ -128,12 +131,19 @@ def verify_transaction(transaction):
 else:
 verify_request(transaction["client-request"])
 
+if "proxy-request" not in transaction and fabricate_proxy_requests:
+if "proxy-response" not in transaction:
+raise VerifyRequestError('proxy-response not found in transaction 
with a client-request')
+transaction["proxy-request"] = transaction["client-request"]
+if "server-response" not in transaction:
+transaction["server-response"] = transaction["proxy-response"]
+
 # proxy-response nodes can be empty.
 if "proxy-response" not in transaction:
 raise VerifyResponseError('proxy-response not found in transaction')
 
 if "proxy-request" in transaction or "server-response" in transaction:
-# proxy-request nodes can be empty.
+# proxy-request nodes can be empty, so no need to verify_response.
 if "proxy-request" not in transaction:
 raise VerifyRequestError('proxy-request not found in transaction')
 
@@ -143,13 +153,16 @@ def verify_transaction(transaction):
 verify_response(transaction["server-response"])
 
 
-def verify_session(session):
+def verify_session(session, fabricate_proxy_requests=False):
 """ Function to verify that a session looks complete.
 
 A valid session contains a valid list of transactions.
 
 Args:
 transaction (json object)
+fabricate_proxy_requests (bool) Whether the post-processor should
+  fabricate proxy requests if they don't exist because the proxy served
+  the response locally.
 
 Raises:
 VerifyError if there is a problem with the session.
@@ -159,7 +172,7 @@ def verify_session(session):
 if "transactions" not in session or not session["transactions"]:
 raise VerifySessionError('No transactions found in session.')
 for transaction in session["transactions"]:
-verify_transaction(transaction)
+verify_transaction(transaction, fabricate_proxy_requests)
 
 
 def write_sessions(sessions, filename, indent):
@@ -203,14 +216,14 @@ def parse_json(replay_file):
 try:
 parsed_json = json.load(fd)
 except Exception as e:
-message = e.split(':')[0]
-logging.exception("Failed to load %s as a JSON object.", replay_file)
+message = e.msg.split(':')[0]
+logging.error("Failed to load %s as a JSON object: %s", replay_file, e)
 raise ParseJSONError(message)
 
 return parsed_json
 
 
-def readAndCombine(replay_dir, num_sessions_per_file, indent, out_dir):
+def

[trafficserver] 01/02: traffic_dump post_process.py

2020-03-10 Thread bcall 
This is an automated email from the ASF dual-hosted git repository.

bcall pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git

commit 801243747c28224a22737a714e1125e8754a207e
Author: bneradt 
AuthorDate: Tue Feb 25 16:48:07 2020 +

traffic_dump post_process.py

(cherry picked from commit 17c48bfc16980496129cff2925ce46b6b0b3b05c)
---
 plugins/experimental/traffic_dump/post_process.py | 392 ++
 1 file changed, 392 insertions(+)

diff --git a/plugins/experimental/traffic_dump/post_process.py 
b/plugins/experimental/traffic_dump/post_process.py
new file mode 100755
index 000..dd4a7ea
--- /dev/null
+++ b/plugins/experimental/traffic_dump/post_process.py
@@ -0,0 +1,392 @@
+#!/usr/bin/env python3
+'''
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+'''
+
+from collections import defaultdict
+from copy import deepcopy
+from queue import Queue
+from threading import Thread
+import argparse
+import json
+import logging
+import os
+import sys
+
+description = '''
+Post process replay files produced by traffic_dump and clean it of any
+incomplete transactions or sessions which got partially written because Traffic
+Server was interrupted mid-connection. This also merges sessions to the same
+client and, by default, formats the output files with human readable spacing.
+'''
+
+
+# Base replay file template with basic elements
+TEMPLATE = json.loads('{"meta": {"version":"1.0"},"sessions":[]}')
+
+
+class PostProcessError(Exception):
+''' Base class for post processing errors.
+'''
+def __init__(self, message=None):
+self.message = message
+
+def __str__(self, *args):
+if self.message:
+return self.message
+else:
+return 'PostProcessError raised'
+
+
+class VerifyError(PostProcessError):
+''' Base class for node node verification errors.
+'''
+pass
+
+
+class VerifyRequestError(VerifyError):
+''' There was a problem verifying a request node.
+'''
+pass
+
+
+class VerifyResponseError(VerifyError):
+''' There was a problem verifying a response node.
+'''
+pass
+
+
+class VerifySessionError(VerifyError):
+''' There was a problem verifying a session node.
+'''
+pass
+
+
+def verify_request(request):
+""" Function to verify request with method, url, and headers
+Args:
+request (json object)
+
+Raises:
+VerifyRequestError if there is a problem with the request.
+"""
+if not request:
+raise VerifyRequestError('No request found.')
+if "method" not in request or not request["method"]:
+raise VerifyRequestError("Request did not have a method.")
+if "url" not in request or not request["url"]:
+raise VerifyRequestError("Request did not have a url.")
+if "headers" not in request or not request["headers"]:
+raise VerifyRequestError("Request did not have headers.")
+
+
+def verify_response(response):
+""" Function to verify response with status
+Args:
+response (json object)
+
+Raises:
+VerifyResponseError if there is a problem with the response.
+"""
+if not response:
+raise VerifyResponseError("No response found.")
+if "status" not in response or not response["status"]:
+raise VerifyResponseError("Response did not have a status.")
+
+
+def verify_transaction(transaction):
+""" Function to verify that a transaction looks complete.
+
+Args:
+transaction (json object)
+
+Raises:
+VerifySessionError if there is no transaction.
+VerifyRequestError if there is a problem with a request.
+VerifyResponseError if there is a problem with a response.
+"""
+if not transaction:
+raise VerifySessionError('No transaction found in the session.')
+
+if "client-request" not in transaction:
+raise VerifyRequestError('client-request not found in transaction')
+else:
+verify_request(transaction["client-request"])
+
+# proxy-response nodes can be empty.
+if "proxy-response" not in transaction:
+raise VerifyResponseError('proxy-response not found in transaction')
+
+if "proxy-request" in transaction or

[trafficserver] branch 9.0.x updated (28ae091 -> 16efdab)

2020-03-10 Thread bcall 
This is an automated email from the ASF dual-hosted git repository.

bcall pushed a change to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git.


from 28ae091  Reduce minimum config files neeed for golden_tests.
 new 8012437  traffic_dump post_process.py
 new 16efdab  traffic_dump: Fixing content:size collection.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 plugins/experimental/traffic_dump/post_process.py  | 425 +
 plugins/experimental/traffic_dump/traffic_dump.cc  |  51 ++-
 .../{explicit_target.gold => post_with_body.gold}  |   2 +-
 .../pluginTest/traffic_dump/traffic_dump.test.py   |  45 +++
 .../pluginTest/traffic_dump/verify_replay.py   |  28 +-
 5 files changed, 534 insertions(+), 17 deletions(-)
 create mode 100755 plugins/experimental/traffic_dump/post_process.py
 copy tests/gold_tests/pluginTest/traffic_dump/gold/{explicit_target.gold => 
post_with_body.gold} (63%)

[trafficserver] branch master updated (4f0a6be -> afa11d9)

2020-03-10 Thread amc 
This is an automated email from the ASF dual-hosted git repository.

amc pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git.


from 4f0a6be  Make all_headers test more resilient to timimng
 add afa11d9  Doc: traffic_manager - Add documentation for exp backoff and 
configuration

No new revisions were added by this update.

Summary of changes:
 doc/admin-guide/files/records.config.en.rst| 18 +-
 doc/appendices/command-line/traffic_manager.en.rst | 14 ++
 2 files changed, 31 insertions(+), 1 deletion(-)

[trafficserver] branch 9.0.x updated: Reduce minimum config files neeed for golden_tests.

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git


The following commit(s) were added to refs/heads/9.0.x by this push:
 new 28ae091  Reduce minimum config files neeed for golden_tests.
28ae091 is described below

commit 28ae0913ab4dc49deec3789cd12aeb3cd848af76
Author: Damian Meden 
AuthorDate: Thu Feb 27 14:31:32 2020 +

Reduce minimum config files neeed for golden_tests.

(cherry picked from commit c4aef5549d52050d8c375ecdf6ede571932178aa)
---
 tests/gold_tests/autest-site/min_cfg/cache.config | 1 -
 tests/gold_tests/autest-site/min_cfg/hosting.config   | 1 -
 tests/gold_tests/autest-site/min_cfg/logging.yaml | 1 -
 tests/gold_tests/autest-site/min_cfg/parent.config| 1 -
 tests/gold_tests/autest-site/min_cfg/plugin.config| 1 -
 tests/gold_tests/autest-site/min_cfg/readme.txt   | 5 +++--
 tests/gold_tests/autest-site/min_cfg/records.config   | 1 -
 tests/gold_tests/autest-site/min_cfg/remap.config | 1 -
 tests/gold_tests/autest-site/min_cfg/sni.yaml | 1 -
 tests/gold_tests/autest-site/min_cfg/socks.config | 1 -
 tests/gold_tests/autest-site/min_cfg/splitdns.config  | 1 -
 tests/gold_tests/autest-site/min_cfg/ssl_multicert.config | 1 -
 tests/gold_tests/autest-site/min_cfg/volume.config| 1 -
 13 files changed, 3 insertions(+), 14 deletions(-)

diff --git a/tests/gold_tests/autest-site/min_cfg/cache.config 
b/tests/gold_tests/autest-site/min_cfg/cache.config
deleted file mode 100644
index 6ac656b..000
--- a/tests/gold_tests/autest-site/min_cfg/cache.config
+++ /dev/null
@@ -1 +0,0 @@
-# this files just needs to exist
\ No newline at end of file
diff --git a/tests/gold_tests/autest-site/min_cfg/hosting.config 
b/tests/gold_tests/autest-site/min_cfg/hosting.config
deleted file mode 100644
index 6ac656b..000
--- a/tests/gold_tests/autest-site/min_cfg/hosting.config
+++ /dev/null
@@ -1 +0,0 @@
-# this files just needs to exist
\ No newline at end of file
diff --git a/tests/gold_tests/autest-site/min_cfg/logging.yaml 
b/tests/gold_tests/autest-site/min_cfg/logging.yaml
deleted file mode 100644
index 6ac656b..000
--- a/tests/gold_tests/autest-site/min_cfg/logging.yaml
+++ /dev/null
@@ -1 +0,0 @@
-# this files just needs to exist
\ No newline at end of file
diff --git a/tests/gold_tests/autest-site/min_cfg/parent.config 
b/tests/gold_tests/autest-site/min_cfg/parent.config
deleted file mode 100644
index 6ac656b..000
--- a/tests/gold_tests/autest-site/min_cfg/parent.config
+++ /dev/null
@@ -1 +0,0 @@
-# this files just needs to exist
\ No newline at end of file
diff --git a/tests/gold_tests/autest-site/min_cfg/plugin.config 
b/tests/gold_tests/autest-site/min_cfg/plugin.config
deleted file mode 100644
index 6ac656b..000
--- a/tests/gold_tests/autest-site/min_cfg/plugin.config
+++ /dev/null
@@ -1 +0,0 @@
-# this files just needs to exist
\ No newline at end of file
diff --git a/tests/gold_tests/autest-site/min_cfg/readme.txt 
b/tests/gold_tests/autest-site/min_cfg/readme.txt
index 8060e96..369a4db 100644
--- a/tests/gold_tests/autest-site/min_cfg/readme.txt
+++ b/tests/gold_tests/autest-site/min_cfg/readme.txt
@@ -1,3 +1,4 @@
-Contains the miniumn set of config file and setting to allow trafficserver to 
start in a usable way.
+Contains the minimum set of config file and setting to allow trafficserver to 
start in a usable way.
 The goal is to remove the need for any of these files to exist.
-Each of these files should provide an understanding of what we need to fix in 
the code
+ip_allow.yaml is needed to allow tests the minimum access to traffic_server.
+storage.config needs to exist for now.
diff --git a/tests/gold_tests/autest-site/min_cfg/records.config 
b/tests/gold_tests/autest-site/min_cfg/records.config
deleted file mode 100644
index 8d97ae7..000
--- a/tests/gold_tests/autest-site/min_cfg/records.config
+++ /dev/null
@@ -1 +0,0 @@
-# some stuff
diff --git a/tests/gold_tests/autest-site/min_cfg/remap.config 
b/tests/gold_tests/autest-site/min_cfg/remap.config
deleted file mode 100644
index 6ac656b..000
--- a/tests/gold_tests/autest-site/min_cfg/remap.config
+++ /dev/null
@@ -1 +0,0 @@
-# this files just needs to exist
\ No newline at end of file
diff --git a/tests/gold_tests/autest-site/min_cfg/sni.yaml 
b/tests/gold_tests/autest-site/min_cfg/sni.yaml
deleted file mode 100644
index 6ac656b..000
--- a/tests/gold_tests/autest-site/min_cfg/sni.yaml
+++ /dev/null
@@ -1 +0,0 @@
-# this files just needs to exist
\ No newline at end of file
diff --git a/tests/gold_tests/autest-site/min_cfg/socks.config 
b/tests/gold_tests/autest-site/min_cfg/socks.config
deleted file mode 100644
index 6ac656b..000
--- a/tests/gold_tests/autest-site/min_cfg/socks.config
+++ /dev/null
@@ -1 +0,0 @@
-# this files just needs to exist
\ No newline at end of file
diff

[trafficserver] 02/02: Fix clang-format

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git

commit 2186c2993cf668cb5b82975b72660fef1ad35273
Author: Damian Meden 
AuthorDate: Fri Feb 28 10:24:10 2020 +

Fix clang-format

(cherry picked from commit b118bd96b8966208e2c5306a199bee97c8b5b501)
---
 mgmt/ConfigManager.cc | 5 +++--
 mgmt/ConfigManager.h  | 3 ++-
 mgmt/FileManager.cc   | 3 ++-
 mgmt/FileManager.h| 3 ++-
 src/traffic_manager/AddConfigFilesHere.cc | 5 ++---
 5 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/mgmt/ConfigManager.cc b/mgmt/ConfigManager.cc
index b2dfe79..19304e0 100644
--- a/mgmt/ConfigManager.cc
+++ b/mgmt/ConfigManager.cc
@@ -42,8 +42,9 @@
 #define TS_ARCHIVE_STAT_MTIME(t) ((t).st_mtime * 10)
 #endif
 
-ConfigManager::ConfigManager(const char *fileName_, const char *configName_, 
bool root_access_needed_, bool isRequired_, ConfigManager *parentConfig_)
-  : root_access_needed(root_access_needed_),  isRequired(isRequired_), 
parentConfig(parentConfig_)
+ConfigManager::ConfigManager(const char *fileName_, const char *configName_, 
bool root_access_needed_, bool isRequired_,
+ ConfigManager *parentConfig_)
+  : root_access_needed(root_access_needed_), isRequired(isRequired_), 
parentConfig(parentConfig_)
 {
   ExpandingArray existVer(25, true); // Existing versions
   struct stat fileInfo;
diff --git a/mgmt/ConfigManager.h b/mgmt/ConfigManager.h
index f7b3e3e..b10a9ae 100644
--- a/mgmt/ConfigManager.h
+++ b/mgmt/ConfigManager.h
@@ -54,7 +54,8 @@ class ConfigManager
 {
 public:
   // fileName_ should be rooted or a base file name.
-  ConfigManager(const char *fileName_, const char *configName_, bool 
root_access_needed, bool isRequired_, ConfigManager *parentConfig_);
+  ConfigManager(const char *fileName_, const char *configName_, bool 
root_access_needed, bool isRequired_,
+ConfigManager *parentConfig_);
   ~ConfigManager();
 
   // Manual take out of lock required
diff --git a/mgmt/FileManager.cc b/mgmt/FileManager.cc
index e38f6d3..fce09a0 100644
--- a/mgmt/FileManager.cc
+++ b/mgmt/FileManager.cc
@@ -102,7 +102,8 @@ FileManager::addFile(const char *fileName, const char 
*configName, bool root_acc
 
 // caller must hold the lock
 void
-FileManager::addFileHelper(const char *fileName, const char *configName, bool 
root_access_needed, bool isRequired, ConfigManager *parentConfig)
+FileManager::addFileHelper(const char *fileName, const char *configName, bool 
root_access_needed, bool isRequired,
+   ConfigManager *parentConfig)
 {
   ink_assert(fileName != nullptr);
 
diff --git a/mgmt/FileManager.h b/mgmt/FileManager.h
index 8d9a899..64725b6 100644
--- a/mgmt/FileManager.h
+++ b/mgmt/FileManager.h
@@ -94,7 +94,8 @@ private:
   ink_mutex cbListLock; // Protects the CallBack List
   DLL cblist;
   std::unordered_map bindings;
-  void addFileHelper(const char *fileName, const char *configName, bool 
root_access_needed, bool isRequired, ConfigManager *parentConfig);
+  void addFileHelper(const char *fileName, const char *configName, bool 
root_access_needed, bool isRequired,
+ ConfigManager *parentConfig);
 };
 
 void initializeRegistry(); // implemented in AddConfigFilesHere.cc
diff --git a/src/traffic_manager/AddConfigFilesHere.cc 
b/src/traffic_manager/AddConfigFilesHere.cc
index 1f9d2da..3fdf572 100644
--- a/src/traffic_manager/AddConfigFilesHere.cc
+++ b/src/traffic_manager/AddConfigFilesHere.cc
@@ -29,9 +29,8 @@
 
 extern FileManager *configFiles;
 
-
-static constexpr bool REQUIRED { true };
-static constexpr bool NOT_REQUIRED { false };
+static constexpr bool REQUIRED{true};
+static constexpr bool NOT_REQUIRED{false};
 /
  *
  *  AddConfigFilesHere.cc - Structs for config files and

[trafficserver] branch 9.0.x updated (51c3ea2 -> 2186c29)

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a change to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git.


from 51c3ea2  Use same filename variable to keep consistency
 new 1b4c237  Make traffic_manager be flexible when opening config files.
 new 2186c29  Fix clang-format

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 mgmt/ConfigManager.cc | 14 +++--
 mgmt/ConfigManager.h  | 10 -
 mgmt/FileManager.cc   | 12 ++-
 mgmt/FileManager.h|  8 +---
 src/traffic_manager/AddConfigFilesHere.cc | 34 ---
 5 files changed, 47 insertions(+), 31 deletions(-)

[trafficserver] 01/02: Make traffic_manager be flexible when opening config files.

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git

commit 1b4c237c9da474c7016e01d7039b56856f26af1d
Author: Damian Meden 
AuthorDate: Thu Feb 27 13:34:34 2020 +

Make traffic_manager be flexible when opening config files.

(cherry picked from commit 9074db07b86f249dab9c3a5a8809ffb6318912cd)
---
 mgmt/ConfigManager.cc | 13 ++--
 mgmt/ConfigManager.h  |  9 +++-
 mgmt/FileManager.cc   | 11 +-
 mgmt/FileManager.h|  7 ---
 src/traffic_manager/AddConfigFilesHere.cc | 35 +--
 5 files changed, 44 insertions(+), 31 deletions(-)

diff --git a/mgmt/ConfigManager.cc b/mgmt/ConfigManager.cc
index 1eb94aa..b2dfe79 100644
--- a/mgmt/ConfigManager.cc
+++ b/mgmt/ConfigManager.cc
@@ -42,8 +42,8 @@
 #define TS_ARCHIVE_STAT_MTIME(t) ((t).st_mtime * 10)
 #endif
 
-ConfigManager::ConfigManager(const char *fileName_, const char *configName_, 
bool root_access_needed_, ConfigManager *parentConfig_)
-  : root_access_needed(root_access_needed_), parentConfig(parentConfig_)
+ConfigManager::ConfigManager(const char *fileName_, const char *configName_, 
bool root_access_needed_, bool isRequired_, ConfigManager *parentConfig_)
+  : root_access_needed(root_access_needed_),  isRequired(isRequired_), 
parentConfig(parentConfig_)
 {
   ExpandingArray existVer(25, true); // Existing versions
   struct stat fileInfo;
@@ -59,14 +59,15 @@ ConfigManager::ConfigManager(const char *fileName_, const 
char *configName_, boo
   configName = ats_strdup(configName_);
 
   ink_mutex_init();
-
   // Check to make sure that our configuration file exists
   //
   if (statFile() < 0) {
-// If we can't find an active version because there is none we have a hard 
failure.
-mgmt_fatal(0, "[ConfigManager::ConfigManager] Unable to find configuration 
file %s.\n\tStat failed : %s\n", fileName,
-   strerror(errno));
+mgmt_log("[ConfigManager::ConfigManager] %s Unable to load: %s", fileName, 
strerror(errno));
 
+if (isRequired) {
+  mgmt_fatal(0, "[ConfigManager::ConfigManager] Unable to open required 
configuration file %s.\n\tStat failed : %s\n", fileName,
+ strerror(errno));
+}
   } else {
 fileLastModified = TS_ARCHIVE_STAT_MTIME(fileInfo);
   }
diff --git a/mgmt/ConfigManager.h b/mgmt/ConfigManager.h
index f32cb2a..f7b3e3e 100644
--- a/mgmt/ConfigManager.h
+++ b/mgmt/ConfigManager.h
@@ -54,7 +54,7 @@ class ConfigManager
 {
 public:
   // fileName_ should be rooted or a base file name.
-  ConfigManager(const char *fileName_, const char *configName_, bool 
root_access_needed, ConfigManager *parentConfig_);
+  ConfigManager(const char *fileName_, const char *configName_, bool 
root_access_needed, bool isRequired_, ConfigManager *parentConfig_);
   ~ConfigManager();
 
   // Manual take out of lock required
@@ -104,6 +104,12 @@ public:
 return root_access_needed;
   }
 
+  bool
+  getIsRequired() const
+  {
+return isRequired;
+  }
+
   FileManager *configFiles = nullptr; // Manager to notify on an update.
 
   // noncopyable
@@ -117,6 +123,7 @@ private:
   char *fileName;
   char *configName;
   bool root_access_needed;
+  bool isRequired;
   ConfigManager *parentConfig;
   time_t fileLastModified = 0;
 };
diff --git a/mgmt/FileManager.cc b/mgmt/FileManager.cc
index c23f17f..e38f6d3 100644
--- a/mgmt/FileManager.cc
+++ b/mgmt/FileManager.cc
@@ -92,20 +92,21 @@ FileManager::registerCallback(FileCallbackFunc func)
 //  Pointers to the new objects are stored in the bindings hashtable
 //
 void
-FileManager::addFile(const char *fileName, const char *configName, bool 
root_access_needed, ConfigManager *parentConfig)
+FileManager::addFile(const char *fileName, const char *configName, bool 
root_access_needed, bool isRequired,
+ ConfigManager *parentConfig)
 {
   ink_mutex_acquire();
-  addFileHelper(fileName, configName, root_access_needed, parentConfig);
+  addFileHelper(fileName, configName, root_access_needed, isRequired, 
parentConfig);
   ink_mutex_release();
 }
 
 // caller must hold the lock
 void
-FileManager::addFileHelper(const char *fileName, const char *configName, bool 
root_access_needed, ConfigManager *parentConfig)
+FileManager::addFileHelper(const char *fileName, const char *configName, bool 
root_access_needed, bool isRequired, ConfigManager *parentConfig)
 {
   ink_assert(fileName != nullptr);
 
-  ConfigManager *rb = new ConfigManager(fileName, configName, 
root_access_needed, parentConfig);
+  ConfigManager *rb = new ConfigManager(fileName, configName, 
root_access_needed, isRequired, parentConfig);
   rb->configFiles   = this;
 
   bindings.emplace(rb->getFileName(), rb);
@@ -257,7 +258,7 @@ FileManager::configFileChild(const char *parent, const char 
*child)
   ink_mutex_acquire();

[trafficserver] 02/03: Fix clang-format on SSLUtils.cc

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git

commit ff03254170d2ade0726e7c16e2c8d670d6b7bd21
Author: Damian Meden 
AuthorDate: Thu Feb 27 19:49:11 2020 +

Fix clang-format on SSLUtils.cc

(cherry picked from commit 094fbbe9753d6ab156ff7830930e2367c0af1cd9)
---
 iocore/net/SSLUtils.cc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/iocore/net/SSLUtils.cc b/iocore/net/SSLUtils.cc
index 99433fc..f7fb11f 100644
--- a/iocore/net/SSLUtils.cc
+++ b/iocore/net/SSLUtils.cc
@@ -1592,8 +1592,8 @@ SSLMultiCertConfigLoader::load(SSLCertLookup *lookup)
 {
   const SSLConfigParams *params = this->_params;
 
-  char *tok_state = nullptr;
-  char *line  = nullptr;
+  char *tok_state   = nullptr;
+  char *line= nullptr;
   unsigned line_num = 0;
   matcher_line line_info;

[trafficserver] 03/03: Use same filename variable to keep consistency

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git

commit 51c3ea2532e99707855d50d8eda686c4fca94075
Author: Damian Meden 
AuthorDate: Tue Mar 3 11:22:37 2020 +

Use same filename variable to keep consistency

(cherry picked from commit 7f30e24a39c48a0b79f942156d5f27651f3f7552)
---
 iocore/cache/CacheHosting.cc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/iocore/cache/CacheHosting.cc b/iocore/cache/CacheHosting.cc
index 31c9bc9..d968d5f 100644
--- a/iocore/cache/CacheHosting.cc
+++ b/iocore/cache/CacheHosting.cc
@@ -610,7 +610,7 @@ ConfigVolumes::read_config_file()
   Warning("Cannot open the config file: %s - %s", (const char 
*)config_path, strerror(ec.value()));
   break;
 default:
-  Error("%s failed to load: %s", ts::filename::VOLUME, 
strerror(ec.value()));
+  Error("%s failed to load: %s", (const char *)config_path, 
strerror(ec.value()));
   return;
 }
   }

[trafficserver] 01/03: Add more flexible error handling when open a config file.

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git

commit 73e74ff07ad63f412bd09ff6900a34c37023b9c7
Author: Damian Meden 
AuthorDate: Thu Feb 27 15:20:08 2020 +

Add more flexible error handling when open a config file.

(cherry picked from commit 853b010e31c6d9847e394ea2f12942ad85034aee)
---
 iocore/cache/CacheHosting.cc| 47 +++--
 iocore/dns/SplitDNS.cc  |  3 +--
 iocore/net/SSLUtils.cc  | 22 +++
 proxy/ControlMatcher.cc | 24 +++--
 proxy/http/remap/RemapConfig.cc | 18 +++-
 5 files changed, 67 insertions(+), 47 deletions(-)

diff --git a/iocore/cache/CacheHosting.cc b/iocore/cache/CacheHosting.cc
index 24930c7..31c9bc9 100644
--- a/iocore/cache/CacheHosting.cc
+++ b/iocore/cache/CacheHosting.cc
@@ -27,6 +27,7 @@
 #include "tscore/Tokenizer.h"
 #include "tscore/Regression.h"
 #include "tscore/Filenames.h"
+#include "tscore/ts_file.h"
 
 extern int gndisks;
 
@@ -396,20 +397,22 @@ CacheHostTable::BuildTableFromString(const char 
*config_file_path, char *file_bu
 int
 CacheHostTable::BuildTable(const char *config_file_path)
 {
-  char *file_buf;
-  int ret;
+  std::error_code ec;
+  std::string content{ts::file::load(ts::file::path{config_file_path}, ec)};
 
-  file_buf = readIntoBuffer(config_file_path, matcher_name, nullptr);
-
-  if (file_buf == nullptr) {
-Warning("Cannot read the config file: %s", config_file_path);
-gen_host_rec.Init(type);
-return 0;
+  if (ec) {
+switch (ec.value()) {
+case ENOENT:
+  Warning("Cannot open the config file: %s - %s", config_file_path, 
strerror(ec.value()));
+  break;
+default:
+  Error("%s failed to load: %s", config_file_path, strerror(ec.value()));
+  gen_host_rec.Init(type);
+  return 0;
+}
   }
 
-  ret = BuildTableFromString(config_file_path, file_buf);
-  ats_free(file_buf);
-  return ret;
+  return BuildTableFromString(config_file_path, content.data());
 }
 
 int
@@ -592,23 +595,27 @@ void
 ConfigVolumes::read_config_file()
 {
   ats_scoped_str config_path;
-  char *file_buf;
 
   config_path = RecConfigReadConfigPath("proxy.config.cache.volume_filename");
   ink_release_assert(config_path);
 
   Note("%s loading ...", ts::filename::VOLUME);
 
-  file_buf = readIntoBuffer(config_path, "[CacheVolition]", nullptr);
-  if (file_buf == nullptr) {
-Error("%s failed to load", ts::filename::VOLUME);
-Warning("Cannot read the config file: %s", (const char *)config_path);
-return;
-  }
+  std::error_code ec;
+  std::string content{ts::file::load(ts::file::path{config_path}, ec)};
 
-  BuildListFromString(config_path, file_buf);
-  ats_free(file_buf);
+  if (ec) {
+switch (ec.value()) {
+case ENOENT:
+  Warning("Cannot open the config file: %s - %s", (const char 
*)config_path, strerror(ec.value()));
+  break;
+default:
+  Error("%s failed to load: %s", ts::filename::VOLUME, 
strerror(ec.value()));
+  return;
+}
+  }
 
+  BuildListFromString(config_path, content.data());
   Note("volume.config finished loading");
 
   return;
diff --git a/iocore/dns/SplitDNS.cc b/iocore/dns/SplitDNS.cc
index 2ad8c24..3606adc 100644
--- a/iocore/dns/SplitDNS.cc
+++ b/iocore/dns/SplitDNS.cc
@@ -139,8 +139,7 @@ SplitDNSConfig::reconfigure()
   params->m_DNSSrvrTable= new 
DNS_table("proxy.config.dns.splitdns.filename", modulePrefix, _dest_tags);
 
   if (nullptr == params->m_DNSSrvrTable || (0 == 
params->m_DNSSrvrTable->getEntryCount())) {
-Error("%s failed to load", ts::filename::SPLITDNS);
-Warning("No NAMEDs provided! Disabling SplitDNS");
+Warning("Failed to load %s - No NAMEDs provided! Disabling SplitDNS", 
ts::filename::SPLITDNS);
 gsplit_dns_enabled = 0;
 delete params;
 return;
diff --git a/iocore/net/SSLUtils.cc b/iocore/net/SSLUtils.cc
index 779f030..99433fc 100644
--- a/iocore/net/SSLUtils.cc
+++ b/iocore/net/SSLUtils.cc
@@ -29,6 +29,7 @@
 #include "tscore/ink_mutex.h"
 #include "tscore/Filenames.h"
 #include "records/I_RecHttp.h"
+#include "tscore/ts_file.h"
 
 #include "P_Net.h"
 #include "InkAPIInternal.h"
@@ -1593,7 +1594,6 @@ SSLMultiCertConfigLoader::load(SSLCertLookup *lookup)
 
   char *tok_state = nullptr;
   char *line  = nullptr;
-  ats_scoped_str file_buf;
   unsigned line_num = 0;
   matcher_line line_info;
 
@@ -1601,13 +1601,17 @@ SSLMultiCertConfigLoader::load(SSLCertLookup *lookup)
 
   Note("%s loading ...", ts::filename::SSL_MULTICERT);
 
-  if (params->configFilePath) {
-file_buf = readIntoBuffer(params->configFilePath, __func__, nullptr);
-  }
-
-  if (!file_buf) {
-Error("failed to read SSL certificate configuration from %s", 
params->configFilePath);
-return false;
+  std::error_code ec;
+  std::string content{ts::file::load(ts::file::path{params->configFilePath}, 
ec)};
+

[trafficserver] branch 9.0.x updated (8d54883 -> 51c3ea2)

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a change to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git.


from 8d54883  Add null check to fix error tunnel crash
 new 73e74ff  Add more flexible error handling when open a config file.
 new ff03254  Fix clang-format on SSLUtils.cc
 new 51c3ea2  Use same filename variable to keep consistency

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 iocore/cache/CacheHosting.cc| 47 +++--
 iocore/dns/SplitDNS.cc  |  3 +--
 iocore/net/SSLUtils.cc  | 26 +--
 proxy/ControlMatcher.cc | 24 +++--
 proxy/http/remap/RemapConfig.cc | 18 +++-
 5 files changed, 69 insertions(+), 49 deletions(-)

[trafficserver] branch 9.0.x updated: Add null check to fix error tunnel crash

2020-03-10 Thread zwoop 
This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a commit to branch 9.0.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git


The following commit(s) were added to refs/heads/9.0.x by this push:
 new 8d54883  Add null check to fix error tunnel crash
8d54883 is described below

commit 8d548838c54b2fcead55ada80a71955daf3eeb30
Author: Susan Hinrichs 
AuthorDate: Mon Mar 9 13:52:03 2020 +

Add null check to fix error tunnel crash

(cherry picked from commit 6b634b2c5af6467a9db0aa6f536b19c70a80ccae)
---
 proxy/http/HttpTunnel.cc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proxy/http/HttpTunnel.cc b/proxy/http/HttpTunnel.cc
index 56c1a38..4e69d2a 100644
--- a/proxy/http/HttpTunnel.cc
+++ b/proxy/http/HttpTunnel.cc
@@ -1473,7 +1473,7 @@ HttpTunnel::finish_all_internal(HttpTunnelProducer *p, 
bool chain)
 
   if (action == TCA_PASSTHRU_CHUNKED_CONTENT) {
 // if the only chunked data was in the initial read, make sure we don't 
consume too much
-if (p->bytes_read == 0) {
+if (p->bytes_read == 0 && p->buffer_start != nullptr) {
   int num_read = p->buffer_start->read_avail() - 
p->chunked_handler.chunked_reader->read_avail();
   if (num_read < p->init_bytes_done) {
 p->init_bytes_done = num_read;

[trafficserver] branch master updated: Make all_headers test more resilient to timimng

2020-03-10 Thread shinrich 
This is an automated email from the ASF dual-hosted git repository.

shinrich pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git


The following commit(s) were added to refs/heads/master by this push:
 new 4f0a6be  Make all_headers test more resilient to timimng
4f0a6be is described below

commit 4f0a6be7fd38a09eb407fd872f6a492fbfcbef3f
Author: Susan Hinrichs 
AuthorDate: Mon Mar 9 15:20:18 2020 +

Make all_headers test more resilient to timimng
---
 tests/gold_tests/logging/all_headers.test.py  |  3 +--
 tests/gold_tests/logging/all_headers_sanitizer.py | 26 +--
 2 files changed, 21 insertions(+), 8 deletions(-)

diff --git a/tests/gold_tests/logging/all_headers.test.py 
b/tests/gold_tests/logging/all_headers.test.py
index 12b81f0..7e0f47e 100644
--- a/tests/gold_tests/logging/all_headers.test.py
+++ b/tests/gold_tests/logging/all_headers.test.py
@@ -96,8 +96,7 @@ tr.Processes.Default.ReturnCode = 0
 # Delay to allow TS to flush report to disk, then "sanitize" generated log.
 #
 tr = Test.AddTestRun()
-tr.DelayStart = 10
-tr.Processes.Default.Command = 'python3 {0} {4} < {2} | sh {1} > {3}'.format(
+tr.Processes.Default.Command = 'python3 {0} {2} {4} | sh {1} > {3}'.format(
 os.path.join(Test.TestDirectory, 'all_headers_sanitizer.py'),
 os.path.join(Test.TestDirectory, 'all_headers_sanitizer.sh'),
 os.path.join(ts.Variables.LOGDIR, 'test_all_headers.log'),
diff --git a/tests/gold_tests/logging/all_headers_sanitizer.py 
b/tests/gold_tests/logging/all_headers_sanitizer.py
index 2206b5e..64c22fd 100644
--- a/tests/gold_tests/logging/all_headers_sanitizer.py
+++ b/tests/gold_tests/logging/all_headers_sanitizer.py
@@ -19,6 +19,8 @@ Sanitize the ATS-generated custom log file from the 
all_headers test.
 
 import sys
 import re
+from os import path
+import time
 
 rexl = []
 rexl.append((re.compile(r"\{\{Date\}\:\{[^}]*\}\}"), "({__DATE__}}"))
@@ -29,15 +31,27 @@ rexl.append((re.compile(r"\{\{Server\}\:\{ECS [^}]*\}\}"), 
"({__ECS_SERVER__}}")
 rexl.append((re.compile(r"\{\{Via\}\:\{[^}]*\}\}"), "({__VIA__}}"))
 rexl.append((re.compile(r"\{\{Server\}\:\{ApacheTrafficServer/[0-9.]*\}\}"), 
"({__ATS2_SERVER__}}"))
 rexl.append((re.compile(r"\{\{Age\}\:\{[0-9]*\}\}"), "({__AGE__}}"))
-rexl.append((re.compile(r"\:" + sys.argv[1]), "__TS_PORT__")) # 1st and only 
argument is TS client port
+rexl.append((re.compile(r"\:" + sys.argv[2]), "__TS_PORT__")) # 1st and only 
argument is TS client port
 
 # Handle inconsistencies which I think are caused by different revisions of 
the standard Python http.server.HTTPServer class.
 
 rexl.append((re.compile(r'\{"359670651[^"]*"\}'), '{"{359670651__WEIRD__}"}'))
 rexl.append((re.compile(r'\{\{Accept-Ranges\}:\{bytes\}\}'), ''))
 
-for line in sys.stdin:
-for rex, subStr in rexl:
-line = rex.sub(subStr, line)
-
-print(line)
+# Loop until the file specified in argv[1] becomes availble
+filename = sys.argv[1]
+processed = False
+# Give up looking for file after 2 minutes
+limit_count = 0
+limit_max = 120
+while not processed and limit_count < limit_max:
+  limit_count += 1
+  if not path.exists(filename):
+time.sleep(1);
+  else:
+with open(filename, "r") as f:
+  processed = True
+  for line in f:
+for rex, subStr in rexl:
+  line = rex.sub(subStr, line)
+print(line)

[trafficserver] branch master updated (6b634b2 -> 11e2139)

2020-03-10 Thread shinrich 
This is an automated email from the ASF dual-hosted git repository.

shinrich pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git.


from 6b634b2  Add null check to fix error tunnel crash
 add 11e2139  remove pthread_cancel, use atomic flags to ensure cross 
thread safety, and some other cleanups

No new revisions were added by this update.

Summary of changes:
 .../experimental/ssl_session_reuse/src/Config.h|   1 -
 .../experimental/ssl_session_reuse/src/common.cc   |  12 ++
 .../experimental/ssl_session_reuse/src/common.h|   7 +-
 .../experimental/ssl_session_reuse/src/message.h   |   2 +-
 .../ssl_session_reuse/src/openssl_utils.cc |   2 +-
 .../experimental/ssl_session_reuse/src/publish.cc  | 116 -
 .../ssl_session_reuse/src/session_process.cc   |  16 +-
 .../ssl_session_reuse/src/ssl_key_utils.cc | 183 -
 plugins/experimental/ssl_session_reuse/src/stek.h  |   4 +-
 .../ssl_session_reuse/src/subscriber.cc| 106 ++--
 .../ssl_session_reuse/src/subscriber.h |   3 +-
 11 files changed, 262 insertions(+), 190 deletions(-)

[trafficserver] branch master updated: Add null check to fix error tunnel crash

2020-03-10 Thread shinrich 
This is an automated email from the ASF dual-hosted git repository.

shinrich pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git


The following commit(s) were added to refs/heads/master by this push:
 new 6b634b2  Add null check to fix error tunnel crash
6b634b2 is described below

commit 6b634b2c5af6467a9db0aa6f536b19c70a80ccae
Author: Susan Hinrichs 
AuthorDate: Mon Mar 9 13:52:03 2020 +

Add null check to fix error tunnel crash
---
 proxy/http/HttpTunnel.cc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proxy/http/HttpTunnel.cc b/proxy/http/HttpTunnel.cc
index 56c1a38..4e69d2a 100644
--- a/proxy/http/HttpTunnel.cc
+++ b/proxy/http/HttpTunnel.cc
@@ -1473,7 +1473,7 @@ HttpTunnel::finish_all_internal(HttpTunnelProducer *p, 
bool chain)
 
   if (action == TCA_PASSTHRU_CHUNKED_CONTENT) {
 // if the only chunked data was in the initial read, make sure we don't 
consume too much
-if (p->bytes_read == 0) {
+if (p->bytes_read == 0 && p->buffer_start != nullptr) {
   int num_read = p->buffer_start->read_avail() - 
p->chunked_handler.chunked_reader->read_avail();
   if (num_read < p->init_bytes_done) {
 p->init_bytes_done = num_read;