[trafficserver] branch 9.0.x updated: Updated CHANGELOG
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git The following commit(s) were added to refs/heads/9.0.x by this push: new f038554 Updated CHANGELOG f038554 is described below commit f03855457516bfeb9a9bfcd103c4bae66b2e1a14 Author: Leif Hedstrom AuthorDate: Tue Dec 10 16:57:43 2019 -0700 Updated CHANGELOG --- CHANGELOG-9.0.0 | 13 + 1 file changed, 13 insertions(+) diff --git a/CHANGELOG-9.0.0 b/CHANGELOG-9.0.0 index bcfeeb0..d21ef6f 100644 --- a/CHANGELOG-9.0.0 +++ b/CHANGELOG-9.0.0 @@ -629,6 +629,7 @@ Changes with Apache Traffic Server 9.0.0 #5683 - Cleanup example directory #5686 - Reverse internal order of HPACK Dynamic Table Entries #5690 - Fix race condition in test + #5695 - Add MIMEHdr Garbage Collection to HPACK Dynamic Table #5697 - Coalesce empty objects in HdrHeap #5698 - Remove unused LibBulkIO #5699 - Remove unused header file ink_inout.h @@ -757,6 +758,7 @@ Changes with Apache Traffic Server 9.0.0 #5980 - Fix use-after-free problem related to logging headers #5985 - Add filename to error message in header_rewrite #5986 - Remove hard-coded ssl_multicert.config log reference + #5987 - Removing traffic_cop reference in admin guide introduction #5992 - Cache SSL EC explicitly #6000 - Add QUIC draft-23 support #6003 - Cleanup: do not re-define _proxy_ssn variable in local @@ -782,9 +784,11 @@ Changes with Apache Traffic Server 9.0.0 #6094 - Issue #4294: Handle return value of SSL_write() properly #6095 - For remap_stats, removes configure time dependency on search.h #6096 - Fixing log cleanup candidate selection and adding a test for it. + #6098 - Replace container of HPACK dynamic table from std::vector to std::deque #6099 - doc + unittest TSRemap(Init|NewInstance) failures #6106 - Removes proxy.config.cache.storage_filename #6116 - Updated to clang-format v9.0.0 + #6118 - Avoid IOBufferReader::read_avail() call from MIOBuffer::high_water() #6120 - Fix null pointer dereference reported by clang-analyzer #6125 - Add Metrics to track SSL Handshake attempts #6126 - Fix the thread ready check @@ -805,11 +809,13 @@ Changes with Apache Traffic Server 9.0.0 #6160 - Fixes misc. spelling and whitespace #6163 - Update yaml-cpp to 0.6.3 #6167 - Fixed gcc7 issue with yaml-cpp 0.6.3 + #6168 - Add cache_range_requests autest, add to x-parentselectiion-key to xdebug plugin. #6169 - Fixes cppcheck issues for cookie_remap plugin #6170 - Add test to catch regressions in sni and override configs #6173 - Remove HttpTransact::build_upgrade_response #6174 - Remove useless UDP function #6175 - Removes the ssn_close hook, it can never work + #6176 - Fixing rolled log cleanup parsing for .log. files. #6180 - RBTree - fix potential nullptr dereference #6181 - remap_stats: Fix BufferWriter usage error. #6182 - tslua: Exposes set/get version for server request objects @@ -819,5 +825,12 @@ Changes with Apache Traffic Server 9.0.0 #6189 - Fixed issue with macOS Catalina and pcre 8.43 enabling pcre-jit #6190 - Fix ssl_session_reuse to compile on macOS and FreeBSD #6191 - Limit this check to Linux, where ldd always works + #6198 - Move logging before session could be freed + #6201 - ssl_session_reuse optimization to check if diags is on + #6204 - Lua plugin fix: Account for null in output from TSUrlPercentDecode. + #6207 - Add autopep8 & pyflakes in dev-packages for AuTest + #6209 - Add 100-continue expectation support on H2 connection #6212 - Cleanup trailing whitespaces, and YAML formatting + #6218 - Fix stringstream crash during shutdown #6224 - Fixed build issue with clang5 and Extendible + #6231 - Adjust the refcounts to avoid Mutex leak
[trafficserver] 08/08: Replace container of HPACK dynamic table from std::vector to std::deque
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit 2d1eb8bfa57788bb8a8e5682c2c1400690ca8da5 Author: Masaori Koshiba AuthorDate: Wed Oct 30 11:20:54 2019 +0900 Replace container of HPACK dynamic table from std::vector to std::deque (cherry picked from commit 323d5b59fb84c2bb73c2679bd8924b530e26ce3a) --- proxy/http2/HPACK.cc | 20 proxy/http2/HPACK.h | 4 ++-- 2 files changed, 10 insertions(+), 14 deletions(-) diff --git a/proxy/http2/HPACK.cc b/proxy/http2/HPACK.cc index b13e7c6..4dcf729 100644 --- a/proxy/http2/HPACK.cc +++ b/proxy/http2/HPACK.cc @@ -354,7 +354,7 @@ HpackDynamicTable::~HpackDynamicTable() const MIMEField * HpackDynamicTable::get_header_field(uint32_t index) const { - return this->_headers.at(this->_headers.size() - index - 1); + return this->_headers.at(index); } void @@ -380,8 +380,7 @@ HpackDynamicTable::add_header_field(const MIMEField *field) MIMEField *new_field = this->_mhdr->field_create(name, name_len); new_field->value_set(this->_mhdr->m_heap, this->_mhdr->m_mime, value, value_len); this->_mhdr->field_attach(new_field); -// XXX Because entire Vec instance is copied, Its too expensive! -this->_headers.push_back(new_field); +this->_headers.push_front(new_field); } } @@ -414,7 +413,7 @@ HpackDynamicTable::update_maximum_size(uint32_t new_size) uint32_t HpackDynamicTable::length() const { - return _headers.size(); + return this->_headers.size(); } bool @@ -425,23 +424,20 @@ HpackDynamicTable::_evict_overflowed_entries() return true; } - size_t count = 0; - for (auto : this->_headers) { + for (auto h = this->_headers.rbegin(); h != this->_headers.rend(); ++h) { int name_len, value_len; -h->name_get(_len); -h->value_get(_len); +(*h)->name_get(_len); +(*h)->value_get(_len); this->_current_size -= ADDITIONAL_OCTETS + name_len + value_len; -this->_mhdr->field_delete(h, false); -++count; +this->_mhdr->field_delete(*h, false); +this->_headers.pop_back(); if (this->_current_size <= this->_maximum_size) { break; } } - this->_headers.erase(this->_headers.begin(), this->_headers.begin() + count); - if (this->_headers.size() == 0) { return false; } diff --git a/proxy/http2/HPACK.h b/proxy/http2/HPACK.h index 580792e..dc0b517 100644 --- a/proxy/http2/HPACK.h +++ b/proxy/http2/HPACK.h @@ -28,7 +28,7 @@ #include "HTTP.h" #include "../hdrs/XPACK.h" -#include +#include // It means that any header field can be compressed/decompressed by ATS const static int HPACK_ERROR_COMPRESSION_ERROR = -1; @@ -136,7 +136,7 @@ private: MIMEHdr *_mhdr = nullptr; MIMEHdr *_mhdr_old = nullptr; - std::vector _headers; + std::deque _headers; }; // [RFC 7541] 2.3. Indexing Table
[trafficserver] 05/08: ssl_session_reuse optimization to check if diags is on
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit a269ec302e2d28c08497e2bb93a3e52fcf26580a Author: Bryan Call AuthorDate: Tue Nov 19 10:40:53 2019 +0800 ssl_session_reuse optimization to check if diags is on (cherry picked from commit 3a4fe4e02e61058002de34f70155a70e9927fae6) --- .../experimental/ssl_session_reuse/src/publish.cc | 57 ++ 1 file changed, 37 insertions(+), 20 deletions(-) diff --git a/plugins/experimental/ssl_session_reuse/src/publish.cc b/plugins/experimental/ssl_session_reuse/src/publish.cc index 98bab37..9e2eb7e 100644 --- a/plugins/experimental/ssl_session_reuse/src/publish.cc +++ b/plugins/experimental/ssl_session_reuse/src/publish.cc @@ -35,11 +35,12 @@ #include "Config.h" #include "redis_auth.h" #include "ssl_utils.h" +#include void * RedisPublisher::start_worker_thread(void *arg) { - plugin_threads.store(::pthread_self()); + plugin_threads.store(pthread_self()); ::pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, nullptr); ::pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, nullptr); @@ -117,8 +118,11 @@ RedisPublisher::is_good() ::redisContext * RedisPublisher::setup_connection(const RedisEndpoint ) { - auto my_id = (unsigned long long)::pthread_self(); - TSDebug(PLUGIN, "RedisPublisher::setup_connection: Called by threadId: %llx", my_id); + uint64_t my_id = 0; + if (TSIsDebugTagSet(PLUGIN)) { +my_id = (uint64_t)pthread_self(); +TSDebug(PLUGIN, "RedisPublisher::setup_connection: Called by threadId: %" PRIx64, my_id); + } RedisContextPtr my_context; struct ::timeval timeout; @@ -128,14 +132,15 @@ RedisPublisher::setup_connection(const RedisEndpoint ) for (int i = 0; i < static_cast(m_redisConnectTries); ++i) { my_context.reset(::redisConnectWithTimeout(re.m_hostname.c_str(), re.m_port, timeout)); if (!my_context) { - TSError("RedisPublisher::setup_connection: Connect to host: %s port: %d fail count: %d threadId: %llx", re.m_hostname.c_str(), - re.m_port, i + 1, my_id); + TSError("RedisPublisher::setup_connection: Connect to host: %s port: %d fail count: %d threadId: %" PRIx64, + re.m_hostname.c_str(), re.m_port, i + 1, my_id); } else if (my_context->err) { - TSError("RedisPublisher::setup_connection: Connect to host: %s port: %d fail count: %d threadId: %llx", re.m_hostname.c_str(), - re.m_port, i + 1, my_id); + TSError("RedisPublisher::setup_connection: Connect to host: %s port: %d fail count: %d threadId: %" PRIx64, + re.m_hostname.c_str(), re.m_port, i + 1, my_id); my_context.reset(nullptr); } else { - TSDebug(PLUGIN, "RedisPublisher::setup_connection: threadId: %llx Successfully connected to the redis instance.", my_id); + TSDebug(PLUGIN, "RedisPublisher::setup_connection: threadId: %" PRIx64 " Successfully connected to the redis instance.", + my_id); redisReply *reply = static_cast(redisCommand(my_context.get(), "AUTH %s", redis_passwd.c_str())); @@ -163,8 +168,11 @@ RedisPublisher::setup_connection(const RedisEndpoint ) ::redisReply * RedisPublisher::send_publish(RedisContextPtr , const RedisEndpoint , const Message ) { - auto my_id = (unsigned long long)::pthread_self(); - TSDebug(PLUGIN, "RedisPublisher::send_publish: Called by threadId: %llx", my_id); + uint64_t my_id = 0; + if (TSIsDebugTagSet(PLUGIN)) { +my_id = (uint64_t)pthread_self(); +TSDebug(PLUGIN, "RedisPublisher::send_publish: Called by threadId: %" PRIx64, my_id); + } ::redisReply *current_reply(nullptr); @@ -173,7 +181,8 @@ RedisPublisher::send_publish(RedisContextPtr , const RedisEndpoint , cons ctx.reset(setup_connection(re)); if (!ctx) { -TSError("RedisPublisher::send_publish: Unable to setup a connection to the redis server: %s:%d threadId: %llx try: %d", +TSError("RedisPublisher::send_publish: Unable to setup a connection to the redis server: %s:%d threadId: %" PRIx64 +" try: %d", re.m_hostname.c_str(), re.m_port, my_id, (i + 1)); continue; } @@ -181,12 +190,12 @@ RedisPublisher::send_publish(RedisContextPtr , const RedisEndpoint , cons current_reply = static_cast(::redisCommand(ctx.get(), "PUBLISH %s %s", msg.channel.c_str(), msg.data.c_str())); if (!current_reply) { - TSError("RedisPublisher::send_publish: Unable to get a reply from the server for publish. threadId: %llx try: %d", my_id, - (i + 1)); + TSError("RedisPublisher::send_publish: Unable to get a reply from the server for publish. threadId: %" PRIx64 " try: %d", + my_id, (i + 1)); ctx.reset(nullptr); // Clean up previous attempt } else if (REDIS_REPLY_ERROR == current_reply->type) { -
[trafficserver] 04/08: Lua plugin fix: Account for null in output from TSUrlPercentDecode.
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit 92caab23add95cfe430aa0a9847e7a59674efb31 Author: Alan M. Carroll AuthorDate: Tue Nov 19 13:16:13 2019 -0600 Lua plugin fix: Account for null in output from TSUrlPercentDecode. (cherry picked from commit cd2afa616239896c53cdd61c0ba94bfc1edf5d29) --- doc/admin-guide/plugins/lua.en.rst | 4 ++-- .../api/functions/TSUrlPercentEncode.en.rst| 24 -- plugins/lua/ts_lua_crypto.c| 4 ++-- 3 files changed, 22 insertions(+), 10 deletions(-) diff --git a/doc/admin-guide/plugins/lua.en.rst b/doc/admin-guide/plugins/lua.en.rst index 9f52692..3200f0a 100644 --- a/doc/admin-guide/plugins/lua.en.rst +++ b/doc/admin-guide/plugins/lua.en.rst @@ -1952,8 +1952,8 @@ ts.server_request_get_version **description:** Return the http version string of the server request. -Current possible values are 1.0, 1.1, and 0.9. -:: +Current possible values are 1.0, 1.1, and 0.9. :: + function send_request() local version = ts.server_request.get_version() ts.debug(version) diff --git a/doc/developer-guide/api/functions/TSUrlPercentEncode.en.rst b/doc/developer-guide/api/functions/TSUrlPercentEncode.en.rst index 4e79ad4..d342368 100644 --- a/doc/developer-guide/api/functions/TSUrlPercentEncode.en.rst +++ b/doc/developer-guide/api/functions/TSUrlPercentEncode.en.rst @@ -51,16 +51,28 @@ failed. :func:`TSStringPercentEncode` is similar but operates on a string. If the optional :arg:`map` parameter is provided (not :literal:`NULL`) , it should be a map of characters to encode. -:func:`TSStringPercentDecode` perform percent-decoding of the string in the -:arg:`str` buffer, writing to the :arg:`dst` buffer. The source and -destination can be the same, in which case they overwrite. The decoded string -is always guaranteed to be no longer than the source string. +:func:`TSStringPercentDecode` perform percent-decoding of the string in the :arg:`str` buffer, +writing to the :arg:`dst` buffer. The source and destination can be the same, in which case the +decoded string is written on top of the source string. The decoded string is guaranteed to be +no longer than the source string, but will include a terminating null which, if there are no +escapes, makes the destination one longer than the source. In practice this means the destination +length needs to be bumped up by one to account for the null, and a string can't be decoded in place +if it's not already null terminated with the length of the destination including the null, but the +length of the source *not* including the null. E.g. :: + + static char const ORIGINAL[] = "A string without escapes, but null terminated"; + char * source = TSstrdup(ORIGINAL); // make it writeable. + size_t length; // return value. + // sizeof(ORIGINAL) includes the null, so don't include that in the input. + static const size_t N_CHARS = sizeof(ORIGINAL) - 1; + TSReturnCode result = TSUrlPercentDecode(source, N_CHARS, source, N_CHARS + 1, ); + ink_assert(length == N_CHARS); Return Values = -All these APIs returns a :type:`TSReturnCode`, indicating success -(:data:`TS_SUCCESS`) or failure (:data:`TS_ERROR`) of the operation. +All these APIs returns a :type:`TSReturnCode`, indicating success (:data:`TS_SUCCESS`) or failure +(:data:`TS_ERROR`) of the operation. See Also diff --git a/plugins/lua/ts_lua_crypto.c b/plugins/lua/ts_lua_crypto.c index 73a79a9..7637183 100644 --- a/plugins/lua/ts_lua_crypto.c +++ b/plugins/lua/ts_lua_crypto.c @@ -316,8 +316,8 @@ ts_lua_unescape_uri(lua_State *L) return 1; } - /* the unescaped string can only be smaller */ - dlen = len; + /* the unescaped string can not be larger, but need to account for terminating null. */ + dlen = len + 1; dst = lua_newuserdata(L, dlen); if (TS_SUCCESS == TSStringPercentDecode((const char *)src, len, (char *)dst, dlen, )) {
[trafficserver] 02/08: Add autopep8 & pyflakes in dev-packages for AuTest
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit f9f486ead3d71212f8ac4e9b784c4e957f68aab9 Author: Masaori Koshiba AuthorDate: Wed Nov 20 13:43:39 2019 +0900 Add autopep8 & pyflakes in dev-packages for AuTest (cherry picked from commit 66bf0c41b90eb5e9c46a916d231e69c92f2895f7) --- tests/Pipfile | 2 ++ tests/README.md | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/Pipfile b/tests/Pipfile index aa4a6e8..f98af7f 100644 --- a/tests/Pipfile +++ b/tests/Pipfile @@ -20,6 +20,8 @@ url = "https://pypi.org/simple; verify_ssl = true [dev-packages] +autopep8 = "*" +pyflakes = "*" [packages] autest = "==1.7.3" diff --git a/tests/README.md b/tests/README.md index 32f2a5d..3024d94 100644 --- a/tests/README.md +++ b/tests/README.md @@ -25,7 +25,7 @@ This script will check for the necessary packages needed to create a pipenv that ### Pipfile This file is used to setup a virtual environment using pipenv. It contains information including the packages needed for Autest. A set of commands for pipenv: - * **pipenv install**: create virtual environment from the Pipfile. + * **pipenv install**: create virtual environment from the Pipfile. ( If you're going to add tests, add `-d` option to install dev packages ) * **pipenv shell**: launch a shell with the environment running(type "exit" to leave the shell). * **pipenv run cmd**: run command in the virtual environment without entering a shell, where cmd is the shell command to run. * **pipenv --rm**: remove the environment.
[trafficserver] 03/08: Add 100-continue expectation support on H2 connection
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit 14ebf508edd69ef6c25b81041c3b0ab0a5a16406 Author: Masaori Koshiba AuthorDate: Wed Nov 20 14:05:54 2019 +0900 Add 100-continue expectation support on H2 connection (cherry picked from commit c0fe598af5ad41a73c44d8f10fc5bae6baa555f7) --- proxy/hdrs/HTTP.h | 23 +-- proxy/http2/Http2Stream.cc | 11 +++ tests/gold_tests/h2/gold/httpbin_0_stdout.gold | 9 ++--- tests/gold_tests/h2/gold/httpbin_3_stderr.gold | 9 + tests/gold_tests/h2/gold/httpbin_3_stdout.gold | 7 +++ tests/gold_tests/h2/gold/httpbin_access.gold | 1 + tests/gold_tests/h2/httpbin.test.py| 19 +-- 7 files changed, 68 insertions(+), 11 deletions(-) diff --git a/proxy/hdrs/HTTP.h b/proxy/hdrs/HTTP.h index 30fcf6c..c7c43e1 100644 --- a/proxy/hdrs/HTTP.h +++ b/proxy/hdrs/HTTP.h @@ -623,7 +623,7 @@ public: /// header internals, they must be able to do this. void mark_target_dirty() const; - HTTPStatus status_get(); + HTTPStatus status_get() const; void status_set(HTTPStatus status); const char *reason_get(int *length); @@ -642,6 +642,7 @@ public: bool is_cache_control_set(const char *cc_directive_wks); bool is_pragma_no_cache_set(); bool is_keep_alive_set() const; + bool expect_final_response() const; HTTPKeepAlive keep_alive_get() const; protected: @@ -1001,6 +1002,24 @@ HTTPHdr::is_keep_alive_set() const return this->keep_alive_get() == HTTP_KEEPALIVE; } +/** + Check the status code is informational and expecting final response + - e.g. "100 Continue", "103 Early Hints" + + Please note that "101 Switching Protocol" is not included. + */ +inline bool +HTTPHdr::expect_final_response() const +{ + switch (this->status_get()) { + case HTTP_STATUS_CONTINUE: + case HTTP_STATUS_EARLY_HINTS: +return true; + default: +return false; + } +} + /*- -*/ @@ -1151,7 +1170,7 @@ http_hdr_status_get(HTTPHdrImpl *hh) -*/ inline HTTPStatus -HTTPHdr::status_get() +HTTPHdr::status_get() const { ink_assert(valid()); diff --git a/proxy/http2/Http2Stream.cc b/proxy/http2/Http2Stream.cc index eb98c4c..8da3d86 100644 --- a/proxy/http2/Http2Stream.cc +++ b/proxy/http2/Http2Stream.cc @@ -639,6 +639,17 @@ Http2Stream::update_write_request(IOBufferReader *buf_reader, int64_t write_len, h2_proxy_ssn->connection_state.send_headers_frame(this); } + // Roll back states of response header to read final response + if (this->response_header.expect_final_response()) { +this->response_header_done = false; +response_header.destroy(); +response_header.create(HTTP_TYPE_RESPONSE); +http_parser_clear(_parser); +http_parser_init(_parser); + } + + this->signal_write_event(call_update); + if (this->response_reader->is_read_avail_more_than(0)) { this->_milestones.mark(Http2StreamMilestone::START_TX_DATA_FRAMES); this->send_response_body(call_update); diff --git a/tests/gold_tests/h2/gold/httpbin_0_stdout.gold b/tests/gold_tests/h2/gold/httpbin_0_stdout.gold index c46139d..d8e6107 100644 --- a/tests/gold_tests/h2/gold/httpbin_0_stdout.gold +++ b/tests/gold_tests/h2/gold/httpbin_0_stdout.gold @@ -1,10 +1,5 @@ { - "args": {}, `` - "headers": { -"Accept": "*/*", `` -"Host": `` -"User-Agent": "curl/`` - }, `` - "origin": `` + `` "url": "http://``/get; + `` } diff --git a/tests/gold_tests/h2/gold/httpbin_3_stderr.gold b/tests/gold_tests/h2/gold/httpbin_3_stderr.gold new file mode 100644 index 000..8109f9d --- /dev/null +++ b/tests/gold_tests/h2/gold/httpbin_3_stderr.gold @@ -0,0 +1,9 @@ +`` +> POST /post HTTP/2 +`` +> Expect: 100-continue +`` +< HTTP/2 100`` +`` +< HTTP/2 200`` +`` diff --git a/tests/gold_tests/h2/gold/httpbin_3_stdout.gold b/tests/gold_tests/h2/gold/httpbin_3_stdout.gold new file mode 100644 index 000..6105808 --- /dev/null +++ b/tests/gold_tests/h2/gold/httpbin_3_stdout.gold @@ -0,0 +1,7 @@ +{ + `` + "form": { +"key": "value" + }, + `` +} diff --git a/tests/gold_tests/h2/gold/httpbin_access.gold b/tests/gold_tests/h2/gold/httpbin_access.gold index 3f77947..d409c47 100644 --- a/tests/gold_tests/h2/gold/httpbin_access.gold +++ b/tests/gold_tests/h2/gold/httpbin_access.gold @@ -1,3 +1,4 @@ [``] GET http://127.0.0.1:``/get HTTP/1.1 http/2 `` `` TCP_MISS 200 `` [``] GET http://127.0.0.1:``/bytes/0 HTTP/1.1 http/2 `` `` TCP_MISS 200 0 [``] GET http://127.0.0.1:``/stream-bytes/102400?seed=0 HTTP/1.1 http/2 `` `` TCP_MISS 200 102400 +``
[trafficserver] 06/08: Avoid IOBufferReader::read_avail() call from MIOBuffer::high_water()
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit 1861f369c6cdadc5a7afbce4671fb06bcbdfbc76 Author: Masaori Koshiba AuthorDate: Fri Nov 1 11:31:01 2019 +0900 Avoid IOBufferReader::read_avail() call from MIOBuffer::high_water() (cherry picked from commit 5e639715a52c6e0c09bdf8842989fbd83850c81a) --- iocore/eventsystem/IOBuffer.cc | 20 iocore/eventsystem/I_IOBuffer.h | 19 ++- 2 files changed, 38 insertions(+), 1 deletion(-) diff --git a/iocore/eventsystem/IOBuffer.cc b/iocore/eventsystem/IOBuffer.cc index d2c130b..efa68e3 100644 --- a/iocore/eventsystem/IOBuffer.cc +++ b/iocore/eventsystem/IOBuffer.cc @@ -176,6 +176,26 @@ MIOBuffer::puts(char *s, int64_t len) return 0; } +bool +MIOBuffer::is_max_read_avail_more_than(int64_t size) +{ + bool no_reader = true; + for (auto : this->readers) { +if (reader.allocated()) { + if (reader.is_read_avail_more_than(size)) { +return true; + } + no_reader = false; +} + } + + if (no_reader && this->_writer) { +return (this->_writer->read_avail() > size); + } + + return false; +} + // // IOBufferReader // diff --git a/iocore/eventsystem/I_IOBuffer.h b/iocore/eventsystem/I_IOBuffer.h index f83bc32..f8d317b 100644 --- a/iocore/eventsystem/I_IOBuffer.h +++ b/iocore/eventsystem/I_IOBuffer.h @@ -1094,7 +1094,7 @@ public: bool high_water() { -return max_read_avail() > water_mark; +return is_max_read_avail_more_than(this->water_mark); } /** @@ -1176,8 +1176,25 @@ public: { return !_writer; } + + /** +Get the maximum amount of available data across all of the readers. +If there're no allocated reader, return available data size of current writer. + +This calls IOBufferReader::read_avail() and it could be expensive when it has a ton of IOBufferBlock. +The `is_max_read_avail(int64_t size)` is preferred if possible. + +@return maximum amount of available data + */ int64_t max_read_avail(); + /** +Check if there is more than @a size bytes available to read. + +@return @c true if more than @a size byte are available. + */ + bool is_max_read_avail_more_than(int64_t size); + int max_block_count(); void check_add_block();
[trafficserver] 07/08: Add MIMEHdr Garbage Collection to HPACK Dynamic Table
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit 6a4edd300732a79161a914bf4a7216e9b22ac2b4 Author: Masaori Koshiba AuthorDate: Thu Jul 4 12:03:53 2019 +0900 Add MIMEHdr Garbage Collection to HPACK Dynamic Table Prior this change, the size of HdrHeap which is owned by MIMEHdr of HpackDynamicTable had no limit. Because when MIMEFiled is deleted the allocated memory of the HdrHeap was not freed. To mitigate this issue, when HdrHeap size exceeds the threshold, HpackDynamicTable start using new MIMEHdr and HdrHeap. The old MIMEHdr and HdrHeap will be freed, when all MIMEFiled is deleted by HPACK Dynamic Table Entry Eviction. (cherry picked from commit 2bbcc48195e98b65b681a7e28c9ae1c2ab5d6994) --- proxy/hdrs/HdrHeap.cc | 18 + proxy/hdrs/HdrHeap.h | 2 ++ proxy/http2/HPACK.cc | 56 +++ proxy/http2/HPACK.h | 16 ++- 4 files changed, 82 insertions(+), 10 deletions(-) diff --git a/proxy/hdrs/HdrHeap.cc b/proxy/hdrs/HdrHeap.cc index c10d6e5..1452a8e 100644 --- a/proxy/hdrs/HdrHeap.cc +++ b/proxy/hdrs/HdrHeap.cc @@ -1124,6 +1124,24 @@ HdrHeap::dump_heap(int len) fprintf(stderr, "\n-- End header heap dump ---\n"); } +uint64_t +HdrHeap::total_used_size() const +{ + uint64_t size= 0; + const HdrHeap *h = this; + + while (h) { +size += (h->m_free_start - h->m_data_start); +h = h->m_next; + } + + return size; +} + +// +// HdrStrHeap +// + void HdrStrHeap::free() { diff --git a/proxy/hdrs/HdrHeap.h b/proxy/hdrs/HdrHeap.h index e7c075c..1d83c85 100644 --- a/proxy/hdrs/HdrHeap.h +++ b/proxy/hdrs/HdrHeap.h @@ -287,6 +287,8 @@ public: size_t required_space_for_evacuation(); bool attach_str_heap(char const *h_start, int h_len, RefCountObj *h_ref_obj, int *index); + uint64_t total_used_size() const; + /** Struct to prevent garbage collection on heaps. This bumps the reference count to the heap containing the pointer while the instance of this class exists. When it goes out of scope diff --git a/proxy/http2/HPACK.cc b/proxy/http2/HPACK.cc index 7fa62cc..b13e7c6 100644 --- a/proxy/http2/HPACK.cc +++ b/proxy/http2/HPACK.cc @@ -166,6 +166,21 @@ static const StaticTable STATIC_TABLE[] = {{"", ""}, {"via", ""}, {"www-authenticate", ""}}; +/** + Threshold for total HdrHeap size which used by HPAK Dynamic Table. + The HdrHeap is filled by MIMEHdrImpl and MIMEFieldBlockImpl like below. + This threshold allow to allocate 3 HdrHeap at maximum. + + +--+-+ + HdrHeap 1 (2048): | MIMEHdrImpl(592) | MIMEFieldBlockImpl(528) x 2 | + +--+-+--...--+ + HdrHeap 2 (4096): | MIMEFieldBlockImpl(528) x 7| + ++--...--+--...--+ + HdrHeap 3 (8192): | MIMEFieldBlockImpl(528) x 15 | + ++--...--+--...--+ +*/ +static constexpr uint32_t HPACK_HDR_HEAP_THRESHOLD = sizeof(MIMEHdrImpl) + sizeof(MIMEFieldBlockImpl) * (2 + 7 + 15); + /** * Local functions **/ @@ -318,6 +333,24 @@ HpackIndexingTable::update_maximum_size(uint32_t new_size) return _dynamic_table->update_maximum_size(new_size); } +// +// HpackDynamicTable +// +HpackDynamicTable::~HpackDynamicTable() +{ + this->_headers.clear(); + + this->_mhdr->fields_clear(); + this->_mhdr->destroy(); + delete this->_mhdr; + + if (this->_mhdr_old != nullptr) { +this->_mhdr_old->fields_clear(); +this->_mhdr_old->destroy(); +delete this->_mhdr_old; + } +} + const MIMEField * HpackDynamicTable::get_header_field(uint32_t index) const { @@ -413,9 +446,32 @@ HpackDynamicTable::_evict_overflowed_entries() return false; } + this->_mime_hdr_gc(); + return true; } +/** + When HdrHeap size of current MIMEHdr exceeds the threshold, allocate new MIMEHdr and HdrHeap. + The old MIMEHdr and HdrHeap will be freed, when all MIMEFiled are deleted by HPACK Entry Eviction. + */ +void +HpackDynamicTable::_mime_hdr_gc() +{ + if (this->_mhdr_old == nullptr) { +if (this->_mhdr->m_heap->total_used_size() >= HPACK_HDR_HEAP_THRESHOLD) { + this->_mhdr_old = this->_mhdr; + this->_mhdr = new MIMEHdr(); + this->_mhdr->create(); +} + } else { +if (this->_mhdr_old->fields_count() == 0) { + this->_mhdr_old->destroy(); + this->_mhdr_old = nullptr; +} + } +} + int64_t encode_indexed_header_field(uint8_t *buf_start, const uint8_t *buf_end, uint32_t index) { diff --git
[trafficserver] 01/08: Move logging before session could be freed
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit d1ce03059c6e86c3b52a7afe3ee33fff1f9ac91e Author: Susan Hinrichs AuthorDate: Mon Nov 18 21:59:35 2019 + Move logging before session could be freed (cherry picked from commit c44e0dc7da64186b6954bb08dc793eb00248c1ca) --- proxy/http/HttpSM.cc | 34 +- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/proxy/http/HttpSM.cc b/proxy/http/HttpSM.cc index 2450d2b..02145b7 100644 --- a/proxy/http/HttpSM.cc +++ b/proxy/http/HttpSM.cc @@ -6833,6 +6833,23 @@ HttpSM::kill_this() update_stats(); } +// +// Log Data // +// +SMDebug("http_seq", "[HttpSM::update_stats] Logging transaction"); +if (Log::transaction_logging_enabled() && t_state.api_info.logging_enabled) { + LogAccess accessor(this); + + int ret = Log::access(); + + if (ret & Log::FULL) { +SMDebug("http", "[update_stats] Logging system indicates FULL."); + } + if (ret & Log::FAIL) { +Log::error("failed to log transaction for at least one log object"); + } +} + if (ua_txn) { ua_txn->transaction_done(); } @@ -6851,23 +6868,6 @@ HttpSM::kill_this() HTTP_SM_SET_DEFAULT_HANDLER(nullptr); -// -// Log Data // -// -SMDebug("http_seq", "[HttpSM::update_stats] Logging transaction"); -if (Log::transaction_logging_enabled() && t_state.api_info.logging_enabled) { - LogAccess accessor(this); - - int ret = Log::access(); - - if (ret & Log::FULL) { -SMDebug("http", "[update_stats] Logging system indicates FULL."); - } - if (ret & Log::FAIL) { -Log::error("failed to log transaction for at least one log object"); - } -} - if (redirect_url != nullptr) { ats_free((void *)redirect_url); redirect_url = nullptr;
[trafficserver] branch 9.0.x updated (f8f3f20 -> 2d1eb8b)
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a change to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git. from f8f3f20 Add autest for cache_range_request, add xdebug x-parentselection-key new d1ce030 Move logging before session could be freed new f9f486e Add autopep8 & pyflakes in dev-packages for AuTest new 14ebf50 Add 100-continue expectation support on H2 connection new 92caab2 Lua plugin fix: Account for null in output from TSUrlPercentDecode. new a269ec3 ssl_session_reuse optimization to check if diags is on new 1861f36 Avoid IOBufferReader::read_avail() call from MIOBuffer::high_water() new 6a4edd3 Add MIMEHdr Garbage Collection to HPACK Dynamic Table new 2d1eb8b Replace container of HPACK dynamic table from std::vector to std::deque The 8 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: doc/admin-guide/plugins/lua.en.rst | 4 +- .../api/functions/TSUrlPercentEncode.en.rst| 24 +-- iocore/eventsystem/IOBuffer.cc | 20 ++ iocore/eventsystem/I_IOBuffer.h| 19 +- .../experimental/ssl_session_reuse/src/publish.cc | 57 ++-- plugins/lua/ts_lua_crypto.c| 4 +- proxy/hdrs/HTTP.h | 23 ++- proxy/hdrs/HdrHeap.cc | 18 + proxy/hdrs/HdrHeap.h | 2 + proxy/http/HttpSM.cc | 34 +- proxy/http2/HPACK.cc | 76 ++ proxy/http2/HPACK.h| 20 +++--- proxy/http2/Http2Stream.cc | 11 tests/Pipfile | 2 + tests/README.md| 2 +- tests/gold_tests/h2/gold/httpbin_0_stdout.gold | 9 +-- tests/gold_tests/h2/gold/httpbin_3_stderr.gold | 9 +++ tests/gold_tests/h2/gold/httpbin_3_stdout.gold | 7 ++ tests/gold_tests/h2/gold/httpbin_access.gold | 1 + tests/gold_tests/h2/httpbin.test.py| 19 +- 20 files changed, 277 insertions(+), 84 deletions(-) create mode 100644 tests/gold_tests/h2/gold/httpbin_3_stderr.gold create mode 100644 tests/gold_tests/h2/gold/httpbin_3_stdout.gold
[trafficserver] 02/07: Adjust the refcounts to avoid Mutex leak
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit 0f70c8f37fae72287dadd30bba615d78a71d2d69 Author: Susan Hinrichs AuthorDate: Tue Nov 26 17:06:30 2019 + Adjust the refcounts to avoid Mutex leak (cherry picked from commit 4a503c817766fcad3b9eab3ff3a8e9e73d483ae7) --- iocore/eventsystem/I_Lock.h| 40 +++--- src/traffic_server/InkIOCoreAPI.cc | 19 ++ 2 files changed, 48 insertions(+), 11 deletions(-) diff --git a/iocore/eventsystem/I_Lock.h b/iocore/eventsystem/I_Lock.h index 4850199..c5b8373 100644 --- a/iocore/eventsystem/I_Lock.h +++ b/iocore/eventsystem/I_Lock.h @@ -256,7 +256,7 @@ Mutex_trylock( #ifdef DEBUG const SourceLocation , const char *ahandler, #endif - Ptr , EThread *t) + ProxyMutex *m, EThread *t) { ink_assert(t != nullptr); ink_assert(t == reinterpret_cast(this_thread())); @@ -295,12 +295,26 @@ Mutex_trylock( return true; } +inline bool +Mutex_trylock( +#ifdef DEBUG + const SourceLocation , const char *ahandler, +#endif + Ptr , EThread *t) +{ + return Mutex_trylock( +#ifdef DEBUG +location, ahandler, +#endif +m.get(), t); +} + inline int Mutex_lock( #ifdef DEBUG const SourceLocation , const char *ahandler, #endif - Ptr , EThread *t) + ProxyMutex *m, EThread *t) { ink_assert(t != nullptr); if (m->thread_holding != t) { @@ -327,8 +341,22 @@ Mutex_lock( return true; } +inline int +Mutex_lock( +#ifdef DEBUG + const SourceLocation , const char *ahandler, +#endif + Ptr , EThread *t) +{ + return Mutex_lock( +#ifdef DEBUG +location, ahandler, +#endif +m.get(), t); +} + inline void -Mutex_unlock(Ptr , EThread *t) +Mutex_unlock(ProxyMutex *m, EThread *t) { if (m->nthread_holding) { ink_assert(t == m->thread_holding); @@ -351,6 +379,12 @@ Mutex_unlock(Ptr , EThread *t) } } +inline void +Mutex_unlock(Ptr , EThread *t) +{ + Mutex_unlock(m.get(), t); +} + class WeakMutexLock { private: diff --git a/src/traffic_server/InkIOCoreAPI.cc b/src/traffic_server/InkIOCoreAPI.cc index 25618ef..1740ee1 100644 --- a/src/traffic_server/InkIOCoreAPI.cc +++ b/src/traffic_server/InkIOCoreAPI.cc @@ -235,14 +235,17 @@ TSEventThreadSelf(void) // -// Mutexes +// Mutexes: For TSMutexCreate and TSMutexDestroy, the refcount of the +// ProxyMutex object is not incremented or decremented. If the resulting +// ProxyMutex is passed to a INKContInternal, it's mutex smart pointer +// will take ownership of the ProxyMutex and delete it when the last +// reference is removed. TSMutexDestroy should not be called in that case. // TSMutex TSMutexCreate() { ProxyMutex *mutexp = new_ProxyMutex(); - mutexp->refcount_inc(); // TODO: Remove this when allocations can never fail. sdk_assert(sdk_sanity_check_mutex((TSMutex)mutexp) == TS_SUCCESS); @@ -255,9 +258,9 @@ TSMutexDestroy(TSMutex m) { sdk_assert(sdk_sanity_check_mutex(m) == TS_SUCCESS); ProxyMutex *mutexp = reinterpret_cast(m); - // Decrement the refcount added in TSMutexCreate. Delete if this - // was the last ref count - if (mutexp && mutexp->refcount_dec() == 0) { + + if (mutexp) { +ink_release_assert(mutexp->refcount() == 0); mutexp->free(); } } @@ -296,7 +299,7 @@ void TSMutexLock(TSMutex mutexp) { sdk_assert(sdk_sanity_check_mutex(mutexp) == TS_SUCCESS); - Ptr proxy_mutex(reinterpret_cast(mutexp)); + ProxyMutex *proxy_mutex = reinterpret_cast(mutexp); MUTEX_TAKE_LOCK(proxy_mutex, this_ethread()); } @@ -304,7 +307,7 @@ TSReturnCode TSMutexLockTry(TSMutex mutexp) { sdk_assert(sdk_sanity_check_mutex(mutexp) == TS_SUCCESS); - Ptr proxy_mutex(reinterpret_cast(mutexp)); + ProxyMutex *proxy_mutex = reinterpret_cast(mutexp); return (MUTEX_TAKE_TRY_LOCK(proxy_mutex, this_ethread()) ? TS_SUCCESS : TS_ERROR); } @@ -312,7 +315,7 @@ void TSMutexUnlock(TSMutex mutexp) { sdk_assert(sdk_sanity_check_mutex(mutexp) == TS_SUCCESS); - Ptr proxy_mutex(reinterpret_cast(mutexp)); + ProxyMutex *proxy_mutex(reinterpret_cast(mutexp)); MUTEX_UNTAKE_LOCK(proxy_mutex, this_ethread()); }
[trafficserver] 06/07: fix stringstream crash during shutdown
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit 29134f54955b51d88a13a92a5991912cd07e4008 Author: Fei Deng AuthorDate: Wed Nov 20 15:44:40 2019 -0600 fix stringstream crash during shutdown (cherry picked from commit bed0184bb12be5d58e10ae7aefc589d458b2d83b) --- .../experimental/ssl_session_reuse/src/common.cc| 21 - plugins/experimental/ssl_session_reuse/src/common.h | 3 +-- 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/plugins/experimental/ssl_session_reuse/src/common.cc b/plugins/experimental/ssl_session_reuse/src/common.cc index e552c5a..581f6f9 100644 --- a/plugins/experimental/ssl_session_reuse/src/common.cc +++ b/plugins/experimental/ssl_session_reuse/src/common.cc @@ -30,18 +30,21 @@ #include "common.h" -const unsigned char salt[] = {115, 97, 108, 117, 0, 85, 137, 229}; +const unsigned char salt[] = {115, 97, 108, 117, 0, 85, 137, 229}; +const unsigned char hex_chars[] = {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'}; std::string -hex_str(std::string str) +hex_str(std::string const ) { - std::string hex_str = ""; - for (char : str) { -std::stringstream stream; -stream << std::hex << std::setfill('0') << std::setw(2) << (0xFF & static_cast(c)); -hex_str += stream.str(); - } - return hex_str; + size_t len = str.size() * 2 + 1; + char hex_str[len]; + for (unsigned long int i = 0; i < str.size(); ++i) { +unsigned char c= str.at(i); +hex_str[i * 2] = hex_chars[(c & 0xF0) >> 4]; +hex_str[i * 2 + 1] = hex_chars[(c & 0x0F)]; + } + hex_str[len] = '\0'; + return std::string(hex_str, len); } int diff --git a/plugins/experimental/ssl_session_reuse/src/common.h b/plugins/experimental/ssl_session_reuse/src/common.h index bcec00b..377d9fc 100644 --- a/plugins/experimental/ssl_session_reuse/src/common.h +++ b/plugins/experimental/ssl_session_reuse/src/common.h @@ -24,7 +24,6 @@ #pragma once #include -#include #include #include #include @@ -70,7 +69,7 @@ private: std::mutex threads_mutex; }; -std::string hex_str(std::string str); +std::string hex_str(std::string const ); int encrypt_encode64(const unsigned char *key, int key_length, const unsigned char *in_data, int in_data_len, char *out_data, size_t out_data_size, size_t *out_data_len);
[trafficserver] 05/07: Following catch test naming standards.
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit 13dd05e1043c5f71846496549090aa748da972ae Author: bneradt AuthorDate: Sat Nov 23 22:28:46 2019 + Following catch test naming standards. (cherry picked from commit a33b43e5471e927efcba482dae1c3839dee8d693) --- proxy/logging/LogConfig.cc | 3 +-- proxy/logging/unit-tests/test_LogUtils2.cc | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/proxy/logging/LogConfig.cc b/proxy/logging/LogConfig.cc index 816206d..77e8926 100644 --- a/proxy/logging/LogConfig.cc +++ b/proxy/logging/LogConfig.cc @@ -616,8 +616,7 @@ LogConfig::update_space_used() // // then check if the candidate belongs to any given log type // -std::string_view rolled_filename(entry->d_name, strlen(entry->d_name)); -auto iter = deleting_info.find(LogUtils::get_unrolled_filename(rolled_filename)); +auto iter = deleting_info.find(LogUtils::get_unrolled_filename(entry->d_name)); if (iter == deleting_info.end()) { // We won't delete the log if its name doesn't match any give type. continue; diff --git a/proxy/logging/unit-tests/test_LogUtils2.cc b/proxy/logging/unit-tests/test_LogUtils2.cc index 70ccf6f..d50a89b 100644 --- a/proxy/logging/unit-tests/test_LogUtils2.cc +++ b/proxy/logging/unit-tests/test_LogUtils2.cc @@ -131,7 +131,7 @@ RecSignalManager(int, char const *, std::size_t) { } -TEST_CASE("LogUtilsUnrolled", "get_unrolled_filename") +TEST_CASE("get_unrolled_filename parses possible log files as expected", "[get_unrolled_filename]") { // Rolled log inputs. constexpr ts::TextView with_underscore = "squid.log_some.hostname.com.20191029.18h15m02s-20191029.18h30m02s.old";
[trafficserver] 03/07: Fixing rolled log cleanup parsing for .log. files.
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit 6ab28a55a8dd80e296135de0b34b5c80a0702ea3 Author: bneradt AuthorDate: Fri Nov 15 00:15:42 2019 + Fixing rolled log cleanup parsing for .log. files. (cherry picked from commit 84f0219058488b1634190b4a741720964806dfb3) --- proxy/logging/LogConfig.cc | 12 +++--- proxy/logging/LogUtils.cc | 35 ++ proxy/logging/LogUtils.h | 17 +++ proxy/logging/unit-tests/test_LogUtils2.cc | 23 4 files changed, 78 insertions(+), 9 deletions(-) diff --git a/proxy/logging/LogConfig.cc b/proxy/logging/LogConfig.cc index 6500bc1..efc6011 100644 --- a/proxy/logging/LogConfig.cc +++ b/proxy/logging/LogConfig.cc @@ -616,15 +616,9 @@ LogConfig::update_space_used() // // then check if the candidate belongs to any given log type // -ts::TextView type_name(entry->d_name, strlen(entry->d_name)); -// A rolled log will look something like: -// squid.log_some.hostname.com.20191029.18h15m02s-20191029.18h30m02s.old -// -// The following logic cuts things back to original unrolled file which -// is the key into the deleting_info map (squid.log in the above example). -auto suffix = type_name; -type_name.remove_suffix(suffix.remove_prefix(suffix.find('.') + 1).remove_prefix(suffix.find('_')).size()); -auto iter = deleting_info.find(type_name); +ts::TextView rolled_filename(entry->d_name, strlen(entry->d_name)); +auto type_name = LogUtils::get_unrolled_filename(rolled_filename); +auto iter = deleting_info.find(type_name); if (iter == deleting_info.end()) { // We won't delete the log if its name doesn't match any give type. continue; diff --git a/proxy/logging/LogUtils.cc b/proxy/logging/LogUtils.cc index 95e3852..2a314d7 100644 --- a/proxy/logging/LogUtils.cc +++ b/proxy/logging/LogUtils.cc @@ -478,6 +478,41 @@ LogUtils::seconds_to_next_roll(time_t time_now, int rolling_offset, int rolling_ return ((tr >= sidl ? (tr - sidl) % rolling_interval : (86400 - (sidl - tr)) % rolling_interval)); } +ts::TextView +LogUtils::get_unrolled_filename(ts::TextView rolled_filename) +{ + auto unrolled_name = rolled_filename; + + // A rolled log will look something like: + // squid.log_some.hostname.com.20191029.18h15m02s-20191029.18h30m02s.old + auto suffix = rolled_filename; + + suffix.remove_prefix_at('.'); + // Using the above squid.log example, suffix now looks like: + // log_some.hostname.com.20191029.18h15m02s-20191029.18h30m02s.ol + + if (suffix.find('_') != std::string::npos) { +suffix.remove_prefix_at('_'); +// suffix now looks like: +// some.hostname.com.20191029.18h15m02s-20191029.18h30m02s.ol + } else if (suffix.find('.') != std::string::npos) { +// There was no underscore in the file after the first '.'. +// Some logs have the following format (without the hostname): +// diags.log.20191114.21h43m16s-20191114.21h43m17s.old + +suffix.remove_prefix_at('.'); +// Using the diags.log from our above example, suffix now looks like: +// log.20191114.21h43m16s-20191114.21h43m17s.old + } else { +// If there isn't a '.' or an '_' after the first '.', then this +// doesn't look like a rolled file. +return rolled_filename; + } + + // ' + 1' to remove the '_' or second '.': + return unrolled_name.remove_suffix(suffix.size() + 1); +} + // Checks if the file pointed to by full_filename either is a regular // file or a pipe and has write permission, or, if the file does not // exist, if the path prefix of full_filename names a directory that diff --git a/proxy/logging/LogUtils.h b/proxy/logging/LogUtils.h index 2860db1..ade7418 100644 --- a/proxy/logging/LogUtils.h +++ b/proxy/logging/LogUtils.h @@ -26,6 +26,7 @@ #include "tscore/ink_platform.h" #include "tscore/Arena.h" +#include class MIMEHdr; @@ -62,6 +63,22 @@ int seconds_to_next_roll(time_t time_now, int rolling_offset, int rolling_interv int file_is_writeable(const char *full_filename, off_t *size_bytes = nullptr, bool *has_size_limit = nullptr, uint64_t *current_size_limit_bytes = nullptr); +/** Given a rolled file, determine the unrolled filename. + * + * For example, given this: + * diags.log.20191114.21h43m16s-20191114.21h43m17s.old + * + * Return this: + * diags.log + * + * @param[in] rolled_filename The rolled filename from which to derive the + * unrolled filename. + * + * @return The unrolled filename if it looked like a rolled log file or the + * input filename if it didn't. + */ +ts::TextView get_unrolled_filename(ts::TextView rolled_filename); + // Marshals header tags and values together, with a single
[trafficserver] branch 9.0.x updated (79d510e -> f8f3f20)
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a change to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git. from 79d510e Updated Changelog new ddb9015 Removing traffic_cop reference in Admin Guide introduction. new 0f70c8f Adjust the refcounts to avoid Mutex leak new 6ab28a5 Fixing rolled log cleanup parsing for .log. files. new bd73be1 Review comments. new 13dd05e Following catch test naming standards. new 29134f5 fix stringstream crash during shutdown new f8f3f20 Add autest for cache_range_request, add xdebug x-parentselection-key The 7 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: doc/admin-guide/introduction.en.rst| 14 +- doc/admin-guide/plugins/xdebug.en.rst | 6 + doc/static/images/admin/process.jpg| Bin 136229 -> 0 bytes iocore/eventsystem/I_Lock.h| 40 ++- .../experimental/ssl_session_reuse/src/common.cc | 21 +- .../experimental/ssl_session_reuse/src/common.h| 3 +- plugins/xdebug/xdebug.cc | 55 proxy/logging/LogConfig.cc | 10 +- proxy/logging/LogUtils.cc | 28 ++ proxy/logging/LogUtils.h | 17 ++ proxy/logging/unit-tests/test_LogUtils2.cc | 23 ++ src/traffic_server/InkIOCoreAPI.cc | 19 +- .../cache_range_requests.test.py | 340 + .../cache_range_requests/gold/404.stdout.gold | 8 + .../gold/frange.stderr.gold} | 0 .../cache_range_requests/gold/frange.stdout.gold | 11 + .../gold/full.stderr.gold} | 0 .../cache_range_requests/gold/full.stdout.gold | 9 + .../cache_range_requests/gold/inner.stderr.gold| 1 + .../cache_range_requests/gold/inner.stdout.gold| 11 + .../cache_range_requests/gold/last.stderr.gold | 1 + .../cache_range_requests/gold/last.stdout.gold | 11 + .../cache_range_requests/gold/pselect.stdout.gold | 11 + .../gold/pselect_none.stdout.gold | 10 + 24 files changed, 607 insertions(+), 42 deletions(-) delete mode 100644 doc/static/images/admin/process.jpg create mode 100644 tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests.test.py create mode 100644 tests/gold_tests/pluginTest/cache_range_requests/gold/404.stdout.gold copy tests/gold_tests/pluginTest/{slice/gold/slice_206.stderr.gold => cache_range_requests/gold/frange.stderr.gold} (100%) create mode 100644 tests/gold_tests/pluginTest/cache_range_requests/gold/frange.stdout.gold copy tests/gold_tests/pluginTest/{slice/gold/slice_206.stderr.gold => cache_range_requests/gold/full.stderr.gold} (100%) create mode 100644 tests/gold_tests/pluginTest/cache_range_requests/gold/full.stdout.gold create mode 100644 tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stderr.gold create mode 100644 tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stdout.gold create mode 100644 tests/gold_tests/pluginTest/cache_range_requests/gold/last.stderr.gold create mode 100644 tests/gold_tests/pluginTest/cache_range_requests/gold/last.stdout.gold create mode 100644 tests/gold_tests/pluginTest/cache_range_requests/gold/pselect.stdout.gold create mode 100644 tests/gold_tests/pluginTest/cache_range_requests/gold/pselect_none.stdout.gold
[trafficserver] 01/07: Removing traffic_cop reference in Admin Guide introduction.
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit ddb90155b7697f71896a01a21d5132703084fb8f Author: bneradt AuthorDate: Fri Oct 4 15:15:50 2019 + Removing traffic_cop reference in Admin Guide introduction. The introduction to our Administrator's Guide referenced three processes but only described two: traffic_manager and traffic_server. The third process was probably traffic_cop. But we currently now only run with traffic_server and traffic_manager. This adjusts the number of referenced processes from three to two and removes the image which is not as helpful anymore with only two processes that collaborate. (cherry picked from commit f6dadd5de44a994c0b0920ef171eaaa92d94e3b8) --- doc/admin-guide/introduction.en.rst | 14 +++--- doc/static/images/admin/process.jpg | Bin 136229 -> 0 bytes 2 files changed, 3 insertions(+), 11 deletions(-) diff --git a/doc/admin-guide/introduction.en.rst b/doc/admin-guide/introduction.en.rst index 06effa3..e006ed7 100644 --- a/doc/admin-guide/introduction.en.rst +++ b/doc/admin-guide/introduction.en.rst @@ -165,15 +165,15 @@ bindings in memory, DNS traffic is reduced. |TS| Processes -- -|TS| contains three processes that work together to serve +|TS| contains two processes that work together to serve requests and manage, control, and monitor the health of the system. -- The :program:`traffic_server` process is the transaction processing engine +#. The :program:`traffic_server` process is the transaction processing engine of |TS|. It is responsible for accepting connections, processing protocol requests, and serving documents from the cache or origin server. -- The :program:`traffic_manager` process is the command and control facility +#. The :program:`traffic_manager` process is the command and control facility of the |TS|, responsible for launching, monitoring, and reconfiguring the :program:`traffic_server` process. The :program:`traffic_manager` process is also responsible for the proxy autoconfiguration port, the @@ -187,14 +187,6 @@ requests and manage, control, and monitor the health of the system. first-served order. This connection queueing shields users from any server restart downtime. -The figure below illustrates the |TS| processes. - -.. figure:: ../static/images/admin/process.jpg - :align: center - :alt: Illustration of the |TS| Processes - - Illustration of the |TS| Processes - Administration Tools diff --git a/doc/static/images/admin/process.jpg b/doc/static/images/admin/process.jpg deleted file mode 100644 index f683ac7..000 Binary files a/doc/static/images/admin/process.jpg and /dev/null differ
[trafficserver] 07/07: Add autest for cache_range_request, add xdebug x-parentselection-key
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit f8f3f208df2a96c78de09c8d1fa7b8d0f83dcc41 Author: Brian Olsen AuthorDate: Wed Nov 13 22:07:38 2019 + Add autest for cache_range_request, add xdebug x-parentselection-key (cherry picked from commit 83738c69b9df4b134f20da4abb656c78049318f3) --- doc/admin-guide/plugins/xdebug.en.rst | 6 + plugins/xdebug/xdebug.cc | 55 .../cache_range_requests.test.py | 340 + .../cache_range_requests/gold/404.stdout.gold | 8 + .../cache_range_requests/gold/frange.stderr.gold | 1 + .../cache_range_requests/gold/frange.stdout.gold | 11 + .../cache_range_requests/gold/full.stderr.gold | 1 + .../cache_range_requests/gold/full.stdout.gold | 9 + .../cache_range_requests/gold/inner.stderr.gold| 1 + .../cache_range_requests/gold/inner.stdout.gold| 11 + .../cache_range_requests/gold/last.stderr.gold | 1 + .../cache_range_requests/gold/last.stdout.gold | 11 + .../cache_range_requests/gold/pselect.stdout.gold | 11 + .../gold/pselect_none.stdout.gold | 10 + 14 files changed, 476 insertions(+) diff --git a/doc/admin-guide/plugins/xdebug.en.rst b/doc/admin-guide/plugins/xdebug.en.rst index bb5f376..16e358d 100644 --- a/doc/admin-guide/plugins/xdebug.en.rst +++ b/doc/admin-guide/plugins/xdebug.en.rst @@ -109,3 +109,9 @@ X-Transaction-ID X-Remap If the URL was remapped for a request, this header gives the *to* and *from* field from the line in remap.config that caused the URL to be remapped. + +X-ParentSelection-Key +The ``X-ParentSelection-Key`` header contains the URL that is used to +determine parent selection for an object in the Traffic Server. This +header is particularly useful if a custom parent selection key is +being used. diff --git a/plugins/xdebug/xdebug.cc b/plugins/xdebug/xdebug.cc index 1b6d877..a3a2443 100644 --- a/plugins/xdebug/xdebug.cc +++ b/plugins/xdebug/xdebug.cc @@ -50,6 +50,7 @@ enum { XHEADER_X_DUMP_HEADERS = 1u << 7, XHEADER_X_REMAP = 1u << 8, XHEADER_X_PROBE_HEADERS = 1u << 9, + XHEADER_X_PSELECT_KEY= 1u << 10, }; static int XArgIndex = 0; @@ -325,6 +326,53 @@ InjectTxnUuidHeader(TSHttpTxn txn, TSMBuffer buffer, TSMLoc hdr) } } +static void +InjectParentSelectionKeyHeader(TSHttpTxn txn, TSMBuffer buffer, TSMLoc hdr) +{ + TSMLoc url = TS_NULL_MLOC; + TSMLoc dst = TS_NULL_MLOC; + + struct { +char *ptr; +int len; + } strval = {nullptr, 0}; + + TSDebug("xdebug", "attempting to inject X-ParentSelection-Key header"); + + if (TSUrlCreate(buffer, ) != TS_SUCCESS) { +goto done; + } + + if (TSHttpTxnParentSelectionUrlGet(txn, buffer, url) != TS_SUCCESS) { +goto done; + } + + strval.ptr = TSUrlStringGet(buffer, url, ); + if (strval.ptr == nullptr || strval.len == 0) { +goto done; + } + + // Create a new response header field. + dst = FindOrMakeHdrField(buffer, hdr, "X-ParentSelection-Key", lengthof("X-ParentSelection-Key")); + if (dst == TS_NULL_MLOC) { +goto done; + } + + // Now copy the parent selection lookup URL into the response header. + TSReleaseAssert(TSMimeHdrFieldValueStringInsert(buffer, hdr, dst, 0 /* idx */, strval.ptr, strval.len) == TS_SUCCESS); + +done: + if (dst != TS_NULL_MLOC) { +TSHandleMLocRelease(buffer, hdr, dst); + } + + if (url != TS_NULL_MLOC) { +TSHandleMLocRelease(buffer, TS_NULL_MLOC, url); + } + + TSfree(strval.ptr); +} + static int XInjectResponseHeaders(TSCont /* contp */, TSEvent event, void *edata) { @@ -384,6 +432,10 @@ XInjectResponseHeaders(TSCont /* contp */, TSEvent event, void *edata) writePostBody(data); } + if (xheaders & XHEADER_X_PSELECT_KEY) { +InjectParentSelectionKeyHeader(txn, buffer, hdr); + } + done: TSHttpTxnReenable(txn, TS_EVENT_HTTP_CONTINUE); return TS_EVENT_NONE; @@ -525,6 +577,9 @@ XScanRequestHeaders(TSCont /* contp */, TSEvent event, void *edata) TSHttpTxnTransformedRespCache(txn, 0); TSHttpTxnUntransformedRespCache(txn, 0); + } else if (header_field_eq("x-parentselection-key", value, vsize)) { +xheaders |= XHEADER_X_PSELECT_KEY; + } else if (isFwdFieldValue(std::string_view(value, vsize), fwdCnt)) { if (fwdCnt > 0) { // Decrement forward count in X-Debug header. diff --git a/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests.test.py b/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests.test.py new file mode 100644 index 000..49aaf6e --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests.test.py @@ -0,0 +1,340 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more
[trafficserver] 04/07: Review comments.
This is an automated email from the ASF dual-hosted git repository. zwoop pushed a commit to branch 9.0.x in repository https://gitbox.apache.org/repos/asf/trafficserver.git commit bd73be1a1b0f525706219298bc74ef26efbbc03a Author: bneradt AuthorDate: Wed Nov 20 15:48:46 2019 + Review comments. (cherry picked from commit 745d38f63441f0a6eec64cc81a29f39b9333b8fe) --- proxy/logging/LogConfig.cc | 5 ++--- proxy/logging/LogUtils.cc | 35 ++- 2 files changed, 16 insertions(+), 24 deletions(-) diff --git a/proxy/logging/LogConfig.cc b/proxy/logging/LogConfig.cc index efc6011..816206d 100644 --- a/proxy/logging/LogConfig.cc +++ b/proxy/logging/LogConfig.cc @@ -616,9 +616,8 @@ LogConfig::update_space_used() // // then check if the candidate belongs to any given log type // -ts::TextView rolled_filename(entry->d_name, strlen(entry->d_name)); -auto type_name = LogUtils::get_unrolled_filename(rolled_filename); -auto iter = deleting_info.find(type_name); +std::string_view rolled_filename(entry->d_name, strlen(entry->d_name)); +auto iter = deleting_info.find(LogUtils::get_unrolled_filename(rolled_filename)); if (iter == deleting_info.end()) { // We won't delete the log if its name doesn't match any give type. continue; diff --git a/proxy/logging/LogUtils.cc b/proxy/logging/LogUtils.cc index 2a314d7..4ec68e9 100644 --- a/proxy/logging/LogUtils.cc +++ b/proxy/logging/LogUtils.cc @@ -489,28 +489,21 @@ LogUtils::get_unrolled_filename(ts::TextView rolled_filename) suffix.remove_prefix_at('.'); // Using the above squid.log example, suffix now looks like: - // log_some.hostname.com.20191029.18h15m02s-20191029.18h30m02s.ol - - if (suffix.find('_') != std::string::npos) { -suffix.remove_prefix_at('_'); -// suffix now looks like: -// some.hostname.com.20191029.18h15m02s-20191029.18h30m02s.ol - } else if (suffix.find('.') != std::string::npos) { -// There was no underscore in the file after the first '.'. -// Some logs have the following format (without the hostname): -// diags.log.20191114.21h43m16s-20191114.21h43m17s.old - -suffix.remove_prefix_at('.'); -// Using the diags.log from our above example, suffix now looks like: -// log.20191114.21h43m16s-20191114.21h43m17s.old - } else { -// If there isn't a '.' or an '_' after the first '.', then this -// doesn't look like a rolled file. -return rolled_filename; - } + // log_some.hostname.com.20191029.18h15m02s-20191029.18h30m02s.old - // ' + 1' to remove the '_' or second '.': - return unrolled_name.remove_suffix(suffix.size() + 1); + // Some suffixes do not have the hostname. Rolled diags.log files will look + // something like this, for example: + // diags.log.20191114.21h43m16s-20191114.21h43m17s.old + // + // For these, the second delimeter will be a period. For this reason, we also + // split_prefix_at with a period as well. + if (suffix.split_prefix_at('_') || suffix.split_prefix_at('.')) { +// ' + 1' to remove the '_' or second '.': +return unrolled_name.remove_suffix(suffix.size() + 1); + } + // If there isn't a '.' or an '_' after the first '.', then this + // doesn't look like a rolled file. + return rolled_filename; } // Checks if the file pointed to by full_filename either is a regular
[trafficserver] branch master updated (f6dadd5 -> 8fcb137)
This is an automated email from the ASF dual-hosted git repository. bcall pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/trafficserver.git. from f6dadd5 Removing traffic_cop reference in Admin Guide introduction. add 8fcb137 Adding verify plugin TS maintenance commands No new revisions were added by this update. Summary of changes: .gitignore | 2 + doc/appendices/command-line/traffic_server.en.rst | 44 - doc/release-notes/whats-new.en.rst | 7 +- include/tscore/ts_file.h | 9 + proxy/Plugin.cc| 46 +++-- proxy/Plugin.h | 2 + proxy/http/remap/RemapPluginInfo.h | 10 +- src/traffic_server/traffic_server.cc | 169 +- src/tscore/ts_file.cc | 12 +- src/tscore/unit_tests/test_ts_file.cc | 18 +- .../command_argument/verify_global_plugin.test.py | 198 + .../command_argument/verify_remap_plugin.test.py | 146 +++ .../tools/plugins/conf_remap_stripped.cc | 47 ++--- .../tools/plugins/missing_mangled_definition.c | 15 +- .../tools/plugins/missing_mangled_definition.cc| 17 +- .../tools/plugins/missing_mangled_definition.h | 5 +- .../tools/plugins/missing_ts_plugin_init.cc| 12 +- 17 files changed, 683 insertions(+), 76 deletions(-) create mode 100644 tests/gold_tests/command_argument/verify_global_plugin.test.py create mode 100644 tests/gold_tests/command_argument/verify_remap_plugin.test.py copy proxy/http/remap/unit-tests/plugin_init_fail.cc => tests/tools/plugins/conf_remap_stripped.cc (58%) copy plugins/experimental/uri_signing/timing.c => tests/tools/plugins/missing_mangled_definition.c (86%) copy plugins/experimental/uri_signing/timing.c => tests/tools/plugins/missing_mangled_definition.cc (66%) copy plugins/experimental/uri_signing/timing.c => tests/tools/plugins/missing_mangled_definition.h (87%) copy plugins/experimental/uri_signing/timing.c => tests/tools/plugins/missing_ts_plugin_init.cc (84%)
[trafficserver] branch master updated: Removing traffic_cop reference in Admin Guide introduction.
This is an automated email from the ASF dual-hosted git repository. bcall pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/trafficserver.git The following commit(s) were added to refs/heads/master by this push: new f6dadd5 Removing traffic_cop reference in Admin Guide introduction. f6dadd5 is described below commit f6dadd5de44a994c0b0920ef171eaaa92d94e3b8 Author: bneradt AuthorDate: Fri Oct 4 15:15:50 2019 + Removing traffic_cop reference in Admin Guide introduction. The introduction to our Administrator's Guide referenced three processes but only described two: traffic_manager and traffic_server. The third process was probably traffic_cop. But we currently now only run with traffic_server and traffic_manager. This adjusts the number of referenced processes from three to two and removes the image which is not as helpful anymore with only two processes that collaborate. --- doc/admin-guide/introduction.en.rst | 14 +++--- doc/static/images/admin/process.jpg | Bin 136229 -> 0 bytes 2 files changed, 3 insertions(+), 11 deletions(-) diff --git a/doc/admin-guide/introduction.en.rst b/doc/admin-guide/introduction.en.rst index 06effa3..e006ed7 100644 --- a/doc/admin-guide/introduction.en.rst +++ b/doc/admin-guide/introduction.en.rst @@ -165,15 +165,15 @@ bindings in memory, DNS traffic is reduced. |TS| Processes -- -|TS| contains three processes that work together to serve +|TS| contains two processes that work together to serve requests and manage, control, and monitor the health of the system. -- The :program:`traffic_server` process is the transaction processing engine +#. The :program:`traffic_server` process is the transaction processing engine of |TS|. It is responsible for accepting connections, processing protocol requests, and serving documents from the cache or origin server. -- The :program:`traffic_manager` process is the command and control facility +#. The :program:`traffic_manager` process is the command and control facility of the |TS|, responsible for launching, monitoring, and reconfiguring the :program:`traffic_server` process. The :program:`traffic_manager` process is also responsible for the proxy autoconfiguration port, the @@ -187,14 +187,6 @@ requests and manage, control, and monitor the health of the system. first-served order. This connection queueing shields users from any server restart downtime. -The figure below illustrates the |TS| processes. - -.. figure:: ../static/images/admin/process.jpg - :align: center - :alt: Illustration of the |TS| Processes - - Illustration of the |TS| Processes - Administration Tools diff --git a/doc/static/images/admin/process.jpg b/doc/static/images/admin/process.jpg deleted file mode 100644 index f683ac7..000 Binary files a/doc/static/images/admin/process.jpg and /dev/null differ