[jira] [Work logged] (TS-4042) Add feature to buffer request body before making downstream requests
[ https://issues.apache.org/jira/browse/TS-4042?focusedWorklogId=26502=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-26502 ] ASF GitHub Bot logged work on TS-4042: -- Author: ASF GitHub Bot Created on: 16/Aug/16 14:46 Start Date: 16/Aug/16 14:46 Worklog Time Spent: 10m Work Description: Github user jacksontj commented on the issue: https://github.com/apache/trafficserver/pull/351 @bryancall it'd be nice if we could have some way to "re-enable" the bytes-- such that a plugin could either buffer everything or stream (since the "buffer everything" case is a possible use-case. Issue Time Tracking --- Worklog Id: (was: 26502) Time Spent: 40m (was: 0.5h) > Add feature to buffer request body before making downstream requests > > > Key: TS-4042 > URL: https://issues.apache.org/jira/browse/TS-4042 > Project: Traffic Server > Issue Type: Improvement > Components: Core, CPP API, TS API >Reporter: Brian Geffon >Assignee: Brian Geffon > Fix For: 7.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > We need a way to examine the request body without making a downstream > request, this feature has many use cases including: > - Ability to buffer the body and ensure a full post is received before > committing downstream resources. > - Ability to choose an origin based on request body > - Ability to do request content filtering such as a WAF might provide > before the origin is involved. > Today you have two options to inspect a request body: > 1) Transformations: the problem with transformations is that you only start > receiving the request bytes after a sink has been established, which in this > case is the downstream origin. > 2) Create an intercept and use fetch apis to then send the downstream > request: while this technically works it turns out to be a ton of code and is > in general pretty problematic, we actually tried this approach for a while > and had nothing but problems with it. > We feel it would be ideal if we could intercept the body without breaking the > normal ATS state flow. There used to exist code (and it's still in the core > just #ifdefed out) to drain the request body. I use that code as the basis > for this request buffering code. We added APIs to both the C and C++ APIs so > that this request buffering can be enabled from a plugin and the plugin can > inspect the body as chunks arrive or when it's complete. We've included an > example plugin that will error a transaction if a minimum rate of transfer is > not maintained. > I'm confident that this feature will bring plenty of questions / feedback, so > let's get that party started. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work logged] (TS-4042) Add feature to buffer request body before making downstream requests
[ https://issues.apache.org/jira/browse/TS-4042?focusedWorklogId=26490=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-26490 ] ASF GitHub Bot logged work on TS-4042: -- Author: ASF GitHub Bot Created on: 16/Aug/16 03:38 Start Date: 16/Aug/16 03:38 Worklog Time Spent: 10m Work Description: Github user bryancall commented on the issue: https://github.com/apache/trafficserver/pull/351 Looking over the pull request I didn't see any limits on how much is going to be buffered on the server. It might be good to start the transfer to the origin once a configurable limite gets reached. Issue Time Tracking --- Worklog Id: (was: 26490) Time Spent: 0.5h (was: 20m) > Add feature to buffer request body before making downstream requests > > > Key: TS-4042 > URL: https://issues.apache.org/jira/browse/TS-4042 > Project: Traffic Server > Issue Type: Improvement > Components: Core, CPP API, TS API >Reporter: Brian Geffon >Assignee: Brian Geffon > Fix For: 7.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We need a way to examine the request body without making a downstream > request, this feature has many use cases including: > - Ability to buffer the body and ensure a full post is received before > committing downstream resources. > - Ability to choose an origin based on request body > - Ability to do request content filtering such as a WAF might provide > before the origin is involved. > Today you have two options to inspect a request body: > 1) Transformations: the problem with transformations is that you only start > receiving the request bytes after a sink has been established, which in this > case is the downstream origin. > 2) Create an intercept and use fetch apis to then send the downstream > request: while this technically works it turns out to be a ton of code and is > in general pretty problematic, we actually tried this approach for a while > and had nothing but problems with it. > We feel it would be ideal if we could intercept the body without breaking the > normal ATS state flow. There used to exist code (and it's still in the core > just #ifdefed out) to drain the request body. I use that code as the basis > for this request buffering code. We added APIs to both the C and C++ APIs so > that this request buffering can be enabled from a plugin and the plugin can > inspect the body as chunks arrive or when it's complete. We've included an > example plugin that will error a transaction if a minimum rate of transfer is > not maintained. > I'm confident that this feature will bring plenty of questions / feedback, so > let's get that party started. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work logged] (TS-4042) Add feature to buffer request body before making downstream requests
[ https://issues.apache.org/jira/browse/TS-4042?focusedWorklogId=26484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-26484 ] ASF GitHub Bot logged work on TS-4042: -- Author: ASF GitHub Bot Created on: 16/Aug/16 02:24 Start Date: 16/Aug/16 02:24 Worklog Time Spent: 10m Work Description: Github user bryancall commented on a diff in the pull request: https://github.com/apache/trafficserver/pull/351#discussion_r74867909 --- Diff: proxy/InkAPI.cc --- @@ -8903,3 +8910,46 @@ TSVConnReenable(TSVConn vconn) } } } + +tsapi char * +TSHttpTxnGetClientRequestBody(TSHttpTxn txnp, int *len) --- End diff -- Why are you adding this API if you are not using it? Issue Time Tracking --- Worklog Id: (was: 26484) Time Spent: 20m (was: 10m) > Add feature to buffer request body before making downstream requests > > > Key: TS-4042 > URL: https://issues.apache.org/jira/browse/TS-4042 > Project: Traffic Server > Issue Type: Improvement > Components: Core, CPP API, TS API >Reporter: Brian Geffon >Assignee: Brian Geffon > Fix For: 7.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We need a way to examine the request body without making a downstream > request, this feature has many use cases including: > - Ability to buffer the body and ensure a full post is received before > committing downstream resources. > - Ability to choose an origin based on request body > - Ability to do request content filtering such as a WAF might provide > before the origin is involved. > Today you have two options to inspect a request body: > 1) Transformations: the problem with transformations is that you only start > receiving the request bytes after a sink has been established, which in this > case is the downstream origin. > 2) Create an intercept and use fetch apis to then send the downstream > request: while this technically works it turns out to be a ton of code and is > in general pretty problematic, we actually tried this approach for a while > and had nothing but problems with it. > We feel it would be ideal if we could intercept the body without breaking the > normal ATS state flow. There used to exist code (and it's still in the core > just #ifdefed out) to drain the request body. I use that code as the basis > for this request buffering code. We added APIs to both the C and C++ APIs so > that this request buffering can be enabled from a plugin and the plugin can > inspect the body as chunks arrive or when it's complete. We've included an > example plugin that will error a transaction if a minimum rate of transfer is > not maintained. > I'm confident that this feature will bring plenty of questions / feedback, so > let's get that party started. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work logged] (TS-4042) Add feature to buffer request body before making downstream requests
[
https://issues.apache.org/jira/browse/TS-4042?focusedWorklogId=26482=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-26482
]
ASF GitHub Bot logged work on TS-4042:
--
Author: ASF GitHub Bot
Created on: 16/Aug/16 02:22
Start Date: 16/Aug/16 02:22
Worklog Time Spent: 10m
Work Description: Github user bryancall commented on a diff in the pull
request:
https://github.com/apache/trafficserver/pull/351#discussion_r74867804
--- Diff: plugins/request_buffer/request_buffer.cc ---
@@ -0,0 +1,119 @@
+/* request_buffer.cc - Plugin to enable request buffer for the given
transaction.
+ */
+
+#include
+#include
+#include
+#include
+#include
+
+#include "ts/ts.h"
+#include "ts/ink_defs.h"
+
+#define PLUGIN_NAME "request_buffer"
+
+static const int MIN_BYTE_PER_SEC = 1000;
+static int TXN_INDEX_ARG_TIME;
+struct TimeRecord {
+ timespec start_time;
+ TimeRecord() { clock_gettime(CLOCK_MONOTONIC, _time); }
+};
+bool
+is_post_request(TSHttpTxn txnp)
+{
+ const char *method;
+ int method_len;
+ TSMLoc req_loc;
+ TSMBuffer req_bufp;
+ if (TSHttpTxnClientReqGet(txnp, _bufp, _loc) == TS_ERROR) {
+TSError("Error while retrieving client request header\n");
+return false;
+ }
+ method = TSHttpHdrMethodGet(req_bufp, req_loc, _len);
+ if (static_cast(method_len) != strlen(TS_HTTP_METHOD_POST) ||
strncasecmp(method, TS_HTTP_METHOD_POST, method_len) != 0) {
+TSHandleMLocRelease(req_bufp, TS_NULL_MLOC, req_loc);
+return false;
+ }
+ TSHandleMLocRelease(req_bufp, TS_NULL_MLOC, req_loc);
+ return true;
+}
+bool
+reached_min_speed(TSHttpTxn txnp, int body_len)
+{
+ TimeRecord *timeRecord = (TimeRecord *)TSHttpTxnArgGet(txnp,
TXN_INDEX_ARG_TIME);
+ timespec now_time;
+ clock_gettime(CLOCK_MONOTONIC, _time);
+ double time_diff_in_sec =
+(now_time.tv_sec - timeRecord->start_time.tv_sec) + 1e-9 *
(now_time.tv_nsec - timeRecord->start_time.tv_nsec);
+ TSDebug("http", "time_diff_in_sec = %f, body_len = %d, date_rate =
%f\n", time_diff_in_sec, body_len,
+ body_len / time_diff_in_sec);
+ return body_len / time_diff_in_sec >= MIN_BYTE_PER_SEC;
+}
+static int
+hook_handler(TSCont contp, TSEvent event, void *edata)
+{
+ TSHttpTxn txnp = (TSHttpTxn)(edata);
+ if (event == TS_EVENT_HTTP_READ_REQUEST_HDR && is_post_request(txnp)) {
+// enable the request body buffering
+TSHttpTxnConfigIntSet(txnp, TS_CONFIG_HTTP_REQUEST_BUFFER_ENABLED, 1);
+
+// save the start time for calculating the data rate
+TimeRecord *timeRecord = new TimeRecord();
+TSHttpTxnArgSet(txnp, TXN_INDEX_ARG_TIME, static_cast(timeRecord));
+
+TSHttpTxnHookAdd(txnp, TS_HTTP_REQUEST_BUFFER_READ_HOOK,
TSContCreate(hook_handler, TSMutexCreate()));
+TSHttpTxnHookAdd(txnp, TS_HTTP_REQUEST_BUFFER_READ_COMPLETE_HOOK,
TSContCreate(hook_handler, TSMutexCreate()));
+TSHttpTxnHookAdd(txnp, TS_HTTP_TXN_CLOSE_HOOK,
TSContCreate(hook_handler, TSMutexCreate()));
+ } else if (event == TS_EVENT_HTTP_REQUEST_BUFFER_READ || event ==
TS_EVENT_HTTP_REQUEST_BUFFER_COMPLETE) {
+int64_t ret_len = TSHttpTxnClientReqBodyBytesGet(txnp);
+if (event == TS_EVENT_HTTP_REQUEST_BUFFER_READ &&
!reached_min_speed(txnp, ret_len)) {
+ TSError("[hook_handler] Error : reached_min_speed checking
failed\n");
+ TSHttpTxnReenable(txnp, TS_EVENT_ERROR);
+ return 0;
+}
+
+// get the received request body
+TSIOBufferReader buffer_reader =
TSHttpTxnGetClientRequestBufferReader(txnp);
+int64_t read_avail = TSIOBufferReaderAvail(buffer_reader);
+if (read_avail) {
+ char *body = (char *)TSmalloc(sizeof(char) * read_avail);
+ int64_t consumed = 0;
+ int64_t data_len = 0;
+ const char *char_data = NULL;
+ TSIOBufferBlock block = TSIOBufferReaderStart(buffer_reader);
+ while (block != NULL) {
+char_data = TSIOBufferBlockReadStart(block, buffer_reader,
_len);
+memcpy(body + consumed, char_data, data_len);
+consumed += data_len;
+block = TSIOBufferBlockNext(block);
+ }
+ // play with the body
--- End diff --
Why are you copying the body if you aren't doing anything with it?
Issue Time Tracking
---
Worklog Id: (was: 26482)
Time Spent: 10m
Remaining Estimate: 0h
> Add feature to buffer request body before making downstream requests
>
>
> Key: TS-4042
> URL:
