I think DK is referring to pipelining here. The CDN client can potentially send
requests from separate clients on the same ATS connection when pipelining is
used.
As to disabling the pipelining, you are right - the settings that were on ATS
would not actually do anything (I got mixed up with another proxy :), where we
did have a way to not accept pipeline requests). However, for DK’s scenario
though, assuming the CDN client supported it, it should be possible to turn
pipelining off from the CDN.
@Weixi, Understood. Yes, if you want to be super accurate, you’d have to
implement something custom.
> On Jun 25, 2019, at 7:03 AM, Leif Hedstrom <zw...@apache.org> wrote:
>
>
>
>> On Jun 24, 2019, at 22:42, Dk Jack <dnj0...@gmail.com> wrote:
>>
>> Perhaps. I think we are talking two different things. I was trying to say,
>> it’s a bad idea to stall a client connection when multiple clients are
>> getting multiplexed on the same connection (between cdn and ats). From that
>> perspective, the result is the same on the client side of ats. Stalling one
>> client will cause problems for other client on that connection
>
>
> Hmm, does that happen? When a client makes a request, an origin connection is
> allocated from the pools. I don’t think two client connections can share a
> server connection at the same time; They can be reused between multiple
> client connections though.
>
> — Leif
>>
>> Dk.
>>
>>> On Jun 24, 2019, at 9:01 PM, Sudheer Vinukonda
>>> <sudheervinuko...@yahoo.com.INVALID> wrote:
>>>
>>>
>>>
>>>> On Jun 24, 2019, at 8:33 PM, Dk Jack <dnj0...@gmail.com> wrote:
>>>>
>>>> I suspect connection pools would be using the http pipeline feature to
>>>> multiplex multiple client connections on to the pool connections towards
>>>> ATS.
>>>
>>> Right. But, you can turn off the pipelining support on ATS, if there isn’t
>>> a clear benefit (particularly, true if your client is a CDN which can
>>> already leverage connection pools).
>>>
>>>> In my case, rate-limiting was implemented as a security feature i.e
>>>> rate-limiting malicious IPs etc. Moreover, we don't have control over it at
>>>> the entry point... since most of the time we have to look at the requests
>>>> at the http level (sometimes even into the bodies) to figure if we should
>>>> apply rate-limiting or not...
>>>>
>>>> On Mon, Jun 24, 2019 at 7:28 PM Sudheer Vinukonda
>>>> <sudheervinuko...@yahoo.com.invalid> wrote:
>>>>
>>>>> Interesting setup - But, wouldn’t the CDN have connection pools to ATS
>>>>> already? Curious to know if enabling HTTP pipelining provides benefits
>>>>> beyond what connection pools can give.
>>>>>
>>>>> That said, I don’t think it makes a lot of sense to implement rate
>>>>> limiting for _end clients_ in this sort of a setup. Typically, you’d want
>>>>> to rate limit at the entry point of the user traffic, which means the rate
>>>>> limiting for end users would be done at the CDN in this setup. You could
>>>>> still add rate limiting for the CDN itself but yes in that case it makes
>>>>> sense to just reject the requests and let the CDN retry.
>>>>>
>>>>> @Weixi Li,
>>>>>
>>>>> The more I read about your use case, it seems like you could probably
>>>>> directly leverage the rate limiting that ATS core already supports (unless
>>>>> you’ve other custom requirements). You can configure the below settings
>>>>> per
>>>>> origin service in your remap config, and you should get the behavior your
>>>>> use case described.
>>>>>
>>>>>
>>>>> proxy.config.http.per_server.connection.max
>>>>> Scope
>>>>> CONFIG
>>>>> Type
>>>>> INT
>>>>> Default
>>>>> 0
>>>>> Reloadable
>>>>> Yes
>>>>> Overridable
>>>>> Yes
>>>>> Set a limit for the number of concurrent connections to an upstream server
>>>>> group. A value of 0 disables checking. If a transaction attempts to
>>>>> connect
>>>>> to a group which already has the maximum number of concurrent connections
>>>>> the transaction either rechecks after a delay or a 503
>>>>> (HTTP_STATUS_SERVICE_UNAVAILABLE) error response is sent to the user
>>>>> agent.
>>>>> To configure
>>>>>
>>>>> Number of transactions that can be delayed concurrently
>>>>> See proxy.config.http.per_server.connection.queue_size.
>>>>>
>>>>> How long to delay before rechecking
>>>>> See proxy.config.http.per_server.connection.queue_delay.
>>>>>
>>>>> Upstream server group definition
>>>>>
>>>>>
>>>>>
>>>>>> On Jun 24, 2019, at 6:58 PM, Dk Jack <dnj0...@gmail.com> wrote:
>>>>>>
>>>>>> Leif,
>>>>>> I was talking about the case where ATS is deployed after CDN.
>>>>>>
>>>>>> client --> CDN Network -------> ATS --> Origin-Server
>>>>>>
>>>>>> In most such deployments, the connection between the CDN and ATS is HTTP
>>>>>> 1.1. and it is pipelined. In that case, you cannot stall the clients
>>>>>> because you could run into head-of-line blocking situation.
>>>>>>
>>>>>>
>>>>>>> On Mon, Jun 24, 2019 at 4:07 PM Leif Hedstrom <zw...@apache.org> wrote:
>>>>>>>
>>>>>>> You definitely would not want to stall the client after you’ve made an
>>>>>>> origin connection. I.e. you only want to stall the UA.
>>>>>>>
>>>>>>> — Leif
>>>>>>>
>>>>>>>> On Jun 24, 2019, at 14:42, Dk Jack <dnj0...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Also, stalling will not be good, if you have CDN upstream from you that
>>>>>>> is pipelining requests on the same connection for multiple clients. In
>>>>> our
>>>>>>> plugin we tried delay request, but we couldn’t solve http 1.1 pipeline
>>>>>>> issue. Hence ended up going with the simple approach of deny.
>>>>>>>>
>>>>>>>> Dk.
>>>>>>>>
>>>>>>>>> On Jun 24, 2019, at 1:12 PM, Leif Hedstrom <zw...@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Jun 24, 2019, at 13:12, Weixi Li (BLOOMBERG/ PRINCETON) <
>>>>>>> wli...@bloomberg.net> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Dk,
>>>>>>>>>>
>>>>>>>>>> Thanks a lot for the example. It is really inspiring. I have a
>>>>>>> question though: in case of "bucket->consume()" returning false, the
>>>>> code
>>>>>>> would return a 429 response. In our use-case, we don't want to deny the
>>>>>>> client with a 429 (but want to queue the request instead). How to handle
>>>>>>> the false case then? Should addPlugin() add the plugin itself? (That's
>>>>> why
>>>>>>> I was asking about queue and async timer).
>>>>>>>>>
>>>>>>>>> Oh do you want to stall the client, not deny? Depending on the plugin
>>>>>>> and the hook, you can probably just reschedule the HttpSM to come back
>>>>> on
>>>>>>> say 500ms and try again, rather than let it continue like you normally
>>>>>>> would. Have to make sure they don’t stall fore ever though, and perhaps
>>>>>>> have a priority such that older clients are given tokens before new
>>>>> ones.
>>>>>>>>>
>>>>>>>>> Should be doable, but definitely more complex.
>>>>>>>>>
>>>>>>>>> — Leif
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Weixi
>>>>>>>>>>
>>>>>>>>>> From: dev@trafficserver.apache.org At: 06/24/19 01:28:40To: Weixi
>>>>> Li
>>>>>>> (BLOOMBERG/ PRINCETON ) , dev@trafficserver.apache.org
>>>>>>>>>> Subject: Re: Implementing Rate-limiting in forward proxy mode
>>>>>>>>>>
>>>>>>>>>> Not sure you need queues. Here's a sample implementation based on
>>>>>>>>>> TokenBucket which should do what you want...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> void
>>>>>>>>>> handleReadRequestHeadersPreRemap(Transaction& transaction)
>>>>>>>>>> {
>>>>>>>>>> std::string host;
>>>>>>>>>> Headers& headers = transaction.getClientRequest().getHeaders();
>>>>>>>>>> Headers::iterator ii = headers.find("Host");
>>>>>>>>>>
>>>>>>>>>> if (ii != headers.end()) {
>>>>>>>>>> host = (*ii).values();
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> if (!host.empty()) {
>>>>>>>>>> auto bucket = map.find(host); // assumes u created a token bucket
>>>>>>> per
>>>>>>>>>> site.
>>>>>>>>>> // std::map<std::string,
>>>>>>>>>> std::shared_ptr<TokenBucket>>
>>>>>>>>>> // map.emplace("www.cnn.com",
>>>>>>>>>> // std::make_shred<
>>>>>>>>>> TokenBucket>(rate, burst));
>>>>>>>>>> // rate is number of req.s/min.
>>>>>>>>>> // burst can be same as rate.
>>>>>>>>>>
>>>>>>>>>> if (bucket->consume(1)) { // you are requesting 1 token.
>>>>>>>>>> transaction.resume();
>>>>>>>>>> } else {
>>>>>>>>>> // see CustomResponseTransactionPlugin in atscppapi examples.
>>>>>>>>>> transaction.addPlugin(new
>>>>>>>>>> CustomResponseTransactionPlugin(transaction, 429,
>>>>>>>>>> "Too Many Requests", "Too Many Requests"));
>>>>>>>>>> return;
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> transaction.resume();
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Jun 23, 2019 at 8:04 PM Weixi Li (BLOOMBERG/ PRINCETON) <
>>>>>>>>>> wli...@bloomberg.net> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Dk/Eric/Leif,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the advice. Our use-case simply requires sending out the
>>>>>>> queued
>>>>>>>>>>> requests at a fixed rate (although variable by site). My plan was
>>>>>>> simply
>>>>>>>>>>> put all incoming transactions in separate queues by site, and then
>>>>>>> invoke
>>>>>>>>>>> "transaction.resume()" at fixed interval via a timer for each queue.
>>>>>>> Would
>>>>>>>>>>> the delay (calling "transaction.resume()" asynchronously) disrupt
>>>>> the
>>>>>>> ATS
>>>>>>>>>>> event processing? If yes, what's correct way to avoid that? If no,
>>>>> is
>>>>>>> there
>>>>>>>>>>> any example plugin implementing such a timer (was thinking about
>>>>>>> AsyncTimer
>>>>>>>>>>> in atscppapi)?
>>>>>>>>>>>
>>>>>>>>>>> The TokenBucket is a very interesting data structure, but I've not
>>>>>>> figured
>>>>>>>>>>> out how to apply it yet. Will look into it a bit more.
>>>>>>>>>>>
>>>>>>>>>>> Thanks to Eric's point, TS_HTTP_SEND_REQUEST_HDR_HOOK will indeed be
>>>>>>> too
>>>>>>>>>>> late. But for this use-case the "rate-limit" only applies to
>>>>>>> cache-misses
>>>>>>>>>>> and revalidations (fresh cache-hits should be served as fast as
>>>>>>> possible).
>>>>>>>>>>> Therefore the hook probably should be
>>>>>>> TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK.
>>>>>>>>>>> The plugin then lets the transaction resume immediately if the cache
>>>>>>> hit is
>>>>>>>>>>> fresh, or queue the transaction otherwise.
>>>>>>>>>>>
>>>>>>>>>>> Still learning ATS, let me know if I'm wrong. Thanks!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> From: dev@trafficserver.apache.org At: 06/21/19 23:11:03To: Weixi
>>>>> Li
>>>>>>>>>>> (BLOOMBERG/ PRINCETON ) , dev@trafficserver.apache.org
>>>>>>>>>>> Subject: Re: Implementing Rate-limiting in forward proxy mode
>>>>>>>>>>>
>>>>>>>>>>> Uhm! why async timers? You'd want to implement a leaky/token bucket
>>>>>>> per
>>>>>>>>>>> site. Check out...
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/rigtorp/TokenBucket
>>>>>>>>>>>
>>>>>>>>>>> It's single header file lock free implementation for token bucket
>>>>> and
>>>>>>> it
>>>>>>>>>>> works very well...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jun 21, 2019 at 7:38 PM Weixi Li (BLOOMBERG/ PRINCETON) <
>>>>>>>>>>> wli...@bloomberg.net> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> What a great community! So many good tips in such a short time!
>>>>>>>>>>>>
>>>>>>>>>>>> Especially the atscppai, I would've never noticed it. the async
>>>>>>> examples
>>>>>>>>>>>> look very promising.
>>>>>>>>>>>>
>>>>>>>>>>>> It looks like the following might be necessary (let me know if I'm
>>>>>>>>>>> wrong):
>>>>>>>>>>>> * A hook to TS_HTTP_SEND_REQUEST_HDR_HOOK
>>>>>>>>>>>> * A map of queues (one queue per rate-limited site)
>>>>>>>>>>>> * A map of async timers (one timer per queue)
>>>>>>>>>>>>
>>>>>>>>>>>> I will study the ATS code more to understand the event and
>>>>> threading
>>>>>>>>>>> model
>>>>>>>>>>>> better.
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you all.
>>>>>>>>>>>>
>>>>>>>>>>>> From: dev@trafficserver.apache.org At: 06/21/19 19:52:44To:
>>>>>>>>>>>> dev@trafficserver.apache.org
>>>>>>>>>>>> Subject: Re: Implementing Rate-limiting in forward proxy mode
>>>>>>>>>>>>
>>>>>>>>>>>> I have implemented rate-limit in my plugin using atscppapi. We are
>>>>>>> using
>>>>>>>>>>>> ats in
>>>>>>>>>>>> security context for mitigation. If the request matches certain
>>>>>>> criteria
>>>>>>>>>>>> (ip,
>>>>>>>>>>>> method, host, uri and header values) then we apply rate-limit to
>>>>>>> that ip.
>>>>>>>>>>>>
>>>>>>>>>>>> Dk.
>>>>>>>>>>>>
>>>>>>>>>>>>> On Jun 21, 2019, at 3:15 PM, Leif Hedstrom <zw...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Jun 21, 2019, at 16:09, Weixi Li (BLOOMBERG/ PRINCETON)
>>>>>>>>>>>> <wli...@bloomberg.net> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi team,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We are experimenting with ATS in *forward* proxy mode. Our
>>>>> use-case
>>>>>>>>>>>> requires
>>>>>>>>>>>> a rate-limiting component that enforces rules based on the
>>>>>>> destination.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For all incoming requests targeting "www.cnn.com", we want to
>>>>>>> limit
>>>>>>>>>>>> the
>>>>>>>>>>>> outgoing rate to be 10 requests per minute; for "www.reddit.com",
>>>>> we
>>>>>>>>>>> want
>>>>>>>>>>>> the
>>>>>>>>>>>> rate to be 20 requests per minute; and so on. If there were more
>>>>>>> requests
>>>>>>>>>>>> than
>>>>>>>>>>>> the limit specified, the requests must be queued before they could
>>>>> go
>>>>>>>>>>> out.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Seems very straight forward to implement as a plugin. For example
>>>>>>> the
>>>>>>>>>>>> geo_acl
>>>>>>>>>>>> plugin might be a good start, since it limits access based on
>>>>> source
>>>>>>> IP.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Would be interesting to hear more about your use case too, it’s
>>>>>>> always
>>>>>>>>>>>> exciting to hear about different solutions that people use ATS for.
>>>>>>> Maybe
>>>>>>>>>>>> at
>>>>>>>>>>>> the next ATS summit? :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> — Leif
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is it possible to implement this requirement using a plugin?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If not, we wouldn't mind forking the code and modifying whichever
>>>>>>>>>>> parts
>>>>>>>>>>>> that
>>>>>>>>>>>> would be necessary. But which are the potentially relevant
>>>>>>> components?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If any experts could give us some pointers on the design, that
>>>>>>> would
>>>>>>>>>>> be
>>>>>>>>>>>> really appreciated.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Weixi
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>
>
