[
https://issues.apache.org/jira/browse/KNOX-2832?focusedWorklogId=823120&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823120
]
ASF GitHub Bot logged work on KNOX-2832:
----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Nov/22 15:44
Start Date: 03/Nov/22 15:44
Worklog Time Spent: 10m
Work Description: MrtnBalazs opened a new pull request, #668:
URL: https://github.com/apache/knox/pull/668
## What changes were proposed in this pull request?
The DoS security provider have been removed, and it's functionality have
been moved in the Web App Security provider.
This provider is now able to add a Jetty DoSFilter into to filterchain if
configured.
The configuration options are the same as jetty's DoSFilter and can be found
in jetty's DoSFilter documentation
(https://www.eclipse.org/jetty/documentation/jetty-9/index.html#dos-filter),
EXCEPT that the parameters need a `rate.limiting.` prefix AND when using
delayMs with 0 or higher value or using throttling the
`gateway.servlet.async.supported` configuration must be set to `true` in
`gateway-site.xml` configuration file (it defaults to false)!
Example configuration:
```
<provider>
<role>webappsec</role>
<name>WebAppSec</name>
<enabled>true</enabled>
<param>
<name>rate.limiting.enabled</name>
<value>true</value>
</param>
<param>
<name>rate.limiting.maxRequestsPerSec</name>
<value>2</value>
</param>
<param>
<name>rate.limiting.delayMs</name>
<value>2000</value>
</param>
<param>
<name>rate.limiting.maxWaitMs</name>
<value>20000</value>
</param>
<param>
<name>rate.limiting.throttledRequests</name>
<value>3</value>
</param>
<param>
<name>rate.limiting.throttleMs</name>
<value>20000</value>
</param>
```
## How was this patch tested?
I have written tests into the `WebAppSecContributorTest` class.
These tests test whether the contributor sets the right filters and
parameters or not.
I have also tested the feature manually by sending 10 curl requests to a
server which response time is 3 seconds and enabled DEBUG level logs for
`org.eclipse.jetty.servlets`.
When the feature is disabled:
```
<param>
<name>rate.limiting.enabled</name>
<value>false</value>
</param>
```
There is no effect:
```
2022-11-03 09:58:24,806 2ca3f959-da98-414a-81a3-fe8c9d053ba6 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:58:24,832 93b6bcd0-5af3-4f77-b27a-324cc9e052c6 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:58:24,842 4651eabd-df67-48a2-b3f1-27ab32d0dd5a INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:58:24,859 cb2e9b3b-2d78-4edc-b432-bdc145d52dff INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:58:24,879 c3ee6514-3618-4974-9362-594159216c69 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:58:24,892 cc1f31f4-046b-466e-b4a4-e057c4d06f00 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:58:24,997 a2ff2f2a-5b13-4ded-ae4b-8e959cd94ebd INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:58:25,014 85957a67-3d85-4087-beee-c9d1324efa32 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:58:25,030 0581df96-feab-4221-8050-ae59e6b431ef INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:58:25,049 0fabb544-1cbc-4413-8b99-e19d34ce516e INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
```
When it is enabled with a normal configuration:
```
<provider>
<role>webappsec</role>
<name>WebAppSec</name>
<enabled>true</enabled>
<param>
<name>rate.limiting.enabled</name>
<value>true</value>
</param>
<param>
<name>rate.limiting.maxRequestsPerSec</name>
<value>2</value>
</param>
<param>
<name>rate.limiting.delayMs</name>
<value>2000</value>
</param>
<param>
<name>rate.limiting.maxWaitMs</name>
<value>20000</value>
</param>
<param>
<name>rate.limiting.throttledRequests</name>
<value>3</value>
</param>
<param>
<name>rate.limiting.throttleMs</name>
<value>20000</value>
</param>
<param>
<name>rate.limiting.maxRequestMs</name>
<value>30000</value>
</param>
<param>
<name>rate.limiting.maxIdleTrackerMs</name>
<value>30000</value>
</param>
<param>
<name>rate.limiting.insertHeaders</name>
<value>true</value>
</param>
<param>
<name>rate.limiting.trackSessions</name>
<value>false</value>
</param>
<param>
<name>rate.limiting.remotePort</name>
<value>false</value>
</param>
<param>
<name>rate.limiting.ipWhitelist</name>
<value></value>
</param>
<param>
<name>rate.limiting.managedAttr</name>
<value>false</value>
</param>
</provider>
```
The requests are delayed and throttled:
```
2022-11-03 09:31:30,811 08b8316a-4190-4f16-8464-cb01edc834c7 DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(323)) - Filtering
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@6b2e4211
2022-11-03 09:31:30,811 08b8316a-4190-4f16-8464-cb01edc834c7 DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(335)) - Allowing
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@6b2e4211
2022-11-03 09:31:30,812 08b8316a-4190-4f16-8464-cb01edc834c7 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:31:30,820 d7943185-159b-450c-bb6c-8ab72c495cfc DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(323)) - Filtering
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@39eab2ac
2022-11-03 09:31:30,821 d7943185-159b-450c-bb6c-8ab72c495cfc DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(335)) - Allowing
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@39eab2ac
2022-11-03 09:31:30,821 d7943185-159b-450c-bb6c-8ab72c495cfc INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:31:30,830 44f679f8-941a-4c2b-b8fe-e7516dabcd8f DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(323)) - Filtering
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@6d02dfae
2022-11-03 09:31:30,830 44f679f8-941a-4c2b-b8fe-e7516dabcd8f WARN
servlets.DoSFilter (DoSFilter.java:onRequestOverLimit(1544)) - DOS ALERT:
Request delayed=2000ms, ip=127.0.0.1, overlimit=OverLimit@208a8383[type=IP,
id=127.0.0.1, duration=PT0.019S, count=2], session=null, user=null
2022-11-03 09:31:30,849 52bc8732-690a-4ec8-9daf-adfdfa199bd8 DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(323)) - Filtering
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@117d64af
2022-11-03 09:31:30,850 52bc8732-690a-4ec8-9daf-adfdfa199bd8 WARN
servlets.DoSFilter (DoSFilter.java:onRequestOverLimit(1544)) - DOS ALERT:
Request delayed=2000ms, ip=127.0.0.1, overlimit=OverLimit@72ccc9f[type=IP,
id=127.0.0.1, duration=PT0.03S, count=2], session=null, user=null
2022-11-03 09:31:30,872 16e1b7ec-9d02-4a18-b972-340402228a72 DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(323)) - Filtering
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@24ea7668
2022-11-03 09:31:30,872 16e1b7ec-9d02-4a18-b972-340402228a72 WARN
servlets.DoSFilter (DoSFilter.java:onRequestOverLimit(1544)) - DOS ALERT:
Request delayed=2000ms, ip=127.0.0.1, overlimit=OverLimit@17dcce0f[type=IP,
id=127.0.0.1, duration=PT0.042S, count=2], session=null, user=null
2022-11-03 09:31:30,877 44ef7557-d84f-466e-aeb6-b20db8c2d4d8 DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(323)) - Filtering
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@6c588cf5
2022-11-03 09:31:30,878 44ef7557-d84f-466e-aeb6-b20db8c2d4d8 WARN
servlets.DoSFilter (DoSFilter.java:onRequestOverLimit(1544)) - DOS ALERT:
Request delayed=2000ms, ip=127.0.0.1, overlimit=OverLimit@183a9c66[type=IP,
id=127.0.0.1, duration=PT0.027S, count=2], session=null, user=null
2022-11-03 09:31:30,980 4b3ba59d-fa0a-43b0-a783-746c52253dc8 DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(323)) - Filtering
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@472c85bb
2022-11-03 09:31:30,981 4b3ba59d-fa0a-43b0-a783-746c52253dc8 WARN
servlets.DoSFilter (DoSFilter.java:onRequestOverLimit(1544)) - DOS ALERT:
Request delayed=2000ms, ip=127.0.0.1, overlimit=OverLimit@3c76baca[type=IP,
id=127.0.0.1, duration=PT0.108S, count=2], session=null, user=null
2022-11-03 09:31:30,998 f99b7eba-b439-47a2-bec6-14ae8b2b9205 DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(323)) - Filtering
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@40197a1
2022-11-03 09:31:30,998 f99b7eba-b439-47a2-bec6-14ae8b2b9205 WARN
servlets.DoSFilter (DoSFilter.java:onRequestOverLimit(1544)) - DOS ALERT:
Request delayed=2000ms, ip=127.0.0.1, overlimit=OverLimit@28327311[type=IP,
id=127.0.0.1, duration=PT0.121S, count=2], session=null, user=null
2022-11-03 09:31:31,016 4fbf69d5-7f4b-45ec-b445-ee37ac33be17 DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(323)) - Filtering
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@3c93326f
2022-11-03 09:31:31,016 4fbf69d5-7f4b-45ec-b445-ee37ac33be17 WARN
servlets.DoSFilter (DoSFilter.java:onRequestOverLimit(1544)) - DOS ALERT:
Request delayed=2000ms, ip=127.0.0.1, overlimit=OverLimit@63636127[type=IP,
id=127.0.0.1, duration=PT0.036S, count=2], session=null, user=null
2022-11-03 09:31:31,027 d8f34a59-2cd2-40dd-9914-d15a62642a31 DEBUG
servlets.DoSFilter (DoSFilter.java:doFilter(323)) - Filtering
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@7790a707
2022-11-03 09:31:31,027 d8f34a59-2cd2-40dd-9914-d15a62642a31 WARN
servlets.DoSFilter (DoSFilter.java:onRequestOverLimit(1544)) - DOS ALERT:
Request delayed=2000ms, ip=127.0.0.1, overlimit=OverLimit@2f8095fa[type=IP,
id=127.0.0.1, duration=PT0.029S, count=2], session=null, user=null
2022-11-03 09:31:32,838 89e4d19e-b60e-404a-9aa7-6c394d8ff6b6 DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(385)) - Throttling
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@3037fda7
2022-11-03 09:31:32,838 89e4d19e-b60e-404a-9aa7-6c394d8ff6b6 DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(429)) - Allowing
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@3037fda7
2022-11-03 09:31:32,839 89e4d19e-b60e-404a-9aa7-6c394d8ff6b6 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:31:32,856 29e0b9f8-586f-44b9-a7a5-b24822f55a7e DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(385)) - Throttling
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@538b9ae0
2022-11-03 09:31:32,856 29e0b9f8-586f-44b9-a7a5-b24822f55a7e DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(429)) - Allowing
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@538b9ae0
2022-11-03 09:31:32,856 29e0b9f8-586f-44b9-a7a5-b24822f55a7e INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:31:32,875 a8fc1224-8ad2-417b-9643-ebebffd65aaa DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(385)) - Throttling
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@244f6e4d
2022-11-03 09:31:32,875 a8fc1224-8ad2-417b-9643-ebebffd65aaa DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(429)) - Allowing
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@244f6e4d
2022-11-03 09:31:32,876 a8fc1224-8ad2-417b-9643-ebebffd65aaa INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:31:32,880 f44a7823-8e7f-48bd-b185-c6577e26c264 DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(385)) - Throttling
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@1d925046
2022-11-03 09:31:32,986 98e36ba4-8ba8-4056-9754-31ca0e9f382f DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(385)) - Throttling
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@3b9899be
2022-11-03 09:31:33,002 53fa94f9-f606-4647-b3aa-1a698f0ef0c1 DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(385)) - Throttling
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@480299bd
2022-11-03 09:31:33,019 c479ac57-8c39-41cf-bd22-e467d6cf05d0 DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(385)) - Throttling
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@62b847f4
2022-11-03 09:31:33,031 e4821957-0bec-42d9-aa47-ccd8b81a18e3 DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(385)) - Throttling
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@f7b5325
2022-11-03 09:31:35,853 f44a7823-8e7f-48bd-b185-c6577e26c264 DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(429)) - Allowing
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@1d925046
2022-11-03 09:31:35,853 f44a7823-8e7f-48bd-b185-c6577e26c264 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:31:35,868 98e36ba4-8ba8-4056-9754-31ca0e9f382f DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(429)) - Allowing
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@3b9899be
2022-11-03 09:31:35,869 98e36ba4-8ba8-4056-9754-31ca0e9f382f INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:31:35,892 53fa94f9-f606-4647-b3aa-1a698f0ef0c1 DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(429)) - Allowing
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@480299bd
2022-11-03 09:31:35,893 53fa94f9-f606-4647-b3aa-1a698f0ef0c1 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:31:38,868 c479ac57-8c39-41cf-bd22-e467d6cf05d0 DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(429)) - Allowing
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@62b847f4
2022-11-03 09:31:38,869 c479ac57-8c39-41cf-bd22-e467d6cf05d0 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:31:38,879 e4821957-0bec-42d9-aa47-ccd8b81a18e3 DEBUG
servlets.DoSFilter (DoSFilter.java:throttleRequest(429)) - Allowing
org.apache.knox.gateway.filter.XForwardedHeaderRequestWrapper@f7b5325
2022-11-03 09:31:38,880 e4821957-0bec-42d9-aa47-ccd8b81a18e3 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=tom,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: tom
2022-11-03 09:32:00,813 DEBUG servlets.DoSFilter
(DoSFilter.java:removeFromRateTrackers(1308)) - Tracker removed: 127.0.0.1
```
Issue Time Tracking
-------------------
Worklog Id: (was: 823120)
Remaining Estimate: 0h
Time Spent: 10m
> Convert JettyDOS provider to a rate limiting option in webappsec
> ----------------------------------------------------------------
>
> Key: KNOX-2832
> URL: https://issues.apache.org/jira/browse/KNOX-2832
> Project: Apache Knox
> Issue Type: Task
> Components: Server
> Affects Versions: 2.0.0
> Reporter: Sandor Molnar
> Assignee: Balazs Marton
> Priority: Critical
> Fix For: 2.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It'd be better to remove the recently created {{JettyDos}} security provider
> and have it functionality wired into Knox's existing {{webappsec}} provider
> just like other security elements (e.g. XFrame, CORS, XSS,...)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
