Dong Lin created KAFKA-2171:
-------------------------------
Summary: System Test for Quotas
Key: KAFKA-2171
URL: https://issues.apache.org/jira/browse/KAFKA-2171
Project: Kafka
Issue Type: Sub-task
Reporter: Dong Lin
Assignee: Dong Lin
Motivation and goal:
We want to make sure that following features are working properly for both
consumer and producer: default quota, client-specific quota, and quota sharing
per clientId.
The tests and configuration described aims to cover most of the scenarios. More
test cases with varying configurations (e.g. ackNum) can be added if there is
good reason to do so.
Initial setup and configuration:
In all scenarios, we first create kafka brokers and topic as follows:
- create two kafka broker processes (by default local)
- create a topic with replication factor = 2 and ackNum = -1
- let max_read = max_write = 5MB. The test machine is expected to provide read
(write) throughput at least max_read (max_write).
- we consider two rates are approximately the same if they differ by at most 5%.
Scenario 1: Validate that max_read and max_write are provided by the test
machine(s) using 1 producer and 1 consumer
1) produce data to the topic without rate limit for 30 seconds
2) record the rate of producer
3) then, consume data from the topic without rate limit until he finishes
4) record the rate of consumer
5) verify that the data consumed is identical to the data produced
6) verify that producer rate >= max_write and consumer rate >= max_read
Scenario 2: validate the effectiveness of default write and read quota using 1
producer and 1 consumer
1) configure brokers to use max_write/2 as default write quota and max_read/2
as default read quota
2) produce data to the topic for 30 seconds
3) record the rate of producer
4) then, consume data from the topic until he finishes
5) record the rate of consumer
6) verify that the data consumed is identical to the data produced
7) verify that recorded write (read) rate is within 5% of max_write/2
(max_read/2).
Scenario 3: validate the effectiveness of client-specific write and read quota
using 2 producers and 2 consumers
1) configure brokers to use max_write/2 as default write quota and max_read/2
as default read quota; configure brokers to use max_write/4 for producer_2 and
max_read/4 for consumer_2
2) both producers produce data to the topic for 30 seconds. They use different
clientId.
3) record the rate of producer
4) both consumers consume data from the topic until they finish. They use
different clientId and groupId.
5) record the rate of consumer
6) verify that the data consumed is identical to the data produced
7) verify that producer_1 and producer_2 rates are approximately max_write/2
and max_write/4; verify that consumer_1 and consumer_2 rates are approximately
max_read/2 and max_read/4.
Scenario 4: validate the effectiveness of write and read quota sharing among
clients of same clientId using 2 producers and 2 consumers.
1) configure brokers to use max_write/2 as default write quota and max_read/2
as default read quota
2) both producers produce data to the topic for 30 seconds. They use same
clientId.
3) record the rate of producer
4) both consumers consume data from the topic until they finish. They use same
clientId but different groupId.
5) record the rate of consumer
6) verify that the data consumed is identical to the data produced
7) verify that total rate of producer_1 and producer_2 is approximately
max_write/2; verify that total rate of consumer_1 and consumer_2 is
approximately max_read/2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
