[jira] [Commented] (KAFKA-2171) System Test for Quotas
[ https://issues.apache.org/jira/browse/KAFKA-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571464#comment-14571464 ] Dong Lin commented on KAFKA-2171: - Updated reviewboard https://reviews.apache.org/r/34742/diff/ against branch origin/trunk System Test for Quotas -- Key: KAFKA-2171 URL: https://issues.apache.org/jira/browse/KAFKA-2171 Project: Kafka Issue Type: Sub-task Reporter: Dong Lin Assignee: Dong Lin Attachments: KAFKA-2171.patch, KAFKA-2171.patch Original Estimate: 336h Remaining Estimate: 336h Initial setup and configuration: In all scenarios, we create the following entities and topic: - 1 kafka brokers - 1 topic with replication factor = 1, ackNum = -1, and parition = 6 - 1 producer performance - 2 console consumers - 3 jmx tools, one for each of the producer and consumers - we consider two rates are approximately the same if they differ by at most 10%. Scenario 1: validate the effectiveness of default producer and consumer quota 1) Let default_producer_quota = default_consumer_quota = 2 Bytes/sec 2) Produce 2000 messages of 2 bytes each (with clientId = producer_performance) 3) Wait until producer stops 4) Two consumers consume from the topic (with clientId = group1 and group2 respectively ) 5) verify that actual rate is within 10% of expected rate (quota) Scenario 2: validate the effectiveness of producer and consumer quota override 1) Let default_producer_quota = default_consumer_quota = 2 Bytes/sec Override quota of producer_performance and group1 to be 15000 Bytes/sec 2) Produce 2000 messages of 2 bytes each (with clientId = producer_performance) 3) Wait until producer stops 4) Two consumers consume from the topic (with clientId = group1 and group2 respectively ) 5) verify that actual rate is within 10% of expected rate (quota) Scenario 3: validate the effectiveness of quota sharing 1) Let default_producer_quota = default_consumer_quota = 2 Bytes/sec Override quota of producer_performance and group1 to be 15000 Bytes/sec 2) Produce 2000 messages of 2 bytes each (with clientId = producer_performance) 3) Wait until producer stops 4) Two consumers consume from the topic (with clientId = group1 for both consumers) 5) verify that actual rate is within 10% of expected rate (quota) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2171) System Test for Quotas
[ https://issues.apache.org/jira/browse/KAFKA-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562070#comment-14562070 ] Dong Lin commented on KAFKA-2171: - Created reviewboard https://reviews.apache.org/r/34742/diff/ against branch origin/trunk System Test for Quotas -- Key: KAFKA-2171 URL: https://issues.apache.org/jira/browse/KAFKA-2171 Project: Kafka Issue Type: Sub-task Reporter: Dong Lin Assignee: Dong Lin Attachments: KAFKA-2171.patch Original Estimate: 336h Remaining Estimate: 336h Motivation and goal: We want to make sure that following features are working properly for both consumer and producer: default quota, client-specific quota, and quota sharing per clientId. The tests and configuration described aims to cover most of the scenarios. More test cases with varying configurations (e.g. ackNum) can be added if there is good reason to do so. Initial setup and configuration: In all scenarios, we first create kafka brokers and topic as follows: - create two kafka broker processes (by default local) - create a topic with replication factor = 2 and ackNum = -1 - let max_read = max_write = 5MB. The test machine is expected to provide read (write) throughput at least max_read (max_write). - we consider two rates are approximately the same if they differ by at most 5%. Scenario 1: Validate that max_read and max_write are provided by the test machine(s) using 1 producer and 1 consumer 1) produce data to the topic without rate limit for 30 seconds 2) record the rate of producer 3) then, consume data from the topic without rate limit until he finishes 4) record the rate of consumer 5) verify that the data consumed is identical to the data produced 6) verify that producer rate = max_write and consumer rate = max_read Scenario 2: validate the effectiveness of default write and read quota using 1 producer and 1 consumer 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota 2) produce data to the topic for 30 seconds 3) record the rate of producer 4) then, consume data from the topic until he finishes 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that recorded write (read) rate is within 5% of max_write/2 (max_read/2). Scenario 3: validate the effectiveness of client-specific write and read quota using 2 producers and 2 consumers 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota; configure brokers to use max_write/4 for producer_2 and max_read/4 for consumer_2 2) both producers produce data to the topic for 30 seconds. They use different clientId. 3) record the rate of producer 4) both consumers consume data from the topic until they finish. They use different clientId and groupId. 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that producer_1 and producer_2 rates are approximately max_write/2 and max_write/4; verify that consumer_1 and consumer_2 rates are approximately max_read/2 and max_read/4. Scenario 4: validate the effectiveness of write and read quota sharing among clients of same clientId using 2 producers and 2 consumers. 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota 2) both producers produce data to the topic for 30 seconds. They use same clientId. 3) record the rate of producer 4) both consumers consume data from the topic until they finish. They use same clientId but different groupId. 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that total rate of producer_1 and producer_2 is approximately max_write/2; verify that total rate of consumer_1 and consumer_2 is approximately max_read/2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2171) System Test for Quotas
[ https://issues.apache.org/jira/browse/KAFKA-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541116#comment-14541116 ] Geoffrey Anderson commented on KAFKA-2171: -- Just pinging this thread to make sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake System Test for Quotas -- Key: KAFKA-2171 URL: https://issues.apache.org/jira/browse/KAFKA-2171 Project: Kafka Issue Type: Sub-task Reporter: Dong Lin Assignee: Dong Lin Original Estimate: 336h Remaining Estimate: 336h Motivation and goal: We want to make sure that following features are working properly for both consumer and producer: default quota, client-specific quota, and quota sharing per clientId. The tests and configuration described aims to cover most of the scenarios. More test cases with varying configurations (e.g. ackNum) can be added if there is good reason to do so. Initial setup and configuration: In all scenarios, we first create kafka brokers and topic as follows: - create two kafka broker processes (by default local) - create a topic with replication factor = 2 and ackNum = -1 - let max_read = max_write = 5MB. The test machine is expected to provide read (write) throughput at least max_read (max_write). - we consider two rates are approximately the same if they differ by at most 5%. Scenario 1: Validate that max_read and max_write are provided by the test machine(s) using 1 producer and 1 consumer 1) produce data to the topic without rate limit for 30 seconds 2) record the rate of producer 3) then, consume data from the topic without rate limit until he finishes 4) record the rate of consumer 5) verify that the data consumed is identical to the data produced 6) verify that producer rate = max_write and consumer rate = max_read Scenario 2: validate the effectiveness of default write and read quota using 1 producer and 1 consumer 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota 2) produce data to the topic for 30 seconds 3) record the rate of producer 4) then, consume data from the topic until he finishes 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that recorded write (read) rate is within 5% of max_write/2 (max_read/2). Scenario 3: validate the effectiveness of client-specific write and read quota using 2 producers and 2 consumers 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota; configure brokers to use max_write/4 for producer_2 and max_read/4 for consumer_2 2) both producers produce data to the topic for 30 seconds. They use different clientId. 3) record the rate of producer 4) both consumers consume data from the topic until they finish. They use different clientId and groupId. 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that producer_1 and producer_2 rates are approximately max_write/2 and max_write/4; verify that consumer_1 and consumer_2 rates are approximately max_read/2 and max_read/4. Scenario 4: validate the effectiveness of write and read quota sharing among clients of same clientId using 2 producers and 2 consumers. 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota 2) both producers produce data to the topic for 30 seconds. They use same clientId. 3) record the rate of producer 4) both consumers consume data from the topic until they finish. They use same clientId but different groupId. 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that total rate of producer_1 and producer_2 is approximately max_write/2; verify that total rate of consumer_1 and consumer_2 is approximately max_read/2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
