jackjlli opened a new pull request #5724: URL: https://github.com/apache/incubator-pinot/pull/5724
## Description This PR adds logic to enable/disable query quotas per Pinot broker. This is helpful to test out the capacity of a given broker without affecting other brokers in production. Higher load can be routed to this dedicated host to find out the capacity, which can give us better insights on the current provisioning of the cluster. LinkedIn currently has one service that has been testing in this way. The query quotas are all enabled/disabled together within a broker. Once the broker gets restarted, query quotas would all be re-enabled again whatever the previous states were before. ## Tests done: Started all Pinot components. Broker log: ``` 2020/07/21 11:12:24.296 INFO [HelixExternalViewBasedQueryQuotaManager] [HelixTaskExecutor-message_handle_thread] Initializing rate limiter for table internalTesting_OFFLINE 2020/07/21 11:12:24.296 INFO [HelixExternalViewBasedQueryQuotaManager] [HelixTaskExecutor-message_handle_thread] The number of online brokers for table internalTesting_OFFLINE is 1 2020/07/21 11:12:24.300 INFO [HelixExternalViewBasedQueryQuotaManager] [HelixTaskExecutor-message_handle_thread] Rate limiter for table: internalTesting_OFFLINE has been initialized. Overall rate: 0.01. P er-broker rate: 0.01. Number of online broker instances: 1. Table config stat version: 0 ... 2020/07/21 11:13:22.291 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-0] requestId=1,table=internalTesting_OFFLINE,timeMs=594,docs=10/970,entries=0/240,segments(queried/processed/matched/consuming/unavailable):1/1/1/0/0,consumingFreshnessTimeMs=0,servers=1/1,groupLimitReached=false,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);172.18.180.84_O=95,144,2756,4,query=select * from internalTesting limit 10 2020/07/21 11:13:32.779 INFO [HelixExternalViewBasedQueryQuotaManager] [jersey-server-managed-async-executor-1] Quota is exceeded for table: internalTesting_OFFLINE. Per-broker rate: 0.01. Current qps: 1 2020/07/21 11:13:32.782 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-1] Request 2 exceeds query quota for table:internalTesting, query:select * from internalTesting limit 50 2020/07/21 11:13:41.725 INFO [HelixExternalViewBasedQueryQuotaManager] [jersey-server-managed-async-executor-2] Quota is exceeded for table: internalTesting_OFFLINE. Per-broker rate: 0.01. Current qps: 1 2020/07/21 11:13:41.725 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-2] Request 3 exceeds query quota for table:internalTesting, query:select * from internalTesting limit 30 ``` Make the API call to disable query quotas: ``` 2020/07/21 11:14:01.137 INFO [HelixExternalViewBasedQueryQuotaManager] [HelixTaskExecutor-message_handle_thread] Toggle query quota state to: DISABLE 2020/07/21 11:14:01.137 INFO [HelixExternalViewBasedQueryQuotaManager] [HelixTaskExecutor-message_handle_thread] Finished toggling query quota state to: DISABLE for 1 tables in the current broker. 2020/07/21 11:14:19.332 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-3] requestId=4,table=internalTesting_OFFLINE,timeMs=13,docs=20/970,entries=0/480,segments(queried/processed/matched/consuming/unavailable):1/1/1/0/0,consumingFreshnessTimeMs=0,servers=1/1,groupLimitReached=false,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);172.18.180.84_O=0,8,4515,0,query=select * from internalTesting limit 20 2020/07/21 11:14:22.999 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-4] requestId=5,table=internalTesting_OFFLINE,timeMs=10,docs=40/970,entries=0/960,segments(queried/processed/matched/consuming/unavailable):1/1/1/0/0,consumingFreshnessTimeMs=0,servers=1/1,groupLimitReached=false,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);172.18.180.84_O=0,6,8035,0,query=select * from internalTesting limit 40 2020/07/21 11:14:26.120 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-5] requestId=6,table=internalTesting_OFFLINE,timeMs=18,docs=60/970,entries=0/1440,segments(queried/processed/matched/consuming/unavailable):1/1/1/0/0,consumingFreshnessTimeMs=0,servers=1/1,groupLimitReached=false,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);172.18.180.84_O=0,11,11556,0,query=select * from internalTesting limit 60 ``` Re-enable query quotas: ``` 2020/07/21 11:14:36.406 INFO [HelixExternalViewBasedQueryQuotaManager] [HelixTaskExecutor-message_handle_thread] Toggle query quota state to: ENABLE 2020/07/21 11:14:36.406 INFO [HelixExternalViewBasedQueryQuotaManager] [HelixTaskExecutor-message_handle_thread] Finished toggling query quota state to: ENABLE for 1 tables in the current broker. 2020/07/21 11:14:50.054 INFO [HelixExternalViewBasedQueryQuotaManager] [jersey-server-managed-async-executor-6] Quota is exceeded for table: internalTesting_OFFLINE. Per-broker rate: 0.01. Current qps: 1 2020/07/21 11:14:50.054 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-6] Request 7 exceeds query quota for table:internalTesting, query:select * from internalTesting limit 70 2020/07/21 11:14:53.956 INFO [HelixExternalViewBasedQueryQuotaManager] [jersey-server-managed-async-executor-7] Quota is exceeded for table: internalTesting_OFFLINE. Per-broker rate: 0.01. Current qps: 1 2020/07/21 11:14:53.956 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-7] Request 8 exceeds query quota for table:internalTesting, query:select * from internalTesting limit 80 ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
