npawar opened a new pull request #6322: URL: https://github.com/apache/incubator-pinot/pull/6322
https://github.com/apache/incubator-pinot/issues/6308 API to get status of consumption of a table. This includes 1. consumer status 2. partition to offsets map 3. server name, for consuming segments across all servers. **NOTE: Adding tests is pending.** API on the server returns status for all consumers found in the tableDataManager. ``` curl -i -X GET "http://localhost:8097/tables/transcript/consumingSegmentsInfo"; [{"segmentName":"transcript__1__2__20201204T2044Z","consumerState":"CONSUMING","partitionToOffset":{"1":"6"}}, {"segmentName":"transcript__0__14__20201204T1943Z","consumerState":"CONSUMING","partitionToOffset":{"0":"31"}}] ``` API on the controller calls the servers to get consumer info. It returns a map of segment name to consumption information for all servers ``` curl -i -X GET "http://localhost:9000/tables/transcript/consumingSegmentsInfo"; { "_segmentToConsumingInfoMap": { "transcript__0__14__20201204T1943Z": [ { "_serverName": "Server_10.1.10.1_8098", "_consumerState": "CONSUMING", "_partitionToOffset": {"0": "31"} } ], "transcript__1__2__20201204T2044Z": [ { "_serverName": "Server_10.1.10.1_8098", "_consumerState": "CONSUMING", "_partitionToOffset": {"1": "6"} } ] } } ``` Some manual testing: 1. Segment build failed. Consumer went into ERROR state. But IS, EV and segment metadata looks all good. ``` curl -i -X GET "http://localhost:9000/tables/transcript/consumingSegmentsInfo"; { "_segmentToConsumingInfoMap": { "transcript__0__14__20201204T1943Z": [ { "_serverName": "Server_10.1.10.51_8098", "_consumerState": "CONSUMING", "_partitionToOffset": {"0": "31"}} ], "transcript__1__1__20201204T1935Z": [ { "_serverName": "Server_10.1.10.51_8098", "_consumerState": "NOT_CONSUMING", "_partitionToOffset": {"1": "5"} } ] } } ``` 2. Exception in OFFLINE to CONSUMING state transition. Consumer never created. But IS and segment metadata looks all good. EV in ERROR state. ``` curl -i -X GET "http://localhost:9000/tables/transcript/consumingSegmentsInfo"; { "_segmentToConsumingInfoMap": { "transcript__0__14__20201204T1943Z": [ { "_serverName": "Server_10.1.10.51_8098", "_consumerState": "CONSUMING", "_partitionToOffset": {"0": "31"}} ], "transcript__1__1__20201204T1935Z": [ ] } } ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
