mcvsubbu commented on issue #7192: URL: https://github.com/apache/pinot/issues/7192#issuecomment-885897977
Repeating my message to @mapshen in the slack channel: Please re-run the realtime prov helper and paste the output in this issue. There is no standard for the time taken to build a segment. It is a function of how large your segment is (which includes number of rows, cardinality of columns, number of columns, length of string columns, the type and number of indices you have in there, etc.). In general, DO NOT use number of rows as the limit. It is better to use segment size as the limit. I hope you are doing that. If so, can you share what the size is, and can you reduce the size? It is also possible that all partitions on the machine are completing the segment at the same time. How many partitions do you have on any one machine? It is possible that your machine is (a) swapping or (b) GC-ing or both. What is your heap size? Also, what is your QPS? How much delay can you tolerate for incoming freshness of data? Do you know that Pinot provides a way to measure the freshness of data in your query response? There are knobs that you can tune before starting to propose protocol changes (that can cause unnecessary race conditions in the code and issues for other installations). What is your ingestion rate (per partition)? both in number of messages and number of bytes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
