mcvsubbu opened a new pull request #5737:
URL: https://github.com/apache/incubator-pinot/pull/5737
- Fixed a bug where we were looking for push frequency setting in the table
config
Push frequency is not usually set in realtime table, so changed that to be
an
optional argument instead.
- Instead of push frequency, the user may specify retentionHours, meaning
the number
of hours of most recent data that will be queried most often.
- Added the amount of total mapped memory to the output
- Added the number of segments queried per host to the output.
- Added a link to the documentation to the output.
- Instead of accepting the time taken to consume the segment, introduced an
argument
to accept the ingestion rate per partition. This is determined easier when
looking
at metrics for the topic that we need to ingest, and is more convenient
when
an offline segment is being provided as a sample
- Added the command line arguments printed with the output so that it is
easy for us
to debug when we get questions from the community.
TODO Next:
Can we compute a score and just recommend a configuration? We know we want
to minimize
num segments scanned per host, num hosts, segment size (to fit memory).
Issue #5588
Here is a sample output:
============================================================
RealtimeProvisioningHelperCommand -tableConfigFile
/Users/ssubrama/tmp/samza/realtimeTableConfig.json -numPartitions 16
-pushFrequency null -numHosts 8,6,10 -numHours 6,12,18,24
-sampleCompletedSegmentDir
/Users/ssubrama/tmp/samza/TestSamzaAnalyticsFeatures_1593411480000_1593500340000_0/
-ingestionRate 100 -maxUsableHostMemory 48G -retentionHours 72
Note:
* Table retention and push frequency ignored for determining retentionHours
* See https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime
Memory used per host (Active/Mapped)
numHosts --> 6 |8 |10 |
numHours
6 --------> 5.05G/19.49G |3.37G/12.99G |3.37G/12.99G |
12 --------> 5.89G/20.33G |3.93G/13.55G |3.93G/13.55G |
18 --------> 6.73G/21.49G |4.48G/14.33G |4.48G/14.33G |
24 --------> 7.56G/22G |5.04G/14.66G |5.04G/14.66G |
Optimal segment size
numHosts --> 6 |8 |10 |
numHours
6 --------> 111.98M |111.98M |111.98M |
12 --------> 223.96M |223.96M |223.96M |
18 --------> 335.94M |335.94M |335.94M |
24 --------> 447.92M |447.92M |447.92M |
Consuming memory
numHosts --> 6 |8 |10 |
numHours
6 --------> 1.45G |987.17M |987.17M |
12 --------> 2.61G |1.74G |1.74G |
18 --------> 3.77G |2.52G |2.52G |
24 --------> 4.94G |3.29G |3.29G |
Number of segments queried per host
numHosts --> 6 |8 |10 |
numHours
6 --------> 12 |12 |12 |
12 --------> 6 |6 |6 |
18 --------> 4 |4 |4 |
24 --------> 3 |3 |3 |
## Description
Add a description of your PR here.
A good description should include pointers to an issue or design document,
etc.
## Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order:
Controller, Broker, Server, Minion)
* [ ] Yes (Please label as **<code>backward-incompat</code>**, and complete
the section below on Release Notes)
Does this PR fix a zero-downtime upgrade introduced earlier?
* [ ] Yes (Please label this as **<code>backward-incompat</code>**, and
complete the section below on Release Notes)
Does this PR otherwise need attention when creating release notes? Things to
consider:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
* [ ] Yes (Please label this PR as **<code>release-notes</code>** and
complete the section on Release Notes)
## Release Notes
If you have tagged this as either backward-incompat or release-notes,
you MUST add text here that you would like to see appear in release notes of
the
next release.
If you have a series of commits adding or enabling a feature, then
add this section only in final commit that marks the feature completed.
Refer to earlier release notes to see examples of text
## Documentation
If you have introduced a new feature or configuration, please add it to the
documentation as well.
See
https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
