This is an automated email from the ASF dual-hosted git repository.
jlli pushed a change to branch add-logic-for-lead-controller-resource
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
omit dfb1bd4 Check resource config is enabled from server
omit dae59fe Create a customized MasterSlaveStateModelFactory instead of
inherating one from Helix
omit f2011d1 Address PR comments
omit 266d123 Put HelixManager to constructor
omit 3831dff Address PR comments
omit 59f4fe3 Use resource config to check whether resource is enabled
omit 1d1f261 Address PR comments
omit 23f42fb Move murmur2 functions to an util class in pinot-common
omit 254e29d Move murmur2 functions to an util class in pinot-common
omit 39accc7 Address PR comments; Remove API design out of this PR
omit 57d7f56 Address PR comments
omit 2a85f92 Add logic for lead controller resource on controller side
omit 1aceba0 Add logic for leveraging lead controller resource
add 90e55d3 [TE] Check empty address to fix Email Exception (#4486)
add 7fde375 [TE] Adding Mean Variance Rule Detector (#4466)
add 6cf8c60 Clean up tenant config validation to correctly handle tag
override (#4488)
add 4fea0d4 An auto recommendation for inverted index (#4474)
add eac11ec Disable state transition stats verification in rebalancer
cluster integration test (#4490)
add bae8c1f [Instance Assignment] Add configs for instance assignment
(#4489)
add 8c15047 Adding kafka 2.0 doc for using simple consumer (#4493)
add c88845a [TE] frontend - harleyjj/jsyaml - add jsyaml package to parse
valid yaml as secondary to yamljs (#4480)
add 4f66513 [TE] Flatten Anomaly Results to a list of Map (#4491)
add cd75e12 [TE] Remove the offline suffix for offline pinot table.
(#4492)
add 6c9486e Add a thread in ControllerLeadershipManager to periodically
fetch controller leadership info (#4497)
add ac1f387 Rename metric to reflect what's being counted:
segments-processed vs segments-queried (#4499)
add 2bbdfab [TE] Remove deprecated stage classes and tests (#4496)
add b73427d [TE] [Composite-Alert] Adding Entity GroupKey with Whitelist
Template and Formatter (#4495)
add d1180d9 Down-grade Helix to version 0.8.4 (#4502)
add aea8fca [TE] anomalies table endpoint (#4501)
add 8e02f48 [Instance Assignment] Add implementations for instance
assignment (#4504)
add 6741396 Fix the toggle table state API (#4509)
add af75788 Adding support for Map type fields (#4388)
add fa615df Put Helix manager out of resource manager (#4506)
add a174d81 [TE] Update AnomalyFlattenResource (#4505)
add 61b2a07 [TE] frontend - harleyjj/aiavailability - implements AI
availability table UI (#4510)
add c725316 Add logic for leveraging lead controller resource
add ef3b814 Add logic for lead controller resource on controller side
add 8b33f94 Address PR comments
add c9136c2 Address PR comments; Remove API design out of this PR
add 4272946 Move murmur2 functions to an util class in pinot-common
add 96d1788 Move murmur2 functions to an util class in pinot-common
add 8a2e730 Address PR comments
add e136f2d Use resource config to check whether resource is enabled
add 50a91cd Address PR comments
add fa57488 Put HelixManager to constructor
add fc189aa Address PR comments
add 51b2aac Create a customized MasterSlaveStateModelFactory instead of
inherating one from Helix
add 1390d8e Check resource config is enabled from server
add 4de7b2a Rebase master
add 16bddbb Address PR comments
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (dfb1bd4)
\
N -- N -- N refs/heads/add-logic-for-lead-controller-resource
(16bddbb)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
docs/pluggable_streams.rst | 51 +-
.../common/config/SegmentPartitionConfig.java | 2 +-
.../apache/pinot/common/config/TableConfig.java | 109 ++++-
.../apache/pinot/common/config/TagNameUtils.java | 25 +-
.../config/instance/InstanceAssignmentConfig.java | 75 +++
...InstanceAssignmentConfigMapChildKeyHandler.java | 22 +-
.../instance/InstanceAssignmentConfigUtils.java | 93 ++++
.../config/instance/InstanceConstraintConfig.java | 38 +-
.../instance/InstanceReplicaPartitionConfig.java | 113 +++++
.../config/instance/InstanceTagPoolConfig.java | 86 ++++
.../pinot/common/metrics/ControllerMeter.java | 1 +
.../apache/pinot/common/utils/CommonConstants.java | 10 +-
.../org/apache/pinot/common/utils/DataTable.java | 2 +-
.../org/apache/pinot/common/utils/JsonUtils.java | 30 +-
.../common/utils/helix/LeadControllerUtils.java | 74 +--
.../pinot/common/config/TableConfigTest.java | 69 ++-
.../apache/pinot/controller/ControllerStarter.java | 78 ++-
.../pinot/controller/LeadControllerManager.java | 35 +-
.../pinot/controller/api/pojos/Instance.java | 2 +
.../resources/PinotTableConfigRestletResource.java | 35 +-
.../api/resources/PinotTableRestletResource.java | 91 ++--
.../helix/core/PinotHelixResourceManager.java | 282 ++++-------
.../core/assignment/InstancePartitionsUtils.java | 25 +-
.../HashBasedRotateInstanceConstraintApplier.java | 60 +++
.../instance/InstanceAssignmentDriver.java | 85 ++++
.../instance/InstanceConstraintApplier.java | 25 +-
.../instance/InstanceReplicaPartitionSelector.java | 189 ++++++++
.../instance/InstanceTagPoolSelector.java | 133 ++++++
...ltimeReplicaGroupSegmentAssignmentStrategy.java | 4 +-
.../segment/SegmentAssignmentStrategyFactory.java | 7 +-
.../helix/core/util/HelixSetupUtils.java | 4 +-
.../controller/LeadControllerManagerTest.java | 11 +-
.../pinot/controller/helix/ControllerTest.java | 13 +-
.../controller/helix/PinotControllerModeTest.java | 31 +-
.../helix/core/PinotHelixResourceManagerTest.java | 176 +++----
.../instance/InstanceAssignmentTest.java | 526 +++++++++++++++++++++
...eReplicaGroupSegmentAssignmentStrategyTest.java | 15 +-
...eReplicaGroupSegmentAssignmentStrategyTest.java | 9 +-
.../PinotLLCRealtimeSegmentManagerTest.java | 2 +-
.../helix/core/realtime/SegmentCompletionTest.java | 4 +-
.../ReplicaGroupRebalanceStrategyTest.java | 12 +-
.../sharding/SegmentAssignmentStrategyTest.java | 6 +-
.../pinot/core/data/readers/AvroRecordReader.java | 3 +-
.../pinot/core/data/readers/JSONRecordReader.java | 23 +-
.../function/MapValueTransformFunction.java | 95 ++++
.../function/TransformFunctionFactory.java | 2 +-
.../query/executor/ServerQueryExecutorV1Impl.java | 12 +-
.../core/query/reduce/BrokerReduceService.java | 12 +-
.../pinot/core/query/scheduler/QueryScheduler.java | 2 +-
.../stream/AvroRecordToPinotRowGenerator.java | 6 +-
.../creator/impl/SegmentDictionaryCreator.java | 2 -
.../java/org/apache/pinot/core/util/AvroUtils.java | 35 ++
.../server/realtime/ControllerLeaderLocator.java | 79 +++-
.../function/ValueInTransformFunctionTest.java | 2 -
.../realtime/ControllerLeaderLocatorTest.java | 8 +-
pinot-integration-tests/pom.xml | 2 +
.../tests/LLCRealtimeClusterIntegrationTest.java | 19 +-
.../tests/MapTypeClusterIntegrationTest.java | 223 +++++++++
...eRebalancerAdminToolClusterIntegrationTest.java | 30 +-
pinot-tools/pom.xml | 4 +
.../pinot/tools/admin/PinotAdministrator.java | 5 +-
.../pinot/tools/perf/PerfBenchmarkDriver.java | 30 +-
.../tools/tuner/CollectMetadataForIndexTuning.java | 97 ++++
.../tools/tuner/EntriesScannedQuantileReport.java | 89 ++++
.../pinot/tools/tuner/IndexTunerCommand.java | 199 ++++++++
.../pinot/tools/tuner/driver/TunerDriver.java | 209 ++++++++
.../meta/manager/JsonFileMetaManagerImpl.java | 247 ++++++++++
.../tools/tuner/meta/manager/MetaManager.java | 59 +++
.../manager/collector/ColStatsAccumulatorObj.java | 156 ++++++
.../manager/collector/CompressedFilePathIter.java | 123 +++++
.../tuner/meta/manager/collector/PathWrapper.java | 74 +++
.../collector/SegmentMetadataCollector.java | 272 +++++++++++
.../pinot/tools/tuner/query/src/InputIterator.java | 29 +-
.../tuner/query/src/LogInputIteratorImpl.java | 165 +++++++
.../query/src/parser/BrokerLogParserImpl.java | 76 +++
.../tools/tuner/query/src/parser/QueryParser.java | 23 +-
.../src/stats/wrapper/AbstractQueryStats.java | 23 +-
.../stats/wrapper/IndexSuggestQueryStatsImpl.java | 117 +++++
.../tools/tuner/strategy/AbstractAccumulator.java | 64 +++
.../tools/tuner/strategy/FrequencyAccumulator.java | 46 +-
.../pinot/tools/tuner/strategy/FrequencyImpl.java | 200 ++++++++
.../tuner/strategy/ParseBasedAccumulator.java | 62 +++
.../tools/tuner/strategy/ParserBasedImpl.java | 453 ++++++++++++++++++
.../strategy/QuantileAnalysisAccumulator.java | 88 ++++
.../tools/tuner/strategy/QuantileAnalysisImpl.java | 351 ++++++++++++++
.../pinot/tools/tuner/strategy/TuningStrategy.java | 60 +++
pom.xml | 6 +-
.../app/pods/aiavailability/controller.js | 102 ++++
.../app/pods/aiavailability/route.js | 186 ++++++++
.../app/pods/aiavailability/template.hbs | 60 +++
.../app/pods/anomalies/controller.js | 14 +-
.../pods/components/detection-yaml/component.js | 11 +-
.../app/pods/manage/explore/route.js | 13 +-
.../app/pods/manage/yaml/route.js | 13 +-
thirdeye/thirdeye-frontend/app/router.js | 2 +
.../styles/components/range-pill-selectors.scss | 4 +
thirdeye/thirdeye-frontend/app/utils/anomaly.js | 16 +-
.../thirdeye-frontend/app/utils/api/anomaly.js | 13 +-
thirdeye/thirdeye-frontend/app/utils/utils.js | 5 +-
thirdeye/thirdeye-frontend/ember-cli-build.js | 7 +
thirdeye/thirdeye-frontend/package.json | 1 +
thirdeye/thirdeye-frontend/yarn.lock | 8 +
.../content/EntityGroupKeyContentFormatter.java | 69 ++-
...HierarchicalAnomaliesEmailContentFormatter.java | 5 +-
.../MetricAnomaliesEmailContentFormatter.java | 7 +-
.../dashboard/ThirdEyeDashboardApplication.java | 3 +
.../resources/AnomalyFlattenResource.java | 228 +++++++++
.../datalayer/bao/MergedAnomalyResultManager.java | 2 +
.../bao/jdbc/MergedAnomalyResultManagerImpl.java | 9 +
.../thirdeye/datalayer/pojo/DatasetConfigBean.java | 1 -
.../datasource/pinot/PinotDataSourceMaxTime.java | 7 +-
.../datasource/pinot/PinotThirdEyeDataSource.java | 6 +-
.../pinot/thirdeye/datasource/pinot/PqlUtils.java | 9 +-
.../alert/scheme/DetectionEmailAlerter.java | 23 +-
.../stage/AnomalyDetectionStageWrapper.java | 217 ---------
.../algorithm/stage/AnomalyFilterStageWrapper.java | 105 ----
.../stage/BaselineRuleDetectionStage.java | 144 ------
.../algorithm/stage/BaselineRuleFilterStage.java | 119 -----
.../algorithm/stage/GrouperStageWrapper.java | 89 ----
.../stage/StaticAnomalyDetectionStage.java | 169 -------
.../algorithm/stage/StaticAnomalyFilterStage.java | 62 ---
.../stage/ThresholdRuleDetectionStage.java | 107 -----
.../algorithm/stage/ThresholdRuleFilterStage.java | 90 ----
.../components/MeanVarianceRuleDetector.java | 290 ++++++++++++
.../spec/MeanVarianceRuleDetectorSpec.java | 60 +++
.../apache/pinot/thirdeye/util/ThirdEyeUtils.java | 16 -
.../detector/entity-groupkey-anomaly-report.ftl | 53 +++
.../TestEntityGroupKeyEmailContentFormatter.java | 130 ++++-
.../bao/TestMergedAnomalyResultManager.java | 66 +++
.../thirdeye/datasource/pinot/PqlUtilsTest.java | 4 +-
.../resources/TestAnomalyFlattenResource.java | 144 ++++++
.../stage/AnomalyDetectionStageWrapperTest.java | 87 ----
.../stage/BaselineRuleDetectionStageTest.java | 141 ------
.../algorithm/stage/BaselineRuleFilterTest.java | 173 -------
.../stage/ThresholdRuleDetectionStageTest.java | 93 ----
.../algorithm/stage/ThresholdRuleFilterTest.java | 167 -------
.../components/MeanVarianceRuleDetectorTest.java | 175 +++++++
.../tools/RunAdhocDatabaseQueriesTool.java | 1 +
.../thirdeye/detection/algorithm/timeseries-2y.csv | 107 +++++
...est-entity-groupby-email-content-formatter.html | 2 +
...by-with-whitelist-email-content-formatter.html} | 34 ++
141 files changed, 7554 insertions(+), 2657 deletions(-)
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceAssignmentConfig.java
copy
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaseDetectionStage.java
=>
pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceAssignmentConfigMapChildKeyHandler.java
(64%)
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceAssignmentConfigUtils.java
rename
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/AnomalyDetectionStage.java
=>
pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceConstraintConfig.java
(54%)
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceReplicaPartitionConfig.java
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceTagPoolConfig.java
create mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/HashBasedRotateInstanceConstraintApplier.java
create mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
rename
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/GrouperStage.java
=>
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceConstraintApplier.java
(57%)
create mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaPartitionSelector.java
create mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceTagPoolSelector.java
create mode 100644
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentTest.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/MapValueTransformFunction.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MapTypeClusterIntegrationTest.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/CollectMetadataForIndexTuning.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/EntriesScannedQuantileReport.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/IndexTunerCommand.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/driver/TunerDriver.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/JsonFileMetaManagerImpl.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/MetaManager.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/collector/ColStatsAccumulatorObj.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/collector/CompressedFilePathIter.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/collector/PathWrapper.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/collector/SegmentMetadataCollector.java
copy
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/AnomalyFilterStage.java
=>
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/InputIterator.java
(53%)
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/LogInputIteratorImpl.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/parser/BrokerLogParserImpl.java
copy
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaseDetectionStage.java
=>
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/parser/QueryParser.java
(63%)
rename
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaseDetectionStage.java
=>
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/stats/wrapper/AbstractQueryStats.java
(64%)
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/stats/wrapper/IndexSuggestQueryStatsImpl.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/AbstractAccumulator.java
rename
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/AnomalyFilterStage.java
=>
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/FrequencyAccumulator.java
(51%)
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/FrequencyImpl.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/ParseBasedAccumulator.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/ParserBasedImpl.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/QuantileAnalysisAccumulator.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/QuantileAnalysisImpl.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/TuningStrategy.java
create mode 100644
thirdeye/thirdeye-frontend/app/pods/aiavailability/controller.js
create mode 100644 thirdeye/thirdeye-frontend/app/pods/aiavailability/route.js
create mode 100644
thirdeye/thirdeye-frontend/app/pods/aiavailability/template.hbs
create mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/AnomalyFlattenResource.java
delete mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/AnomalyDetectionStageWrapper.java
delete mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/AnomalyFilterStageWrapper.java
delete mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaselineRuleDetectionStage.java
delete mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaselineRuleFilterStage.java
delete mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/GrouperStageWrapper.java
delete mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/StaticAnomalyDetectionStage.java
delete mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/StaticAnomalyFilterStage.java
delete mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/ThresholdRuleDetectionStage.java
delete mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/ThresholdRuleFilterStage.java
create mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/MeanVarianceRuleDetector.java
create mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/spec/MeanVarianceRuleDetectorSpec.java
create mode 100644
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/datasource/resources/TestAnomalyFlattenResource.java
delete mode 100644
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/stage/AnomalyDetectionStageWrapperTest.java
delete mode 100644
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaselineRuleDetectionStageTest.java
delete mode 100644
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaselineRuleFilterTest.java
delete mode 100644
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/stage/ThresholdRuleDetectionStageTest.java
delete mode 100644
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/stage/ThresholdRuleFilterTest.java
create mode 100644
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/components/MeanVarianceRuleDetectorTest.java
create mode 100644
thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/algorithm/timeseries-2y.csv
copy
thirdeye/thirdeye-pinot/src/test/resources/{test-entity-groupby-email-content-formatter.html
=> test-entity-groupby-with-whitelist-email-content-formatter.html} (73%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
