This is an automated email from the ASF dual-hosted git repository.
kishoreg pushed a change to branch nested-object-indexing
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
discard 4c39301 Wiring up end to end to support indexing nested fields on
complex objects
discard ffd9256 Adding support for Object Type
discard 0766764 Enhancing PQL to support MATCHES predicate, can be used for
searching within text, map, json and other complex objects
discard 2dbe2d8 Adding support for MATCHES Predicate
new b132159 [TE] yaml - subscribed detection names in notification yaml
(#3714)
new 4716f74 [TE] - Fix true anomaly definition to include expected
anomaly (#3709)
new e6dfe31 Move instantiation of objects which depend on
_helixResourceManager to after invocation of PinotHelixResourceManager::start
(#3713)
new f09db48 [TE] alerter - do not send replay anomalies (#3720)
new 8a231a6 Fix SegmentConverter to handle virtual column (#3711)
new 8daad94 [TE] Migration endpoints for anomaly function and application
(#3724)
new 79ef7f9 Minor fix for star-tree v2 metadata (#3733)
new 4dd29d7 Update versions of packages to move to jersey 2.28 (#3730)
new 525102f Replace partition ranges with partitions (#3725)
new d1fa0e1 [TE] Add Path to the Migration endpoints (#3735)
new f22d272 [TE] detection - Support enable/disable detection/alert in
Yaml (#3736)
new 086b72b [TE] aaronucsd/new edit basic yaml editor (#3737)
new d9fb451 Update Pinot admin webpages with right references (#3738)
new 189e672 Fix controller/server/broker admin console (#3740)
new 79f143b [TE] Bump Shade plugin version (#3742)
new a08f892 User doc for Star-Tree index (#3743)
new 2995f3b [TE] aaronucsd/update timeout for the email timer for missing
chart (#3744)
new 1724bc7 Fix SegmentZKMetadataTest.segmentPartitionMetadataTest()
(#3745)
new 335e967 Fix a bug in
PinotSegmentRestletResource.getInstanceToSegmentsMap() (#3746)
new eb7a918 Add integration test for SegmentStatusChecker periodic task
(#3741)
new 73ddf07 [TE] Exclude Jersey from Pinot. Jersey version upgrade in
Pinot causing run-time issues in ThirdEye (#3752)
new 9004e38 Fix SegmentStatusCheckerIntegrationTest setup timings (#3749)
new 587c597 Email share (#3753)
new 6e10553 [TE] Clean up and pretty print yaml (#3754)
new 2f597c5 Revert "User doc for Star-Tree index (#3743)" (#3747)
new c47ef66 [TE] frontend - harleyjj/yaml-editor - accordion for preview
in YAML editor (#3756)
new 0b72fd2 [TE] detection - report missing anomaly endpoint for pipeline
v2 (#3755)
new 8f239f5 [TE] yaml - preview endpoint (#3757)
new ce2d9ee Auto-reformat all java source files (#3739)
new 42d00ce Adding pluggable storage docs (#3751)
new 205482f [TE] Clean-up the migration logs (#3760)
new 4a1c373 Add Documents for Index Techniques (#3761)
new f8a1ff2 Better handle NPE from getting instance config (#3758)
new 4a52a9c [TE] detection - endpoint to show predicted baseline (#3764)
new 37fb02b [TE] Polish the migration endpoint and fix corner cases
(#3765)
new 2ce41cf [TE] Migrate unsubscribed anomaly functions (#3766)
new 96d9c22 Update LICENSE and NOTICE files (#3722)
new 94b34e2 Documentation review on Pinot Overview (#3762)
new ec6a6e0 Remove unnecessary init from crypter (#3770)
new 67c6507 Modify Makefile for docs (#3771)
new 0f4bf85 [TE] rootcause - callgraph gui (#3769)
new 006cba0 Pick up official release of jersey 2.28 (#3768)
new 5f195b6 [TE] detection - yaml translation/migration changes (#3767)
new 26f2aad [TE] Queries for fetching detection & notification owner
report (#3773)
new 7bd25cc [TE] detection - minor fixes for integration test (#3774)
new 7e28022 Remove dependencies from libraries under category X licenses.
(#3718)
new f48fdd3 Update pom files for preparing Apache release (#3772)
new b24f46a Minor fix for integration test (#3777)
new ab04398 Fix wrong Pinot versions (0.016->0.1.0-SNAPSHOT) (#3778)
new 1c7b03f Add log for delete table API (#3763)
new 7e14e4f [TE] Clean up the useless exceptions (#3779)
new 98ef340 Add maven-gpg-plugin and checksum-maven-plugin (#3776)
new 1be4b5c [TE] Replace response message with exceptions; Reuse
notification endpoint in Create Alert (#3780)
new cc658e4 [TE] Clean up the log messages and exceptions (#3782)
new e9ab3b6 Add license header for codemirror/ruby.js (#3781)
new e9d4100 Add methods toPrettyJsonString() and toSingleLineJsonString()
to Schema (#3783)
new 4937e15 fix BitmapInvertedIndexTest on Windows (#3786)
new 103bc78 Updating README.md (#3784)
new 881de3f [TE] add anomaly classification to detection anomalies
endpoint (#3789)
new 0e7d657 [TE] frontend - aaronucsd/Added the new preview pills and
performance header (#3792)
new 866dfc7 Fixing jackson dependency issues with swagger (#3793)
new c21ac1e [TE] Fix logging bug in migration endpoint (#3794)
new a1a7f42 [TE] Anomalies page bug fixes (#3795)
new 2f28352 Fix some indentation for the pom files (#3797)
new 20dbcc4 [TE] frontend - harleyjj/yaml-editor - improves yaml
autocomplete (#3732)
new 280c192 [TE] frontend - harleyjj/yaml-editor - implements simple
graph for preview (#3787)
new e121b83 [TE] frontend - harleyjj/yaml-editor - link Documentation
buttons to YAML Documentation (#3800)
new ada802e [TE] API fix for partially migrated alerts in dashboard
(#3801)
new a93fc77 [TE] yaml - change default preview config id (#3799)
new 2321864 Update LICENSE and NOTICE for jersey version update (#3791)
new d8f9140 [TE] Include a detection description field to describe the
alert (#3798)
new 521d551 [TE] Retain create owners of anomaly functions and alerts
(#3804)
new ab203a9 [PINOT-7658] Moving design documents to cwiki (#3802)
new 46f67a4 [TE] Retain status of inactive functions (#3803)
new cd2dc04 [TE] dashboard link point to yaml if the anomaly is detected
by new pipeline (#3805)
new fdd7102 [TE] Remove duplicate alert validation for update (#3806)
new 2587998 [TE] Validation info to display on UI (#3809)
new 5dec913 [TE] frontend - harleyjj/alert-details - updates to
alert-details component (#3811)
new ac058c5 [TE] Refactor detection & preview API for better debugging
(#3812)
new 8b90a54 [TE] yaml - onboarding tasks (#3790)
new a51d418 [TE] yaml-editor - makes yaml autocomplete insert dataset
field correctly (#3817)
new 2a5b348 [TE] Move wrapper classes under api to common (#3815)
new 017e9e5 [TE] detection - caching & configure time granularity (#3810)
new eacb020 Refactor Hadoop Jobs (#3813)
new 4039e57 [TE] frontend - aarouncsd/add debug global flag (#3818)
new eb0d5b6 [TE] yaml - error messages (#3816)
new 2fb6371 Refactor periodic task (#3819)
new 1e65579 Updated hyperlinks and Fully Qualified Class Name for
KafkaConsumerFactory in documentation (#3826)
new 0df5f81 [TE] Endpoint to migrate all the applications (#3824)
new 4650440 Fix a typo and javadoc in SegmentCreationJob (#3825)
new d7fb38b Removed all references to linkedin from the docs (#3828)
new 16eb7c7 [TE] Return yaml errors after grouping and use consistent
convention (subscription over notification) (#3822)
new 8524ca4 [TE] frontend - aaronucsd/update endpoint names to
subscription (#3831)
new e518d0a [TE] Extend anomalies endpoint to fetch by metric/dataset and
true anomalies (#3832)
new 7832abf [TE] frontend - harleyjj/alert-details - add duration to
anomaly table and hide preview when yaml field updated (#3823)
new e4a05b2 Update maven config files for preparing release (#3833)
new d3fad80 Refactor SegmentNameGenerators and integrate them into Hadoop
(#3821)
new 3217d80 Add progress reporter to SegmentCreationMapper (#3835)
new fb9c1dc Clarify all methods in PinotFS (#3836)
new 7ca6841 In SegmentCreationJob, fix the issue where ControllerRestApi
is used when _pushLocations is not set (#3837)
new 612b4fb [TE] Move Ember configs into thirdeye-configs - Attempt to
isolate LinkedIn specific stuff in UI (#3839)
new 729ba77 Add headers for docs (#3840)
new 093f917 Bug fix in SegmentCreationJob and SegmentCreationMapper
(#3844)
new 2a38602 Remove temp files from maven-release plugin from source
tarbell (#3845)
new 8ecfbcb [TE] frontend - aaronucsd/fixed the alert and subscription
error messages (#3841)
new 008bb5c [TE] frontend - harleyjj/alert-overview - implements alert
overview for new pipeline using alert-details component (#3843)
new a3dd11b [TE] Enable Piwik tracking ref: #3839 (#3846)
new e89e637 Adding support for MATCHES Predicate
new a055202 Enhancing PQL to support MATCHES predicate, can be used for
searching within text, map, json and other complex objects
new 6ad8a96 Adding support for Object Type
new fa3ba9b Wiring up end to end to support indexing nested fields on
complex objects
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4c39301)
\
N -- N -- N refs/heads/nested-object-indexing (fa3ba9b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 5751 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
DISCLAIMER | 10 +
LICENSE | 30 +-
LICENSE => LICENSE-binary | 186 +-
NOTICE | 42 +-
NOTICE-binary | 321 ++++
README.md | 199 +-
.../org/apache/pinotdruidbenchmark/DataMerger.java | 3 +-
.../pinotdruidbenchmark/DruidResponseTime.java | 9 +-
.../pinotdruidbenchmark/PinotResponseTime.java | 6 +-
docs/High-level-stream.png | Bin 39800 -> 0 bytes
docs/Low-level-stream.png | Bin 37306 -> 0 bytes
docs/Makefile | 2 +-
docs/PlanNode.png | Bin 41056 -> 0 bytes
docs/ServerSegmentCompletion.dot.png | Bin 72968 -> 0 bytes
docs/architecture.rst | 29 +-
docs/client_api.rst | 25 +-
docs/commit-happy-path-1.png | Bin 22933 -> 0 bytes
docs/commit-happy-path-2.png | Bin 22939 -> 0 bytes
docs/committer-failed.png | Bin 15412 -> 0 bytes
docs/controller-failed.png | Bin 23599 -> 0 bytes
docs/controller-segment-completion.png | Bin 85092 -> 0 bytes
docs/delayed-server.png | Bin 21690 -> 0 bytes
docs/expressionTree.jpg | Bin 12881 -> 0 bytes
docs/expressions_udf.rst | 120 --
docs/in_production.rst | 19 +
docs/index.rst | 43 +-
docs/index_techniques.rst | 75 +
docs/intro.rst | 29 +-
docs/llc.rst | 164 --
docs/management_api.rst | 19 +
docs/multiple-server-failure.png | Bin 18036 -> 0 bytes
docs/multitenancy.rst | 20 +-
docs/parseTree.png | Bin 55136 -> 0 bytes
docs/partition_aware_routing.rst | 141 --
docs/pinot_hadoop.rst | 31 +-
docs/pluggable_storage.rst | 40 +
docs/pluggable_streams.rst | 40 +-
docs/pql_examples.rst | 19 +
docs/reference.rst | 21 +-
docs/schema_timespec.rst | 111 --
docs/segment-consumer-fsm.png | Bin 30109 -> 0 bytes
docs/segment-creation.png | Bin 9830 -> 0 bytes
docs/segment-helix-fsm.png | Bin 7494 -> 0 bytes
docs/segment_fetcher.rst | 29 +-
docs/trying_pinot.rst | 34 +-
docs/zk-setup.png | Bin 39246 -> 0 bytes
licenses-binary/LICENSE-angular.txt | 21 +
licenses-binary/LICENSE-antlr.txt | 8 +
licenses-binary/LICENSE-args4j.txt | 19 +
.../LICENSE-azure-data-lake-store-sdk.txt | 24 +
licenses-binary/LICENSE-cddl-1.0.txt | 384 ++++
licenses-binary/LICENSE-cddl-1.1.txt | 759 ++++++++
licenses-binary/LICENSE-codemirror.txt | 21 +
licenses-binary/LICENSE-cpl-1.0.txt | 88 +
licenses-binary/LICENSE-epl-1.0.txt | 86 +
licenses-binary/LICENSE-epl-2.0.txt | 637 +++++++
licenses-binary/LICENSE-foundation.txt | 22 +
licenses-binary/LICENSE-handlebars.txt | 19 +
licenses-binary/LICENSE-jcabi.txt | 27 +
licenses-binary/LICENSE-jline.txt | 32 +
licenses-binary/LICENSE-jopt-simple.txt | 23 +
licenses-binary/LICENSE-jquery-datatables.txt | 12 +
licenses-binary/LICENSE-jquery.txt | 20 +
licenses-binary/LICENSE-js-beautify.txt | 9 +
licenses-binary/LICENSE-mimepull.txt | 28 +
licenses-binary/LICENSE-normalize.txt | 21 +
licenses-binary/LICENSE-paranamer.txt | 28 +
licenses-binary/LICENSE-scala.txt | 30 +
licenses-binary/LICENSE-slf4j.txt | 21 +
licenses-binary/LICENSE-underscore.txt | 23 +
licenses/LICENSE-angular.txt | 21 +
licenses/LICENSE-codemirror.txt | 21 +
licenses/LICENSE-foundation.txt | 22 +
licenses/LICENSE-handlebars.txt | 19 +
licenses/LICENSE-jquery-datatables.txt | 12 +
licenses/LICENSE-jquery.txt | 20 +
licenses/LICENSE-js-beautify.txt | 9 +
licenses/LICENSE-normalize.txt | 21 +
licenses/LICENSE-underscore.txt | 23 +
pinot-api/pom.xml | 3 +-
.../org/apache/pinot/client/BrokerSelector.java | 3 +-
.../java/org/apache/pinot/client/Connection.java | 22 +-
.../org/apache/pinot/client/ConnectionFactory.java | 6 +-
.../apache/pinot/client/DynamicBrokerSelector.java | 12 +-
.../client/JsonAsyncHttpPinotClientTransport.java | 9 +-
.../apache/pinot/client/PinotClientTransport.java | 7 +-
.../java/org/apache/pinot/client/ResultSet.java | 4 +-
.../apache/pinot/client/SimpleBrokerSelector.java | 2 +-
.../java/org/apache/pinot/client/TextTable.java | 2 +-
.../apache/pinot/client/ConnectionFactoryTest.java | 3 +-
.../apache/pinot/client/ResultSetGroupTest.java | 8 +-
pinot-azure-filesystem/pom.xml | 5 +-
.../org/apache/pinot/filesystem/AzurePinotFS.java | 46 +-
.../datalake/store/MockADLFileInputStream.java | 21 +-
.../pinot/filesystem/test/AzurePinotFSTest.java | 13 +-
pinot-broker/pom.xml | 7 +-
.../apache/pinot/broker/api/RequestStatistics.java | 4 +-
.../broker/api/resources/PinotBrokerDebug.java | 17 +-
.../api/resources/PinotBrokerHealthCheck.java | 6 +-
.../broker/api/resources/PinotClientRequest.java | 10 +-
.../pinot/broker/broker/AccessControlFactory.java | 6 +-
.../broker/AllowAllAccessControlFactory.java | 1 +
.../broker/broker/BrokerAdminApiApplication.java | 4 +-
.../pinot/broker/broker/BrokerServerBuilder.java | 10 +-
...okerResourceOnlineOfflineStateModelFactory.java | 8 +-
.../broker/broker/helix/ClusterChangeMediator.java | 12 +-
.../broker/broker/helix/HelixBrokerStarter.java | 88 +-
.../helix/LiveInstancesChangeListenerImpl.java | 10 +-
.../TimeboundaryRefreshMessageHandlerFactory.java | 181 +-
.../broker/pruner/PartitionZKMetadataPruner.java | 14 +-
.../apache/pinot/broker/queryquota/HitCounter.java | 2 +-
.../broker/queryquota/TableQueryQuotaManager.java | 13 +-
.../requesthandler/BaseBrokerRequestHandler.java | 16 +-
.../requesthandler/BrokerRequestHandler.java | 3 +-
.../requesthandler/BrokerRequestOptimizer.java | 17 +-
.../ConnectionPoolBrokerRequestHandler.java | 10 +-
...enNestedPredicatesFilterQueryTreeOptimizer.java | 4 +-
...ualitiesToInClauseFilterQueryTreeOptimizer.java | 11 +-
.../broker/requesthandler/OptimizationFlags.java | 3 +-
.../SingleConnectionBrokerRequestHandler.java | 13 +-
.../pinot/broker/routing/CfgBasedRouting.java | 3 +-
.../routing/HelixExternalViewBasedRouting.java | 14 +-
.../apache/pinot/broker/routing/RoutingTable.java | 3 +-
.../broker/routing/RoutingTableBuilderFactory.java | 11 +-
.../pinot/broker/routing/TimeBoundaryService.java | 3 +-
.../BasePartitionAwareRoutingTableBuilder.java | 3 +-
.../DefaultRealtimeRoutingTableBuilder.java | 3 +-
.../LowLevelConsumerRoutingTableBuilder.java | 9 +-
.../builder/LowLevelRoutingTableBuilderUtil.java | 12 +-
.../PartitionAwareOfflineRoutingTableBuilder.java | 15 +-
.../PartitionAwareRealtimeRoutingTableBuilder.java | 6 +-
.../builder/RoutingTableInstancePruner.java | 4 +-
.../routing/selector/MergedSegmentSelector.java | 8 +-
.../pinot/broker/broker/BrokerTestUtils.java | 3 +-
.../EmptyBrokerOnlineOfflineStateModelFactory.java | 1 -
...EmptySegmentOnlineOfflineStateModelFactory.java | 1 -
.../broker/broker/HelixBrokerStarterTest.java | 117 +-
.../broker/broker/HelixBrokerStarterUtilsTest.java | 4 +-
.../broker/pruner/SegmentZKMetadataPrunerTest.java | 6 +-
.../queryquota/TableQueryQuotaManagerTest.java | 152 +-
.../broker/requesthandler/FilterOptimizerTest.java | 26 +-
...tiesToInClauseFilterQueryTreeOptimizerTest.java | 54 +-
.../requesthandler/RangeMergeOptimizerTest.java | 4 +-
.../broker/routing/RandomRoutingTableTest.java | 6 +-
.../pinot/broker/routing/RoutingTableTest.java | 159 +-
.../broker/routing/TimeBoundaryServiceTest.java | 14 +-
.../BalancedRandomRoutingTableBuilderTest.java | 3 +-
.../HighLevelConsumerRoutingTableBuilderTest.java | 3 +-
.../LargeClusterRoutingTableBuilderTest.java | 13 +-
.../LowLevelConsumerRoutingTableBuilderTest.java | 11 +-
...rtitionAwareOfflineRoutingTableBuilderTest.java | 36 +-
...titionAwareRealtimeRoutingTableBuilderTest.java | 35 +-
.../selector/MergedSegmentSelectorTest.java | 25 +-
.../pinot/broker/util/FakePropertyStore.java | 4 +-
pinot-common/pom.xml | 3 +-
.../main/java/org/apache/pinot/common/Utils.java | 3 +-
.../config/AdjustTableNameChildKeyTransformer.java | 29 +-
.../pinot/common/config/ChildKeyTransformer.java | 1 +
.../pinot/common/config/ColumnPartitionConfig.java | 75 +-
.../apache/pinot/common/config/CombinedConfig.java | 22 +-
.../pinot/common/config/CombinedConfigLoader.java | 33 +-
...CombinedConfigSeparatorChildKeyTransformer.java | 20 +-
.../org/apache/pinot/common/config/ConfigDoc.java | 3 +
.../apache/pinot/common/config/Deserializer.java | 53 +-
.../org/apache/pinot/common/config/Duration.java | 3 +-
.../apache/pinot/common/config/IndexingConfig.java | 26 +-
.../common/config/NamedListChildKeyHandler.java | 48 +-
.../apache/pinot/common/config/QuotaConfig.java | 4 +-
.../config/RemapTableTypesChildKeyTransformer.java | 34 +-
.../common/config/ReplicaGroupStrategyConfig.java | 7 +-
.../apache/pinot/common/config/RoutingConfig.java | 4 +-
.../common/config/SegmentPartitionConfig.java | 6 +-
.../SegmentsValidationAndRetentionConfig.java | 22 +-
.../org/apache/pinot/common/config/Serializer.java | 19 +-
.../common/config/SimpleMapChildKeyHandler.java | 3 +-
.../apache/pinot/common/config/TableConfig.java | 25 +-
.../org/apache/pinot/common/config/TagConfig.java | 1 -
.../apache/pinot/common/config/TagNameUtils.java | 1 -
.../pinot/common/config/TagOverrideConfig.java | 4 +-
.../config/TaskConfigMapChildKeyHandler.java | 18 +-
.../org/apache/pinot/common/config/Tenant.java | 7 +-
.../apache/pinot/common/config/TenantConfig.java | 1 +
.../common/config/TypedMapChildKeyHandler.java | 33 +-
.../pinot/common/data/DateTimeFieldSpec.java | 3 +-
.../common/data/DateTimeFormatPatternSpec.java | 5 +-
.../pinot/common/data/DateTimeFormatSpec.java | 19 +-
.../pinot/common/data/DateTimeFormatUnitSpec.java | 34 +-
.../pinot/common/data/DateTimeGranularitySpec.java | 15 +-
.../org/apache/pinot/common/data/FieldSpec.java | 22 +-
.../apache/pinot/common/data/MetricFieldSpec.java | 4 +-
.../java/org/apache/pinot/common/data/Schema.java | 69 +-
.../pinot/common/data/StarTreeIndexSpec.java | 30 +-
.../apache/pinot/common/data/TimeFieldSpec.java | 4 +-
.../pinot/common/data/TimeGranularitySpec.java | 8 +-
.../common/exception/InvalidConfigException.java | 1 -
.../common/exception/TableNotFoundException.java | 1 -
.../pinot/common/lineage/SegmentMergeLineage.java | 9 +-
.../messages/TimeboundaryRefreshMessage.java | 31 +-
.../pinot/common/metadata/ZKMetadataProvider.java | 39 +-
.../metadata/instance/InstanceZKMetadata.java | 6 +-
.../metadata/segment/ColumnPartitionMetadata.java | 148 +-
.../segment/LLCRealtimeSegmentZKMetadata.java | 14 +-
.../metadata/segment/OfflineSegmentZKMetadata.java | 6 +-
.../segment/RealtimeSegmentZKMetadata.java | 8 +-
.../metadata/segment/SegmentPartitionMetadata.java | 35 +-
.../common/metadata/segment/SegmentZKMetadata.java | 23 +-
.../SegmentZKMetadataCustomMapModifier.java | 3 +-
.../pinot/common/metrics/AbstractMetrics.java | 113 +-
.../pinot/common/metrics/AggregatedCounter.java | 6 +-
.../pinot/common/metrics/AggregatedHistogram.java | 4 +-
.../pinot/common/metrics/AggregatedMeter.java | 12 +-
.../common/metrics/AggregatedMetricsRegistry.java | 3 +-
.../apache/pinot/common/metrics/BrokerGauge.java | 9 +-
.../pinot/common/metrics/BrokerQueryPhase.java | 6 +-
.../apache/pinot/common/metrics/BrokerTimer.java | 9 +-
.../pinot/common/metrics/ControllerGauge.java | 39 +-
.../pinot/common/metrics/ControllerMeter.java | 3 +-
.../pinot/common/metrics/ControllerTimer.java | 6 +-
.../apache/pinot/common/metrics/LatencyMetric.java | 1 -
.../apache/pinot/common/metrics/MetricsHelper.java | 5 +-
.../apache/pinot/common/metrics/ServerGauge.java | 8 +-
.../pinot/common/metrics/ServerQueryPhase.java | 6 +-
.../apache/pinot/common/metrics/ServerTimer.java | 6 +-
.../pinot/common/metrics/ValidationMetrics.java | 34 +-
.../common/partition/PartitionAssignment.java | 6 +-
.../partition/ReplicaGroupPartitionAssignment.java | 2 +-
.../ReplicaGroupPartitionAssignmentGenerator.java | 3 +-
.../protocols/SegmentCompletionProtocol.java | 78 +-
.../request/transform/TransformExpressionTree.java | 4 +-
.../pinot/common/response/BrokerResponse.java | 3 +-
.../response/broker/BrokerResponseNative.java | 11 +-
.../common/response/broker/GroupByResult.java | 1 -
.../common/restlet/resources/RebalanceResult.java | 1 +
.../common/restlet/resources/TableSegments.java | 1 +
.../org/apache/pinot/common/segment/ReadMode.java | 7 +-
.../common/segment/fetcher/HdfsSegmentFetcher.java | 12 +-
.../common/segment/fetcher/HttpSegmentFetcher.java | 3 +-
.../segment/fetcher/HttpsSegmentFetcher.java | 6 +-
.../segment/fetcher/LocalFileSegmentFetcher.java | 3 +-
.../segment/fetcher/PinotFSSegmentFetcher.java | 3 +-
.../common/segment/fetcher/SegmentFetcher.java | 3 +-
.../segment/fetcher/SegmentFetcherFactory.java | 10 +-
.../common/utils/ClientSSLContextGenerator.java | 12 +-
.../apache/pinot/common/utils/CommonConstants.java | 28 +-
.../org/apache/pinot/common/utils/DataSchema.java | 10 +-
.../org/apache/pinot/common/utils/DataSize.java | 8 +-
.../org/apache/pinot/common/utils/DataTable.java | 3 +-
.../common/utils/FileUploadDownloadClient.java | 123 +-
.../org/apache/pinot/common/utils/FileUtils.java | 6 +-
.../apache/pinot/common/utils/HLCSegmentName.java | 8 +-
.../org/apache/pinot/common/utils/HashUtil.java | 8 +-
.../org/apache/pinot/common/utils/JsonUtils.java | 36 +-
.../pinot/common/utils/KafkaStarterUtils.java | 3 +-
.../apache/pinot/common/utils/LLCSegmentName.java | 8 +-
.../org/apache/pinot/common/utils/LLCUtils.java | 1 +
.../pinot/common/utils/NamedThreadFactory.java | 1 -
.../org/apache/pinot/common/utils/NetUtil.java | 3 +-
.../java/org/apache/pinot/common/utils/Pairs.java | 10 +-
.../pinot/common/utils/PrimitiveArrayUtils.java | 11 +-
.../org/apache/pinot/common/utils/SchemaUtils.java | 17 +-
.../org/apache/pinot/common/utils/SegmentName.java | 9 +-
.../apache/pinot/common/utils/ServiceStatus.java | 25 +-
.../pinot/common/utils/TarGzCompressionUtils.java | 16 +-
.../org/apache/pinot/common/utils/TenantRole.java | 3 +-
.../org/apache/pinot/common/utils/ZkStarter.java | 8 +-
.../pinot/common/utils/helix/HelixHelper.java | 12 +-
.../PinotHelixPropertyStoreZnRecordProvider.java | 3 +-
.../common/utils/request/HavingQueryTree.java | 11 +-
.../pinot/common/utils/retry/BaseRetryPolicy.java | 3 +-
.../pinot/common/utils/retry/RetryPolicy.java | 3 +-
.../pinot/common/utils/time/TimeConverter.java | 10 +-
.../apache/pinot/common/utils/time/TimeUtils.java | 17 +-
.../common/utils/webhdfs/WebHdfsV1Client.java | 27 +-
.../apache/pinot/pql/parsers/AbstractCompiler.java | 2 +
.../org/apache/pinot/pql/parsers/Pql2Compiler.java | 3 +-
.../pinot/pql/parsers/pql2/ast/BaseAstNode.java | 2 +-
.../parsers/pql2/ast/BetweenPredicateAstNode.java | 8 +-
.../parsers/pql2/ast/BooleanOperatorAstNode.java | 5 +-
.../pql2/ast/ComparisonPredicateAstNode.java | 7 +-
.../pinot/pql/parsers/pql2/ast/HavingAstNode.java | 1 -
.../pql/parsers/pql2/ast/IdentifierAstNode.java | 11 +-
.../pinot/pql/parsers/pql2/ast/LimitAstNode.java | 5 +-
.../pql/parsers/pql2/ast/PredicateAstNode.java | 2 +-
.../pql2/ast/RegexpLikePredicateAstNode.java | 6 +-
.../pinot/pql/parsers/pql2/ast/SelectAstNode.java | 8 +-
.../parsers/pql2/ast/StarColumnListAstNode.java | 2 +-
.../pql/parsers/pql2/ast/TableNameAstNode.java | 4 +-
.../org/apache/pinot/startree/hll/HllConfig.java | 15 +-
.../apache/pinot/startree/hll/HllConstants.java | 4 +-
.../apache/pinot/startree/hll/HllSizeUtils.java | 4 +-
.../pinot/common/config/IndexingConfigTest.java | 9 +-
.../pinot/common/config/QuotaConfigTest.java | 21 +-
.../pinot/common/config/TableConfigTest.java | 65 +-
.../pinot/common/config/TagOverrideConfigTest.java | 40 +-
.../org/apache/pinot/common/config/TenantTest.java | 3 +-
.../common/config/TestConfigSerialization.java | 3 +-
.../pinot/common/data/DateTimeFormatSpecTest.java | 292 +--
.../common/data/DateTimeGranularitySpecTest.java | 46 +-
.../apache/pinot/common/data/FieldSpecTest.java | 9 +-
.../org/apache/pinot/common/data/SchemaTest.java | 95 +-
.../pinot/common/http/MultiGetRequestTest.java | 15 +-
.../common/lineage/SegmentMergeLineageTest.java | 18 +-
.../metadata/MetadataEqualsHashCodeTest.java | 12 +-
.../common/metadata/SegmentZKMetadataTest.java | 39 +-
.../segment/ColumnPartitionMetadataTest.java | 71 +
.../pinot/common/metrics/MetricsHelperTest.java | 3 +-
.../common/partition/IdealStateBuilderUtil.java | 6 +-
.../StreamPartitionAssignmentGeneratorTest.java | 3 +-
.../pinot/common/query/gen/AvroQueryGenerator.java | 32 +-
.../transform/TransformExpressionTreeTest.java | 4 +-
.../segment/fetcher/SegmentFetcherFactoryTest.java | 16 +-
.../apache/pinot/common/utils/DataSchemaTest.java | 3 +-
.../apache/pinot/common/utils/DataSizeTest.java | 11 +-
.../common/utils/FileUploadDownloadClientTest.java | 31 +-
.../pinot/common/utils/SegmentNameBuilderTest.java | 27 +-
.../pinot/common/utils/ServiceStatusTest.java | 32 +-
.../common/utils/TarGzCompressionUtilsTest.java | 13 +-
.../org/apache/pinot/common/utils/UtilsTest.java | 26 +-
.../pinot/common/utils/retry/RetryPolicyTest.java | 3 +-
.../apache/pinot/pql/parsers/Pql2CompilerTest.java | 12 +-
.../pinot/request/BrokerResponseNativeTest.java | 9 +-
pinot-controller/pom.xml | 7 +-
.../apache/pinot/controller/ControllerConf.java | 64 +-
.../apache/pinot/controller/ControllerStarter.java | 81 +-
.../pinot/controller/api/CallableHttpGet.java | 1 -
.../api/ControllerAdminApiApplication.java | 31 +-
.../api/events/DefaultMetadataEventNotifier.java | 1 +
.../DefaultMetadataEventNotifierFactory.java | 1 -
.../api/events/MetadataEventNotifier.java | 1 +
.../pinot/controller/api/pojos/Instance.java | 8 +-
.../pinot/controller/api/resources/Constants.java | 4 +-
.../api/resources/ControllerResponseFilter.java | 4 +-
.../api/resources/FileUploadPathProvider.java | 6 +-
.../controller/api/resources/HeaderAdder.java | 13 +-
.../resources/LLCSegmentCompletionHandlers.java | 54 +-
.../api/resources/LandingPageHandler.java | 1 +
.../api/resources/PinotControllerHealthCheck.java | 4 +-
.../resources/PinotInstanceRestletResource.java | 45 +-
.../api/resources/PinotSchemaRestletResource.java | 6 +-
.../api/resources/PinotSegmentRestletResource.java | 15 +-
.../PinotSegmentUploadRestletResource.java | 41 +-
.../resources/PinotTableConfigRestletResource.java | 87 +-
.../api/resources/PinotTableIndexingConfigs.java | 10 +-
.../api/resources/PinotTableInstances.java | 4 +-
.../api/resources/PinotTableMetadataConfigs.java | 10 +-
.../api/resources/PinotTableRestletResource.java | 25 +-
.../controller/api/resources/PinotTableSchema.java | 10 +-
.../api/resources/PinotTableSegmentConfigs.java | 22 +-
.../api/resources/PinotTableTenantConfigs.java | 10 +-
.../api/resources/PinotTenantRestletResource.java | 105 +-
.../controller/api/resources/PqlQueryResource.java | 16 +-
.../pinot/controller/api/resources/StateType.java | 4 +-
.../pinot/controller/api/resources/TableSize.java | 30 +-
.../pinot/controller/api/resources/TableViews.java | 24 +-
.../resources/WebApplicationExceptionMapper.java | 6 +-
.../controller/api/upload/SegmentValidator.java | 17 +-
.../pinot/controller/api/upload/ZKOperator.java | 31 +-
.../helix/ControllerRequestBuilderUtil.java | 68 +-
.../helix/ControllerRequestURLBuilder.java | 35 +-
.../EmptyBrokerOnlineOfflineStateModelFactory.java | 1 -
...EmptySegmentOnlineOfflineStateModelFactory.java | 4 +-
.../controller/helix/SegmentStatusChecker.java | 289 ++-
.../helix/core/PinotHelixResourceManager.java | 182 +-
.../helix/core/PinotTableIdealStateBuilder.java | 12 +-
.../helix/core/SegmentDeletionManager.java | 31 +-
.../controller/helix/core/TableRebalancer.java | 33 +-
.../helix/core/minion/ClusterInfoProvider.java | 8 +-
.../core/minion/PinotHelixTaskResourceManager.java | 23 +-
.../helix/core/minion/PinotTaskManager.java | 125 +-
.../generator/ConvertToRawIndexTaskGenerator.java | 4 +-
.../core/periodictask/ControllerPeriodicTask.java | 170 +-
.../realtime/PinotLLCRealtimeSegmentManager.java | 86 +-
.../core/realtime/PinotRealtimeSegmentManager.java | 54 +-
.../core/realtime/SegmentCompletionManager.java | 159 +-
.../helix/core/realtime/TableConfigCache.java | 9 +-
.../segment/CommittingSegmentDescriptor.java | 3 +-
.../segment/FlushThresholdUpdateManager.java | 4 +-
.../SegmentSizeBasedFlushThresholdUpdater.java | 12 +-
.../rebalance/DefaultRebalanceSegmentStrategy.java | 16 +-
.../core/rebalance/RebalanceSegmentStrategy.java | 6 +-
.../ReplicaGroupRebalanceSegmentStrategy.java | 19 +-
.../core/relocation/RealtimeSegmentRelocator.java | 82 +-
.../helix/core/retention/RetentionManager.java | 102 +-
.../BalanceNumSegmentAssignmentStrategy.java | 8 +-
.../core/sharding/BucketizedSegmentStrategy.java | 8 +-
.../core/sharding/RandomAssignmentStrategy.java | 7 +-
.../ReplicaGroupSegmentAssignmentStrategy.java | 20 +-
.../core/sharding/SegmentAssignmentStrategy.java | 1 -
.../sharding/SegmentAssignmentStrategyEnum.java | 2 +-
.../sharding/SegmentAssignmentStrategyFactory.java | 1 -
.../helix/core/util/HelixSetupUtils.java | 29 +-
.../helix/core/util/ZKMetadataUtils.java | 2 +-
.../controller/util/AutoAddInvertedIndex.java | 16 +-
.../pinot/controller/util/TableSizeReader.java | 18 +-
.../BrokerResourceValidationManager.java | 53 +-
.../validation/OfflineSegmentIntervalChecker.java | 47 +-
.../RealtimeSegmentValidationManager.java | 68 +-
.../controller/validation/StorageQuotaChecker.java | 44 +-
.../src/main/resources/landing/index.html | 2 +-
.../src/main/resources/static/index.html | 2 +-
.../resources/static/js/lib/codemirror/ruby.js | 3 +
.../resources/webapp/js/lib/codemirror/ruby.js | 3 +
.../api/resources/AccessControlTest.java | 2 +-
.../api/resources/FileUploadPathProviderTest.java | 10 +-
.../api/resources/PinotFileUploadTest.java | 21 +-
.../PinotInstanceRestletResourceTest.java | 37 +-
.../resources/PinotSchemaRestletResourceTest.java | 22 +-
.../resources/PinotSegmentRestletResourceTest.java | 22 +-
.../resources/PinotTableRestletResourceTest.java | 75 +-
.../resources/PinotTenantRestletResourceTest.java | 30 +-
.../SegmentCompletionProtocolDeserTest.java | 40 +-
.../api/resources/ServerTableSizeReaderTest.java | 6 +-
.../api/resources/TableSizeReaderTest.java | 97 +-
.../controller/api/resources/TableViewsTest.java | 71 +-
.../helix/ControllerInstanceToggleTest.java | 27 +-
.../helix/ControllerPeriodicTaskStarterTest.java | 97 +
.../controller/helix/ControllerSentinelTestV2.java | 48 +-
.../controller/helix/ControllerTenantTest.java | 57 +-
.../pinot/controller/helix/ControllerTest.java | 53 +-
.../controller/helix/PinotResourceManagerTest.java | 26 +-
.../controller/helix/SegmentStatusCheckerTest.java | 270 +--
.../helix/core/PinotHelixResourceManagerTest.java | 61 +-
.../periodictask/ControllerPeriodicTaskTest.java | 172 +-
.../PinotLLCRealtimeSegmentManagerTest.java | 141 +-
.../helix/core/realtime/SegmentCompletionTest.java | 166 +-
.../segment/FlushThresholdUpdaterTest.java | 98 +-
.../rebalance/DefaultRebalanceStrategyTest.java | 99 +-
.../ReplicaGroupRebalanceStrategyTest.java | 70 +-
.../relocation/RealtimeSegmentRelocatorTest.java | 78 +-
.../helix/core/retention/RetentionManagerTest.java | 57 +-
.../sharding/SegmentAssignmentStrategyTest.java | 124 +-
.../core/util/SegmentDeletionManagerTest.java | 21 +-
.../helix/core/util/TableRebalancerTest.java | 1 +
.../controller/utils/ReplicaGroupTestUtils.java | 14 +-
.../controller/utils/SegmentMetadataMockUtils.java | 11 +-
.../validation/StorageQuotaCheckerTest.java | 34 +-
.../validation/ValidationManagerTest.java | 44 +-
pinot-core/pom.xml | 3 +-
.../org/apache/pinot/core/bloom/BloomFilter.java | 6 +-
.../apache/pinot/core/bloom/BloomFilterUtil.java | 4 +-
.../pinot/core/bloom/GuavaOnHeapBloomFilter.java | 14 +-
.../pinot/core/common/BlockDocIdValueSet.java | 1 -
.../pinot/core/common/BlockMultiValIterator.java | 2 -
.../org/apache/pinot/core/common/DataSource.java | 5 +-
.../apache/pinot/core/common/ObjectSerDeUtils.java | 3 +-
.../org/apache/pinot/core/common/Predicate.java | 37 +-
.../core/common/datatable/DataTableBuilder.java | 30 +-
.../core/common/datatable/DataTableFactory.java | 6 +-
.../core/common/datatable/DataTableImplV2.java | 21 +-
.../core/common/predicate/BaseInPredicate.java | 4 +-
.../pinot/core/common/predicate/EqPredicate.java | 4 +-
.../pinot/core/common/predicate/NEqPredicate.java | 4 +-
.../core/common/predicate/RangePredicate.java | 5 +-
.../core/common/predicate/RegexpLikePredicate.java | 9 +-
.../org/apache/pinot/core/data/GenericRow.java | 6 +-
.../data/function/FunctionExpressionEvaluator.java | 9 +-
.../pinot/core/data/function/FunctionInfo.java | 5 +-
.../pinot/core/data/function/FunctionInvoker.java | 13 +-
.../pinot/core/data/function/FunctionRegistry.java | 3 +-
.../core/data/manager/BaseTableDataManager.java | 6 +-
.../core/data/manager/InstanceDataManager.java | 15 +-
.../pinot/core/data/manager/TableDataManager.java | 6 +-
.../manager/offline/OfflineTableDataManager.java | 3 +-
.../realtime/HLRealtimeSegmentDataManager.java | 70 +-
.../realtime/LLRealtimeSegmentDataManager.java | 153 +-
.../manager/realtime/RealtimeTableDataManager.java | 8 +-
.../realtime/SegmentBuildTimeLeaseExtender.java | 15 +-
.../data/partition/PartitionFunctionFactory.java | 4 +-
.../pinot/core/data/readers/AvroRecordReader.java | 15 +-
.../pinot/core/data/readers/CSVRecordReader.java | 12 +-
.../apache/pinot/core/data/readers/FileFormat.java | 7 +-
.../pinot/core/data/readers/JSONRecordReader.java | 12 +-
.../readers/MultiplePinotSegmentRecordReader.java | 6 +-
.../data/readers/PinotSegmentRecordReader.java | 9 +-
.../pinot/core/data/readers/RecordReader.java | 9 +-
.../core/data/readers/RecordReaderFactory.java | 6 +-
.../pinot/core/data/readers/RecordReaderUtils.java | 11 +-
.../core/data/readers/ThriftRecordReader.java | 226 +--
.../data/readers/ThriftRecordReaderConfig.java | 23 +-
.../recordtransformer/CompoundTransformer.java | 4 +-
.../core/data/recordtransformer/PinotDataType.java | 40 +-
.../data/recordtransformer/TimeTransformer.java | 8 +-
.../generator/SegmentGeneratorConfig.java | 21 +-
.../immutable/ImmutableSegmentImpl.java | 4 +-
.../immutable/ImmutableSegmentLoader.java | 10 +-
.../indexsegment/mutable/MutableSegmentImpl.java | 60 +-
.../io/compression/ChunkCompressorFactory.java | 3 +-
.../reader/BaseSingleColumnMultiValueReader.java | 5 +-
.../reader/BaseSingleColumnSingleValueReader.java | 8 +-
.../apache/pinot/core/io/reader/ReaderContext.java | 1 -
.../impl/FixedByteSingleValueMultiColReader.java | 6 +-
.../reader/impl/v1/BaseChunkSingleValueReader.java | 3 +-
.../reader/impl/v1/FixedBitMultiValueReader.java | 3 +-
.../reader/impl/v1/FixedBitSingleValueReader.java | 3 +-
.../core/io/reader/impl/v1/SortedIndexReader.java | 3 +-
.../io/reader/impl/v1/SortedIndexReaderImpl.java | 4 +-
.../BaseSingleColumnMultiValueReaderWriter.java | 3 +-
.../BaseSingleColumnSingleValueReaderWriter.java | 3 +-
.../BaseSingleValueMultiColumnReaderWriter.java | 3 +-
.../RealtimeIndexOffHeapMemoryManager.java | 9 +-
...ixedByteSingleColumnMultiValueReaderWriter.java | 3 +-
...xedByteSingleColumnSingleValueReaderWriter.java | 6 +-
...ixedByteSingleValueMultiColumnReaderWriter.java | 3 +-
.../io/util/DictionaryDelegatingValueReader.java | 3 +-
.../core/io/util/FixedBitIntReaderWriter.java | 7 +-
.../core/io/util/FixedByteValueReaderWriter.java | 3 +-
.../apache/pinot/core/io/util/PinotDataBitSet.java | 11 +-
.../org/apache/pinot/core/io/util/ValueReader.java | 3 +-
.../pinot/core/io/writer/DataFileWriter.java | 1 +
.../io/writer/SingleColumnMultiValueWriter.java | 1 -
.../io/writer/SingleColumnSingleValueWriter.java | 1 -
.../impl/FixedByteSingleValueMultiColWriter.java | 6 +-
.../core/io/writer/impl/MmapMemoryManager.java | 5 +-
.../writer/impl/MutableOffHeapByteArrayStore.java | 6 +-
.../core/io/writer/impl/OffHeapStringStore.java | 6 +-
.../writer/impl/v1/BaseChunkSingleValueWriter.java | 6 +-
.../writer/impl/v1/FixedBitMultiValueWriter.java | 9 +-
.../writer/impl/v1/FixedBitSingleValueWriter.java | 32 +-
.../impl/v1/FixedByteChunkSingleValueWriter.java | 3 +-
.../impl/v1/VarByteChunkSingleValueWriter.java | 6 +-
.../core/metadata/DefaultMetadataExtractor.java | 4 +-
.../pinot/core/metadata/MetadataExtractor.java | 3 +-
.../core/metadata/MetadataExtractorFactory.java | 28 +-
.../pinot/core/minion/BackfillDateTimeColumn.java | 18 +-
.../pinot/core/minion/RawIndexConverter.java | 21 +-
.../apache/pinot/core/minion/SegmentConverter.java | 10 +-
.../apache/pinot/core/minion/SegmentPurger.java | 9 +-
.../minion/rollup/MergeRollupSegmentConverter.java | 36 +-
.../apache/pinot/core/minion/rollup/MergeType.java | 4 +-
.../rollup/aggregate/MaxValueAggregator.java | 1 +
.../core/minion/segment/MapperRecordReader.java | 8 +-
.../core/minion/segment/ReducerRecordReader.java | 3 +-
.../core/operator/CombineGroupByOperator.java | 4 +-
.../pinot/core/operator/CombineOperator.java | 17 +-
.../pinot/core/operator/ExecutionStatistics.java | 11 +-
.../operator/blocks/IntermediateResultsBlock.java | 3 +-
.../operator/dociditerators/AndDocIdIterator.java | 6 +-
.../dociditerators/BitmapDocIdIterator.java | 10 +-
.../dociditerators/IndexBasedDocIdIterator.java | 4 +-
.../RangelessBitmapDocIdIterator.java | 8 +-
.../dociditerators/SVScanDocIdIterator.java | 1 +
.../dociditerators/ScanBasedDocIdIterator.java | 1 +
.../dociditerators/SortedDocIdIterator.java | 4 +-
.../core/operator/docidsets/AndBlockDocIdSet.java | 2 +-
.../docidsets/ScanBasedMultiValueDocIdSet.java | 7 +-
.../docidsets/ScanBasedSingleValueDocIdSet.java | 6 +-
.../core/operator/docidsets/SortedDocIdSet.java | 2 +-
.../core/operator/docidsets/StarTreeDocIdSet.java | 1 +
.../SortedInvertedIndexBasedFilterOperator.java | 4 +-
.../predicate/PredicateEvaluatorProvider.java | 11 +-
.../predicate/RangePredicateEvaluatorFactory.java | 6 +-
.../DateTimeConversionTransformFunction.java | 20 +-
.../function/MultiplicationTransformFunction.java | 3 +-
.../datetime/BaseDateTimeTransformer.java | 33 +-
.../pinot/core/periodictask/BasePeriodicTask.java | 148 +-
.../pinot/core/periodictask/PeriodicTask.java | 29 +-
.../core/periodictask/PeriodicTaskScheduler.java | 9 +-
.../core/plan/AggregationGroupByPlanNode.java | 5 +-
.../pinot/core/plan/AggregationPlanNode.java | 8 +-
.../apache/pinot/core/plan/CombinePlanNode.java | 11 +-
.../plan/DictionaryBasedAggregationPlanNode.java | 2 +-
.../org/apache/pinot/core/plan/FilterPlanNode.java | 5 +-
.../core/plan/maker/InstancePlanMakerImplV2.java | 4 +-
.../DistinctCountHLLAggregationFunction.java | 6 +-
.../function/customobject/QuantileDigest.java | 8 +-
.../aggregation/groupby/GroupByResultHolder.java | 1 -
.../core/query/config/QueryExecutorConfig.java | 6 +-
.../core/query/config/QueryPlannerConfig.java | 6 +-
.../core/query/config/SegmentPrunerConfig.java | 7 +-
.../query/executor/ServerQueryExecutorV1Impl.java | 7 +-
.../core/query/pruner/AbstractSegmentPruner.java | 4 +-
.../core/query/pruner/PartitionSegmentPruner.java | 27 +-
.../core/query/reduce/BrokerReduceService.java | 32 +-
.../pinot/core/query/reduce/CombineService.java | 5 +-
.../query/reduce/HavingClauseComparisonTree.java | 9 +-
.../query/scheduler/MultiLevelPriorityQueue.java | 36 +-
.../core/query/scheduler/PriorityScheduler.java | 18 +-
.../pinot/core/query/scheduler/QueryScheduler.java | 16 +-
.../query/scheduler/QuerySchedulerFactory.java | 12 +-
.../pinot/core/query/scheduler/SchedulerGroup.java | 1 +
.../query/scheduler/SchedulerPriorityQueue.java | 9 +-
.../query/scheduler/SchedulerQueryContext.java | 10 +-
.../query/scheduler/fcfs/BoundedFCFSScheduler.java | 2 +-
.../resources/BoundedAccountingExecutor.java | 5 +-
.../scheduler/resources/QueryExecutorService.java | 10 +-
.../scheduler/resources/ResourceLimitPolicy.java | 21 +-
.../query/scheduler/resources/ResourceManager.java | 24 +-
.../tokenbucket/TokenPriorityScheduler.java | 3 +-
.../scheduler/tokenbucket/TokenSchedulerGroup.java | 14 +-
.../query/selection/SelectionOperatorUtils.java | 14 +-
.../comparator/CompositeDocIdValComparator.java | 38 +-
.../comparator/DocIdDoubleValComparator.java | 2 +-
.../comparator/DocIdFloatValComparator.java | 2 +-
.../comparator/DocIdIntValComparator.java | 4 +-
.../comparator/DocIdLongValComparator.java | 2 +-
.../DoubleArraySelectionColumnIterator.java | 1 +
.../iterator/DoubleSelectionColumnIterator.java | 1 +
.../FloatArraySelectionColumnIterator.java | 1 +
.../iterator/FloatSelectionColumnIterator.java | 1 +
.../iterator/IntArraySelectionColumnIterator.java | 1 +
.../iterator/IntSelectionColumnIterator.java | 1 +
.../iterator/LongArraySelectionColumnIterator.java | 1 +
.../iterator/LongSelectionColumnIterator.java | 1 +
.../iterator/SelectionColumnIterator.java | 2 +-
.../SelectionMultiValueColumnIterator.java | 1 +
.../StringArraySelectionColumnIterator.java | 1 +
.../iterator/StringSelectionColumnIterator.java | 6 +-
.../org/apache/pinot/core/query/utils/Pair.java | 1 -
.../apache/pinot/core/query/utils/TrieNode.java | 1 -
.../converter/RealtimeSegmentConverter.java | 11 +-
.../converter/stats/RealtimeColumnStatistics.java | 48 +-
.../stats/RealtimeNoDictionaryColStatistics.java | 11 +-
.../RealtimeSegmentSegmentCreationDataSource.java | 3 +-
.../stats/RealtimeSegmentStatsContainer.java | 3 +-
.../core/realtime/impl/RealtimeSegmentConfig.java | 6 +-
.../realtime/impl/RealtimeSegmentStatsHistory.java | 28 +-
.../dictionary/BaseOffHeapMutableDictionary.java | 10 +-
.../dictionary/BaseOnHeapMutableDictionary.java | 3 +-
.../dictionary/BytesOffHeapMutableDictionary.java | 3 +-
.../dictionary/DoubleOffHeapMutableDictionary.java | 3 +-
.../dictionary/FloatOffHeapMutableDictionary.java | 3 +-
.../dictionary/IntOffHeapMutableDictionary.java | 11 +-
.../dictionary/LongOffHeapMutableDictionary.java | 3 +-
.../impl/dictionary/MutableDictionaryFactory.java | 1 -
.../dictionary/StringOffHeapMutableDictionary.java | 7 +-
.../realtime/impl/kafka/ConsumerAndIterator.java | 6 +-
.../impl/kafka/KafkaAvroMessageDecoder.java | 31 +-
.../impl/kafka/KafkaConnectionHandler.java | 12 +-
.../realtime/impl/kafka/KafkaConsumerManager.java | 18 +-
.../impl/kafka/KafkaHighLevelStreamConfig.java | 26 +-
.../impl/kafka/KafkaJSONMessageDecoder.java | 3 +-
.../impl/kafka/KafkaLowLevelStreamConfig.java | 29 +-
.../impl/kafka/KafkaPartitionLevelConsumer.java | 10 +-
.../impl/kafka/KafkaStreamLevelConsumer.java | 19 +-
.../impl/kafka/KafkaStreamMetadataProvider.java | 3 +-
.../pinot/core/realtime/stream/MessageBatch.java | 58 +-
.../pinot/core/realtime/stream/OffsetCriteria.java | 17 +-
.../realtime/stream/PartitionCountFetcher.java | 4 +-
.../realtime/stream/PartitionOffsetFetcher.java | 8 +-
.../pinot/core/realtime/stream/StreamConfig.java | 42 +-
.../realtime/stream/StreamConfigProperties.java | 2 -
.../realtime/stream/StreamConsumerFactory.java | 1 -
.../stream/StreamConsumerFactoryProvider.java | 1 -
.../core/realtime/stream/StreamLevelConsumer.java | 6 +-
.../core/realtime/stream/StreamMessageDecoder.java | 3 +-
.../segment/creator/ColumnIndexCreationInfo.java | 12 +-
.../core/segment/creator/ColumnStatistics.java | 128 +-
.../core/segment/creator/InvertedIndexCreator.java | 3 +-
.../RecordReaderSegmentCreationDataSource.java | 3 +-
.../pinot/core/segment/creator/SegmentCreator.java | 6 +-
.../creator/SegmentIndexCreationDriver.java | 11 +-
.../creator/SegmentPreIndexStatsCollector.java | 10 +-
.../creator/SegmentPreIndexStatsContainer.java | 3 +-
.../creator/impl/SegmentColumnarIndexCreator.java | 41 +-
.../creator/impl/SegmentDictionaryCreator.java | 46 +-
.../impl/SegmentIndexCreationDriverImpl.java | 51 +-
.../creator/impl/bloom/BloomFilterCreator.java | 8 +-
.../fwd/MultiValueUnsortedForwardIndexCreator.java | 6 +-
.../fwd/SingleValueFixedByteRawIndexCreator.java | 3 +-
.../fwd/SingleValueSortedForwardIndexCreator.java | 6 +-
.../SingleValueUnsortedForwardIndexCreator.java | 3 +-
.../fwd/SingleValueVarByteRawIndexCreator.java | 6 +-
.../inv/OffHeapBitmapInvertedIndexCreator.java | 15 +-
.../impl/inv/OnHeapBitmapInvertedIndexCreator.java | 3 +-
.../stats/AbstractColumnStatisticsCollector.java | 39 +-
.../stats/IntColumnPreIndexStatsCollector.java | 1 -
.../stats/SegmentPreIndexStatsCollectorImpl.java | 26 +-
.../stats/StringColumnPreIndexStatsCollector.java | 1 -
.../pinot/core/segment/index/ColumnMetadata.java | 44 +-
.../core/segment/index/SegmentMetadataImpl.java | 2 +-
.../segment/index/column/ColumnIndexContainer.java | 2 +-
.../index/column/PhysicalColumnIndexContainer.java | 15 +-
.../index/converter/SegmentFormatConverter.java | 5 +-
.../converter/SegmentV1V2ToV3FormatConverter.java | 44 +-
.../segment/index/loader/IndexLoadingConfig.java | 1 -
.../core/segment/index/loader/LoaderUtils.java | 10 +-
.../segment/index/loader/SegmentPreProcessor.java | 9 +-
.../loader/bloomfilter/BloomFilterHandler.java | 12 +-
.../ColumnMinMaxValueGenerator.java | 29 +-
.../ColumnMinMaxValueGeneratorMode.java | 5 +-
.../defaultcolumn/BaseDefaultColumnHandler.java | 37 +-
.../defaultcolumn/DefaultColumnStatistics.java | 32 +-
.../defaultcolumn/V1DefaultColumnHandler.java | 3 +-
.../defaultcolumn/V3DefaultColumnHandler.java | 3 +-
.../loader/invertedindex/InvertedIndexHandler.java | 80 +-
.../index/readers/BitmapInvertedIndexReader.java | 9 +-
.../segment/index/readers/BloomFilterReader.java | 3 +-
.../core/segment/index/readers/Dictionary.java | 1 -
.../index/readers/ImmutableDictionaryReader.java | 3 +-
.../index/readers/OnHeapDoubleDictionary.java | 4 +-
.../index/readers/OnHeapFloatDictionary.java | 4 +-
.../segment/index/readers/OnHeapIntDictionary.java | 4 +-
.../index/readers/OnHeapLongDictionary.java | 4 +-
.../core/segment/memory/BasePinotLBuffer.java | 6 +-
.../pinot/core/segment/memory/PinotByteBuffer.java | 6 +-
.../pinot/core/segment/memory/PinotDataBuffer.java | 35 +-
.../segment/memory/PinotNativeOrderLBuffer.java | 6 +-
.../segment/memory/PinotNonNativeOrderLBuffer.java | 6 +-
.../segment/name/DefaultSegmentNameGenerator.java | 140 --
.../segment/name/FixedSegmentNameGenerator.java | 36 +-
.../name/NormalizedDateSegmentNameGenerator.java | 189 +-
.../core/segment/name/SegmentNameGenerator.java | 17 +-
.../segment/name/SimpleSegmentNameGenerator.java | 59 +
.../core/segment/store/ColumnIndexDirectory.java | 11 +-
.../pinot/core/segment/store/ColumnIndexType.java | 1 +
.../core/segment/store/FilePerIndexDirectory.java | 16 +-
.../pinot/core/segment/store/IndexEntry.java | 1 +
.../apache/pinot/core/segment/store/IndexKey.java | 1 -
.../pinot/core/segment/store/SegmentDirectory.java | 9 +-
.../segment/store/SegmentLocalFSDirectory.java | 30 +-
.../segment/store/SingleFileIndexDirectory.java | 63 +-
.../virtualcolumn/BaseVirtualColumnProvider.java | 7 +-
.../virtualcolumn/DocIdVirtualColumnProvider.java | 17 +-
.../IntSingleValueDataFileReader.java | 3 +-
.../SingleStringVirtualColumnProvider.java | 17 +-
.../virtualcolumn/VirtualColumnProvider.java | 4 +
.../VirtualColumnProviderFactory.java | 6 +-
.../pinot/core/startree/OffHeapStarTree.java | 18 +-
.../core/startree/OffHeapStarTreeBuilder.java | 77 +-
.../pinot/core/startree/StarTreeBuilder.java | 15 +-
.../pinot/core/startree/StarTreeBuilderUtils.java | 5 +-
.../pinot/core/startree/StarTreeDataTable.java | 3 +-
.../apache/pinot/core/startree/hll/HllUtil.java | 4 +-
.../startree/v2/builder/BaseSingleTreeBuilder.java | 48 +-
.../startree/v2/builder/MultipleTreesBuilder.java | 16 +-
.../v2/builder/OffHeapSingleTreeBuilder.java | 42 +-
.../startree/v2/builder/SingleTreeBuilder.java | 3 +-
.../startree/v2/builder/StarTreeIndexCombiner.java | 9 +-
.../v2/builder/StarTreeV2BuilderConfig.java | 4 +-
.../startree/v2/store/StarTreeIndexContainer.java | 19 +-
.../pinot/core/transport/AsyncQueryResponse.java | 3 +-
.../pinot/core/transport/DataTableHandler.java | 4 +-
.../pinot/core/transport/ServerChannels.java | 16 +-
.../pinot/core/transport/ServerResponse.java | 4 +-
.../java/org/apache/pinot/core/util/AvroUtils.java | 12 +-
.../java/org/apache/pinot/core/util/CrcUtils.java | 6 +-
.../pinot/core/util/FixedIntArrayOffHeapIdMap.java | 3 +-
.../java/org/apache/pinot/core/util/IdMap.java | 2 -
.../java/org/apache/pinot/core/util/SizeUtil.java | 3 +-
.../pinot/core/util/SortedRangeIntersection.java | 1 +
.../pinot/core/util/trace/TraceCallable.java | 6 +-
.../apache/pinot/core/util/trace/TraceContext.java | 9 +-
.../ServerSegmentCompletionProtocolHandler.java | 6 +-
.../apache/pinot/core/common/DataFetcherTest.java | 7 +-
.../core/common/RealtimeNoDictionaryTest.java | 6 +-
.../core/common/datatable/DataTableSerDeTest.java | 13 +-
.../core/common/docidsets/BitmapDocIdSetTest.java | 3 +-
.../core/common/docidsets/SortedDocIdSetTest.java | 5 +-
.../function/FunctionExpressionEvaluatorTest.java | 17 +-
.../data/manager/BaseTableDataManagerTest.java | 16 +-
.../realtime/LLRealtimeSegmentDataManagerTest.java | 187 +-
.../readers/BackfillDateTimeRecordReaderTest.java | 16 +-
.../core/data/readers/CSVRecordReaderTest.java | 12 +-
.../core/data/readers/JSONRecordReaderTest.java | 9 +-
.../MultiplePinotSegmentRecordReaderTest.java | 9 +-
.../data/readers/PinotSegmentRecordReaderTest.java | 12 +-
.../pinot/core/data/readers/RecordReaderTest.java | 6 +-
.../core/data/readers/ThriftRecordReaderTest.java | 12 +-
.../pinot/core/data/readers/ThriftSampleData.java | 2003 ++++++++++----------
.../data/recordtransformer/PinotDataTypeTest.java | 21 +-
.../recordtransformer/RecordTransformerTest.java | 6 +-
.../MutableSegmentImplAggregateMetricsTest.java | 8 +-
.../mutable/MutableSegmentImplTest.java | 8 +-
.../mutable/MutableSegmentImplTestUtils.java | 18 +-
.../pinot/core/io/util/PinotDataBitSetTest.java | 19 +-
.../impl/MmapMemoryManagerFileCleanupTest.java | 3 +-
.../core/io/writer/impl/MmapMemoryManagerTest.java | 9 +-
.../impl/MutableOffHeapByteArrayStoreTest.java | 9 +-
.../io/writer/impl/OffHeapStringStoreTest.java | 9 +-
.../metadata/MetadataExtractorFactoryTest.java | 4 +-
.../minion/MergeRollupSegmentConverterTest.java | 34 +-
.../pinot/core/minion/SegmentConverterTest.java | 58 +-
.../pinot/core/minion/SegmentPurgerTest.java | 9 +-
.../dociditerators/BitmapDocIdIteratorTest.java | 9 +-
.../RealtimeSingleValueIteratorTest.java | 15 +-
.../operator/filter/FilterOperatorUtilsTest.java | 30 +-
.../function/AdditionTransformFunctionTest.java | 16 +-
.../function/BaseTransformFunctionTest.java | 3 +-
.../DateTimeConversionTransformFunctionTest.java | 14 +-
.../function/DivisionTransformFunctionTest.java | 18 +-
.../MultiplicationTransformFunctionTest.java | 14 +-
.../function/SubtractionTransformFunctionTest.java | 18 +-
.../TimeConversionTransformFunctionTest.java | 20 +-
.../function/ValueInTransformFunctionTest.java | 13 +-
.../datetime/DateTimeConverterTest.java | 28 +-
.../periodictask/PeriodicTaskSchedulerTest.java | 40 +-
.../pinot/core/plan/CombinePlanNodeTest.java | 4 +-
...adataAndDictionaryAggregationPlanMakerTest.java | 232 +--
.../NoDictionaryInPredicateEvaluatorTest.java | 15 +-
.../NoDictionaryRangePredicateEvaluatorTest.java | 6 +-
.../scheduler/MultiLevelPriorityQueueTest.java | 28 +-
.../query/scheduler/PrioritySchedulerTest.java | 46 +-
.../query/scheduler/TestSchedulerGroupFactory.java | 1 +
.../resources/BoundedAccountingExecutorTest.java | 3 +-
.../scheduler/resources/ResourceManagerTest.java | 4 +-
.../tokenbucket/TokenSchedulerGroupTest.java | 15 +-
.../impl/RealtimeSegmentStatsHistoryTest.java | 36 +-
.../impl/dictionary/DictionaryPerfRunner.java | 22 +-
.../impl/dictionary/MultiValueDictionaryTest.java | 6 +-
.../impl/dictionary/MutableDictionaryTest.java | 19 +-
.../kafka/AvroRecordToPinotRowGeneratorTest.java | 6 +-
.../impl/kafka/KafkaLowLevelStreamConfigTest.java | 50 +-
.../kafka/KafkaPartitionLevelConsumerTest.java | 78 +-
.../ConsumingSegmentAssignmentStrategyTest.java | 30 +-
.../core/realtime/stream/StreamConfigTest.java | 222 ++-
.../core/segment/index/ColumnMetadataTest.java | 29 +-
.../segment/index/SegmentMetadataImplTest.java | 12 +-
.../SegmentV1V2ToV3FormatConverterTest.java | 6 +-
.../index/creator/BloomFilterCreatorTest.java | 25 +-
.../segment/index/creator/RawIndexCreatorTest.java | 31 +-
.../SegmentGenerationWithBytesTypeTest.java | 22 +-
.../SegmentGenerationWithTimeColumnTest.java | 9 +-
.../index/creator/SegmentPartitionTest.java | 107 +-
.../core/segment/index/loader/LoaderTest.java | 34 +-
.../core/segment/index/loader/LoaderUtilsTest.java | 2 +-
.../index/loader/SegmentPreProcessorTest.java | 30 +-
.../readers/ImmutableDictionaryReaderTest.java | 55 +-
.../core/segment/memory/PinotDataBufferTest.java | 132 +-
.../name/DefaultSegmentNameGeneratorTest.java | 155 --
.../NormalizedDateSegmentNameGeneratorTest.java | 189 +-
.../name/SimpleSegmentNameGeneratorTest.java | 58 +
.../store/ColumnIndexDirectoryTestHelper.java | 56 +-
.../segment/store/FilePerIndexDirectoryTest.java | 26 +-
.../segment/store/SegmentDirectoryPathsTest.java | 3 +-
.../segment/store/SegmentLocalFSDirectoryTest.java | 20 +-
.../store/SingleFileIndexDirectoryTest.java | 43 +-
.../pinot/core/startree/BaseStarTreeIndexTest.java | 21 +-
.../core/startree/OffHeapStarTreeBuilderTest.java | 6 +-
.../pinot/core/startree/StarTreeDataTableTest.java | 3 +-
.../startree/StarTreeIndexTestSegmentHelper.java | 6 +-
.../pinot/core/startree/SumStarTreeIndexTest.java | 25 +-
.../pinot/core/startree/TestStarTreeMetadata.java | 7 +-
.../core/startree/hll/HllIndexCreationTest.java | 10 +-
.../core/startree/hll/HllStarTreeIndexTest.java | 33 +-
.../core/startree/hll/HllTypeConversionTest.java | 4 +-
.../OffHeapStarTreeBuilderWithHllFieldTest.java | 15 +-
.../hll/SegmentWithHllIndexCreateHelper.java | 21 +-
.../pinot/core/startree/v2/BaseStarTreeV2Test.java | 26 +-
.../v2/PreAggregatedMinMaxRangeStarTreeV2Test.java | 4 +-
.../apache/pinot/core/transport/DummyServer.java | 14 +-
.../pinot/core/transport/QueryRouterTest.java | 17 +-
.../org/apache/pinot/core/util/CrcUtilsTest.java | 12 +-
.../pinot/core/util/trace/TraceContextTest.java | 14 +-
.../pinot/index/persist/AvroDataPublisherTest.java | 20 +-
.../pinot/index/reader/FileReaderTestUtils.java | 5 +-
.../pinot/index/reader/FixedBitMultiValueTest.java | 57 +-
.../FixedByteWidthRowColDataFileReaderTest.java | 6 +-
.../index/reader/SortedForwardIndexReaderTest.java | 9 +-
.../FixedByteChunkSingleValueReaderWriteTest.java | 24 +-
...ByteSingleColumnMultiValueReaderWriterTest.java | 21 +-
...yteSingleColumnSingleValueReaderWriterTest.java | 15 +-
...ByteSingleValueMultiColumnReaderWriterTest.java | 3 +-
.../VarByteChunkSingleValueReaderWriteTest.java | 9 +-
.../FixedByteWidthRowColDataFileWriterTest.java | 58 +-
.../pinot/queries/BaseMultiValueQueriesTest.java | 22 +-
.../pinot/queries/BaseSingleValueQueriesTest.java | 33 +-
.../apache/pinot/queries/FastHllQueriesTest.java | 55 +-
...nerSegmentAggregationMultiValueQueriesTest.java | 100 +-
...erSegmentAggregationSingleValueQueriesTest.java | 74 +-
...terSegmentAggregationMultiValueQueriesTest.java | 108 +-
...erSegmentAggregationSingleValueQueriesTest.java | 16 +-
.../queries/PercentileTDigestMVQueriesTest.java | 8 +-
.../queries/PercentileTDigestQueriesTest.java | 16 +-
.../DefaultAggregationExecutorTest.java | 6 +-
.../AggregationGroupByTrimmingServiceTest.java | 5 +-
.../DictionaryBasedGroupKeyGeneratorTest.java | 7 +-
.../groupby/NoDictionaryGroupKeyGeneratorTest.java | 24 +-
.../pinot/query/executor/QueryExecutorTest.java | 9 +-
.../query/pruner/ColumnValueSegmentPrunerTest.java | 17 +-
.../selection/SelectionOperatorServiceTest.java | 39 +-
.../converter/RealtimeSegmentConverterTest.java | 9 +-
.../pinot/reduce/HavingClauseComparisonTests.java | 40 +-
.../v1/creator/BitmapInvertedIndexCreatorTest.java | 15 +-
.../v1/creator/BitmapInvertedIndexTest.java | 13 +-
.../pinot/segments/v1/creator/BlocksTest.java | 13 +-
.../segments/v1/creator/DictionariesTest.java | 28 +-
.../v1/creator/FixedIntArrayIdMapTest.java | 5 +-
.../pinot/segments/v1/creator/IntArraysTest.java | 12 +-
.../segments/v1/creator/SegmentTestUtils.java | 12 +-
.../realtime/ControllerLeaderLocatorTest.java | 3 +-
.../test/java/org/apache/pinot/util/TestUtils.java | 12 +-
pinot-distribution/pinot-assembly.xml | 29 +-
pinot-distribution/pinot-source-assembly.xml | 51 +
pinot-distribution/pom.xml | 108 +-
pinot-filesystem/pom.xml | 3 +-
.../org/apache/pinot/filesystem/LocalPinotFS.java | 39 +-
.../java/org/apache/pinot/filesystem/PinotFS.java | 109 +-
.../apache/pinot/filesystem/PinotFSFactory.java | 3 +-
.../apache/pinot/filesystem/LocalPinotFSTest.java | 22 +-
.../pinot/filesystem/PinotFSFactoryTest.java | 30 +-
pinot-hadoop-filesystem/pom.xml | 5 +-
.../org/apache/pinot/filesystem/HadoopPinotFS.java | 35 +-
pinot-hadoop/README.md | 20 +
pinot-hadoop/pom.xml | 35 +-
.../pinot/hadoop/PinotHadoopJobLauncher.java | 36 +-
.../org/apache/pinot/hadoop/io/FileHandler.java | 234 +--
.../pinot/hadoop/io/JsonPinotOutputFormat.java | 3 +-
.../apache/pinot/hadoop/io/PinotOutputFormat.java | 336 ++--
.../org/apache/pinot/hadoop/io/PinotRecord.java | 80 +-
.../pinot/hadoop/io/PinotRecordSerialization.java | 32 +-
.../apache/pinot/hadoop/io/PinotRecordWriter.java | 146 +-
.../apache/pinot/hadoop/job/BaseSegmentJob.java | 84 +
.../apache/pinot/hadoop/job/ControllerRestApi.java | 82 +-
.../pinot/hadoop/job/DefaultControllerRestApi.java | 142 ++
.../pinot/hadoop/job/JobConfigConstants.java | 25 +-
.../pinot/hadoop/job/SegmentCreationJob.java | 432 ++---
.../apache/pinot/hadoop/job/SegmentTarPushJob.java | 85 +-
.../apache/pinot/hadoop/job/SegmentUriPushJob.java | 97 +-
.../mapper/HadoopSegmentCreationMapReduceJob.java | 328 ----
.../hadoop/job/mapper/SegmentCreationMapper.java | 319 ++++
.../apache/pinot/hadoop/utils/PushLocation.java | 40 +-
.../pinot/hadoop/io/PinotOutputFormatTest.java | 264 ++-
pinot-integration-tests/pom.xml | 3 +-
.../tests/AdminConsoleIntegrationTest.java | 96 +
...umSegmentAssignmentStrategyIntegrationTest.java | 31 +-
.../tests/BaseClusterIntegrationTest.java | 19 +-
.../tests/BaseClusterIntegrationTestSet.java | 44 +-
.../tests/ChaosMonkeyIntegrationTest.java | 65 +-
.../tests/ClusterIntegrationTestUtils.java | 20 +-
.../pinot/integration/tests/ClusterTest.java | 157 +-
...vertToRawIndexMinionClusterIntegrationTest.java | 21 +-
.../DeleteAPIHybridClusterIntegrationTest.java | 63 +-
...lakyConsumerRealtimeClusterIntegrationTest.java | 9 +-
...mentBuildPushOfflineClusterIntegrationTest.java | 25 +-
.../tests/HybridClusterIntegrationTest.java | 50 +-
...ridClusterIntegrationTestCommandLineRunner.java | 30 +-
.../tests/LLCRealtimeClusterIntegrationTest.java | 9 +-
...onaryAggregationPlanClusterIntegrationTest.java | 23 +-
.../MultiNodesOfflineClusterIntegrationTest.java | 18 +-
.../tests/NewConfigApplyIntegrationTest.java | 40 +-
.../tests/OfflineClusterIntegrationTest.java | 48 +-
.../tests/PinotURIUploadIntegrationTest.java | 46 +-
.../pinot/integration/tests/QueryGenerator.java | 15 +-
.../tests/RealtimeClusterIntegrationTest.java | 32 +-
.../tests/SegmentCompletionIntegrationTests.java | 45 +-
.../tests/SimpleMinionClusterIntegrationTest.java | 9 +-
.../tests/StarTreeClusterIntegrationTest.java | 26 +-
.../tests/StarTreeV2ClusterIntegrationTest.java | 16 +-
.../tests/UploadRefreshDeleteIntegrationTest.java | 37 +-
.../tasks/SegmentStatusCheckerIntegrationTest.java | 288 +++
pinot-minion/pom.xml | 3 +-
.../org/apache/pinot/minion/MinionStarter.java | 19 +-
.../BaseMultipleSegmentsConversionExecutor.java | 21 +-
.../BaseSingleSegmentConversionExecutor.java | 18 +-
.../executor/ConvertToRawIndexTaskExecutor.java | 10 +-
.../pinot/minion/executor/PinotTaskExecutor.java | 3 +-
.../pinot/minion/executor/PurgeTaskExecutor.java | 13 +-
.../minion/executor/SegmentConversionUtils.java | 6 +-
.../minion/executor/PurgeTaskExecutorTest.java | 14 +-
pinot-perf/README.md | 41 +
pinot-perf/pom.xml | 72 +-
.../org/apache/pinot/perf/BenchmarkDictionary.java | 37 +-
.../pinot/perf/BenchmarkDictionaryCreation.java | 34 +-
.../perf/BenchmarkFixedIntArrayOffHeapIdMap.java | 13 +-
.../pinot/perf/BenchmarkOfflineIndexReader.java | 19 +-
.../pinot/perf/BenchmarkOrDocIdIterator.java | 13 +-
.../apache/pinot/perf/BenchmarkQueryEngine.java | 17 +-
.../perf/BenchmarkRealtimeConsumptionSpeed.java | 12 +-
.../pinot/perf/BenchmarkStringDictionary.java | 12 +-
.../org/apache/pinot/perf/DictionaryDumper.java | 3 +-
.../pinot/perf/ForwardIndexWriterBenchmark.java | 15 +-
.../org/apache/pinot/perf/RawIndexBenchmark.java | 12 +-
.../org/apache/pinot/perf/RealtimeStressTest.java | 22 +-
.../pinot/perf/StringDictionaryPerfTest.java | 9 +-
pinot-server/pom.xml | 17 +-
.../server/api/resources/HealthCheckResource.java | 7 +-
.../server/api/resources/SchedulerResource.java | 1 -
.../server/api/resources/TableSizeResource.java | 12 +-
.../pinot/server/api/resources/TablesResource.java | 23 +-
.../pinot/server/conf/NettyServerConfig.java | 3 +-
.../org/apache/pinot/server/conf/ServerConf.java | 3 +-
.../server/request/ScheduledRequestHandler.java | 3 +-
.../apache/pinot/server/starter/ServerBuilder.java | 6 +-
.../server/starter/helix/AdminApiApplication.java | 23 +-
.../helix/DefaultHelixStarterServerConfig.java | 8 +-
.../starter/helix/HelixInstanceDataManager.java | 25 +-
.../helix/HelixInstanceDataManagerConfig.java | 11 +-
.../server/starter/helix/HelixServerStarter.java | 65 +-
.../starter/helix/SegmentFetcherAndLoader.java | 18 +-
.../helix/SegmentMessageHandlerFactory.java | 13 +-
.../SegmentOnlineOfflineStateModelFactory.java | 6 +-
.../server/api/resources/BaseResourceTest.java | 19 +-
.../server/api/resources/TablesResourceTest.java | 26 +-
.../realtime/RealtimeTableDataManagerTest.java | 24 +-
.../request/ScheduledRequestHandlerTest.java | 13 +-
pinot-tools/pom.xml | 9 +-
.../apache/pinot/tools/AbstractBaseCommand.java | 7 +-
.../pinot/tools/AutoAddInvertedIndexTool.java | 34 +-
.../apache/pinot/tools/ClusterStateVerifier.java | 10 +-
.../main/java/org/apache/pinot/tools/Command.java | 5 +-
.../org/apache/pinot/tools/HybridQuickstart.java | 4 +-
.../apache/pinot/tools/PinotIdealstateChanger.java | 13 +-
.../apache/pinot/tools/PinotNumReplicaChanger.java | 9 +-
.../apache/pinot/tools/PinotSegmentRebalancer.java | 30 +-
.../org/apache/pinot/tools/PinotToolLauncher.java | 17 +-
.../org/apache/pinot/tools/PinotZKChanger.java | 16 +-
.../java/org/apache/pinot/tools/Quickstart.java | 5 +-
.../org/apache/pinot/tools/RealtimeQuickStart.java | 4 +-
.../org/apache/pinot/tools/SegmentDumpTool.java | 6 +-
.../java/org/apache/pinot/tools/SpeedTest.java | 8 +-
.../apache/pinot/tools/StarTreeIndexViewer.java | 9 +-
.../org/apache/pinot/tools/UpdateSegmentState.java | 13 +-
.../apache/pinot/tools/ValidateTableRetention.java | 10 +-
.../pinot/tools/admin/PinotAdministrator.java | 42 +-
.../org/apache/pinot/tools/admin/PinotBroker.java | 3 +-
.../apache/pinot/tools/admin/PinotController.java | 3 +-
.../org/apache/pinot/tools/admin/PinotServer.java | 3 +-
.../admin/command/AbstractBaseAdminCommand.java | 9 +-
.../tools/admin/command/AddSchemaCommand.java | 3 +-
.../pinot/tools/admin/command/AddTableCommand.java | 21 +-
.../tools/admin/command/AddTenantCommand.java | 28 +-
.../admin/command/ApplyTableConfigCommand.java | 19 +-
.../admin/command/AvroSchemaToPinotSchema.java | 5 +-
.../command/BackfillDateTimeColumnCommand.java | 30 +-
.../admin/command/ChangeNumReplicasCommand.java | 6 +-
.../tools/admin/command/ChangeTableState.java | 36 +-
.../tools/admin/command/CreateSegmentCommand.java | 8 +-
.../tools/admin/command/DeleteClusterCommand.java | 7 +-
.../tools/admin/command/GenerateDataCommand.java | 29 +-
.../tools/admin/command/MoveReplicaGroup.java | 70 +-
.../tools/admin/command/PostQueryCommand.java | 9 +-
.../tools/admin/command/QuickstartRunner.java | 51 +-
.../command/RealtimeProvisioningHelperCommand.java | 16 +-
.../tools/admin/command/RebalanceTableCommand.java | 11 +-
.../admin/command/ShowClusterInfoCommand.java | 14 +-
.../tools/admin/command/StartBrokerCommand.java | 12 +-
.../admin/command/StartControllerCommand.java | 16 +-
.../tools/admin/command/StartKafkaCommand.java | 12 +-
.../tools/admin/command/StartServerCommand.java | 15 +-
.../tools/admin/command/StartZookeeperCommand.java | 19 +-
.../tools/admin/command/StopProcessCommand.java | 9 +-
.../admin/command/StreamAvroIntoKafkaCommand.java | 30 +-
.../tools/admin/command/UploadSegmentCommand.java | 14 +-
.../tools/admin/command/ValidateConfigCommand.java | 9 +-
.../admin/command/VerifyClusterStateCommand.java | 6 +-
.../tools/admin/command/VerifySegmentState.java | 7 +-
.../pinot/tools/backfill/BackfillSegmentUtils.java | 11 +-
.../pinot/tools/data/generator/AvroWriter.java | 9 +-
.../pinot/tools/data/generator/DataGenerator.java | 25 +-
.../tools/data/generator/DataGeneratorSpec.java | 5 +-
.../pinot/tools/data/generator/Generator.java | 1 +
.../tools/data/generator/GeneratorFactory.java | 1 -
.../tools/data/generator/NumberGenerator.java | 1 -
.../tools/data/generator/RangeDoubleGenerator.java | 3 +-
.../tools/data/generator/RangeFloatGenerator.java | 3 +-
.../tools/data/generator/RangeIntGenerator.java | 3 +-
.../tools/data/generator/RangeLongGenerator.java | 3 +-
.../pinot/tools/perf/PerfBenchmarkDriver.java | 41 +-
.../pinot/tools/perf/PerfBenchmarkDriverConf.java | 4 +-
.../pinot/tools/perf/PerfBenchmarkRunner.java | 27 +-
.../org/apache/pinot/tools/perf/QueryRunner.java | 62 +-
.../apache/pinot/tools/perf/ZookeeperLauncher.java | 3 +-
.../tools/query/comparison/ClusterStarter.java | 13 +-
.../tools/query/comparison/QueryComparison.java | 25 +-
.../query/comparison/QueryComparisonConfig.java | 10 +-
.../query/comparison/StarQueryComparison.java | 6 +-
.../query/comparison/StarTreeQueryGenerator.java | 3 +-
.../realtime/provisioning/MemoryEstimator.java | 39 +-
.../apache/pinot/tools/scan/query/Aggregation.java | 8 +-
.../apache/pinot/tools/scan/query/AvgFunction.java | 4 +-
.../pinot/tools/scan/query/CountFunction.java | 1 +
.../tools/scan/query/EqualsPredicateFilter.java | 1 +
.../apache/pinot/tools/scan/query/MinFunction.java | 1 +
.../tools/scan/query/MinMaxRangeFunction.java | 1 +
.../tools/scan/query/NotInPredicateFilter.java | 1 +
.../pinot/tools/scan/query/NotPredicateFilter.java | 3 +-
.../pinot/tools/scan/query/PredicateFilter.java | 3 +-
.../pinot/tools/scan/query/QueryResponse.java | 4 +-
.../apache/pinot/tools/scan/query/ResultTable.java | 9 +-
.../tools/scan/query/ScanBasedQueryProcessor.java | 6 +-
.../tools/scan/query/SegmentQueryProcessor.java | 9 +-
.../apache/pinot/tools/scan/query/SumFunction.java | 1 +
.../org/apache/pinot/tools/scan/query/Utils.java | 1 +
.../converter/ColumnarToStarTreeConverter.java | 22 +-
.../converter/DictionaryToRawIndexConverter.java | 9 +-
.../converter/PinotSegmentConvertCommand.java | 15 +-
.../converter/PinotSegmentToAvroConverter.java | 3 +-
.../converter/PinotSegmentToCsvConverter.java | 3 +-
.../converter/PinotSegmentToJsonConverter.java | 3 +-
.../segment/converter/SegmentMergeCommand.java | 26 +-
.../pinot/tools/streams/AirlineDataStream.java | 10 +-
.../pinot/tools/streams/MeetupRsvpStream.java | 3 +-
pinot-transport/pom.xml | 9 +-
.../common/AbstractCompositeListenableFuture.java | 12 +-
.../transport/common/AsyncResponseFuture.java | 18 +-
.../pinot/transport/common/CompositeFuture.java | 30 +-
.../pinot/transport/common/LinkedDequeue.java | 7 +-
.../pinot/transport/common/SelectingFuture.java | 17 +-
.../transport/common/ServerResponseFuture.java | 3 +-
.../pinot/transport/conf/TransportClientConf.java | 7 +-
.../transport/config/ConnectionPoolConfig.java | 18 +-
.../pinot/transport/config/RoutingTableConfig.java | 1 -
.../transport/metrics/AggregatedPoolStats.java | 10 +-
.../transport/metrics/NettyClientMetrics.java | 7 +-
.../transport/metrics/NettyServerMetrics.java | 3 +-
.../apache/pinot/transport/metrics/PoolStats.java | 1 -
.../transport/metrics/TransportServerMetrics.java | 1 -
.../transport/netty/NettyClientConnection.java | 13 +-
.../apache/pinot/transport/netty/NettyServer.java | 20 +-
.../transport/netty/NettyTCPClientConnection.java | 76 +-
.../pinot/transport/netty/NettyTCPServer.java | 13 +-
.../netty/PooledNettyClientResourceManager.java | 7 +-
.../org/apache/pinot/transport/pool/AsyncPool.java | 1 -
.../apache/pinot/transport/pool/AsyncPoolImpl.java | 18 +-
.../pool/AsyncPoolResourceManagerAdapter.java | 6 +-
.../apache/pinot/transport/pool/KeyedPoolImpl.java | 10 +-
.../transport/pool/PooledResourceManager.java | 1 -
.../transport/scattergather/ScatterGatherImpl.java | 3 +-
.../src/test/java/org/apache/pinot/Checkable.java | 3 +-
.../src/test/java/org/apache/pinot/TestUtils.java | 3 +-
.../transport/common/CompositeFutureTest.java | 30 +-
.../pinot/transport/common/ResponseFutureTest.java | 19 +-
.../transport/common/SelectingFutureTest.java | 18 +-
.../pinot/transport/common/ServerInstanceTest.java | 3 +-
.../transport/netty/NettyCloseChannelTest.java | 10 +-
.../NettySingleConnectionIntegrationTest.java | 12 +-
.../transport/perf/ScatterGatherPerfClient.java | 21 +-
.../transport/perf/ScatterGatherPerfServer.java | 26 +-
.../transport/perf/ScatterGatherPerfTester.java | 132 +-
.../pinot/transport/pool/KeyedPoolImplTest.java | 57 +-
.../transport/scattergather/ScatterGatherTest.java | 9 +-
pom.xml | 201 +-
thirdeye/pom.xml | 20 +-
thirdeye/thirdeye-frontend/app/models/anomalies.js | 3 +-
.../app/pods/application/controller.js | 4 +-
.../app/pods/application/route.js | 11 +-
.../app/pods/components/alert-details/component.js | 659 +++++++
.../app/pods/components/alert-details/template.hbs | 241 +++
.../app/pods/components/anomaly-graph/component.js | 95 +-
.../modals/yaml-documentation/component.js | 36 -
.../modals/yaml-documentation/template.hbs | 12 -
.../rootcause-callgraph-table/component.js | 137 ++
.../rootcause-callgraph-table/template.hbs | 33 +
.../self-serve-alert-yaml-details/template.hbs | 58 +-
.../app/pods/components/stats-cards/template.hbs | 13 +-
.../app/pods/components/yaml-editor/component.js | 297 ++-
.../app/pods/components/yaml-editor/template.hbs | 97 +-
.../custom/anomalies-table/dimensions/template.hbs | 12 +-
.../thirdeye-frontend/app/pods/home/index/route.js | 2 +-
.../app/pods/home/share-dashboard/controller.js | 238 +--
.../app/pods/home/share-dashboard/route.js | 70 +-
.../app/pods/home/share-dashboard/template.hbs | 8 +-
.../manage/{yaml => explore-new}/controller.js | 0
.../app/pods/manage/explore-new/route.js | 120 ++
.../pods/manage/{yaml => explore-new}/template.hbs | 21 +-
.../app/pods/manage/yaml/route.js | 122 +-
.../app/pods/manage/yaml/template.hbs | 16 +-
.../pods/partials/rootcause/callgraph/template.hbs | 10 +
.../app/pods/rootcause/controller.js | 35 +-
.../app/pods/rootcause/template.hbs | 5 +
.../app/pods/self-serve/create-alert/controller.js | 9 +-
.../app/pods/self-serve/create-alert/route.js | 6 +-
.../app/pods/self-serve/create-alert/template.hbs | 5 +-
.../app/pods/services/api/anomalies/service.js | 4 +-
.../services/rootcause-callgraph-cache/service.js | 106 ++
thirdeye/thirdeye-frontend/app/router.js | 4 +-
.../app/shared/callgraphTableColumns.js | 28 +
thirdeye/thirdeye-frontend/app/styles/app.scss | 1 +
.../app/styles/components/metrics-table.scss | 7 +
.../app/styles/components/rootcause-callgraph.scss | 7 +
.../app/styles/components/yaml-editor.scss | 6 +
thirdeye/thirdeye-frontend/app/utils/anomaly.js | 72 +-
.../thirdeye-frontend/app/utils/api/anomaly.js | 33 +-
thirdeye/thirdeye-frontend/app/utils/constants.js | 88 +-
.../app/utils/manage-alert-utils.js | 4 +-
thirdeye/thirdeye-frontend/app/utils/rca-utils.js | 37 +-
thirdeye/thirdeye-frontend/app/utils/utils.js | 21 +-
thirdeye/thirdeye-frontend/config/environment.js | 9 +-
thirdeye/thirdeye-frontend/package.json | 1 +
.../pods/components/stats-cards/component-test.js | 14 -
.../tests/unit/utils/anomaly-test.js | 6 +-
thirdeye/thirdeye-frontend/yarn.lock | 9 +-
thirdeye/thirdeye-hadoop/pom.xml | 2 +-
.../alert/content/BaseEmailContentFormatter.java | 8 +-
.../alert/fetcher/ContinuumAnomalyFetcher.java | 2 +-
.../alert/fetcher/UnnotifiedAnomalyFetcher.java | 3 +-
.../anomaly/ThirdEyeAnomalyApplication.java | 8 +-
.../anomaly/alert/grouping/AlertGrouper.java | 2 +-
.../alert/grouping/DimensionalAlertGrouper.java | 5 +-
.../anomaly/alert/grouping/DummyAlertGrouper.java | 2 +-
.../HorizontalDimensionalAlertGrouper.java | 2 +-
.../alert/grouping/SimpleGroupedAnomalyMerger.java | 2 +-
.../AlertGroupAuxiliaryInfoProvider.java | 2 +-
...sionalAlertGroupAuxiliaryRecipientProvider.java | 2 +-
.../DummyAlertGroupAuxiliaryInfoProvider.java | 2 +-
.../anomaly/alert/util/AnomalyReportGenerator.java | 2 +-
.../anomaly/alert/util/DataReportHelper.java | 2 +-
.../anomaly/alert/v2/AlertTaskRunnerV2.java | 2 +-
.../classification/ClassificationTaskRunner.java | 7 +-
.../detection/AnomalyDetectionInputContext.java | 4 +-
.../AnomalyDetectionInputContextBuilder.java | 8 +-
.../detection/AnomalyDetectionOutputContext.java | 2 +-
.../anomaly/detection/DetectionJobRunner.java | 4 +-
.../detection/DetectionJobSchedulerUtils.java | 4 +-
.../anomaly/detection/DetectionTaskRunner.java | 4 +-
.../anomaly/events/HolidayEventsLoader.java | 2 +-
.../anomaly/merge/TimeBasedAnomalyMerger.java | 7 +-
.../anomaly/monitor/MonitorConfiguration.java | 2 +-
.../thirdeye/anomaly/monitor/MonitorConstants.java | 2 +-
.../tasks/FunctionCreationOnboardingTask.java | 4 +-
.../pinot/thirdeye/anomaly/task/TaskConstants.java | 1 +
.../thirdeye/anomaly/task/TaskInfoFactory.java | 4 +
.../thirdeye/anomaly/task/TaskRunnerFactory.java | 4 +
.../anomalydetection/context/AnomalyResult.java | 2 +-
.../anomalydetection/context/RawAnomalyResult.java | 2 +-
.../anomalydetection/context/TimeSeriesKey.java | 2 +-
.../datafilter/AverageThresholdDataFilter.java | 4 +-
.../anomalydetection/datafilter/DataFilter.java | 4 +-
.../datafilter/DummyDataFilter.java | 4 +-
.../AbstractModularizedAnomalyFunction.java | 4 +-
.../function/BackwardAnomalyFunctionUtils.java | 4 +-
.../detection/MinMaxThresholdDetectionModel.java | 2 +-
.../detection/SimpleThresholdDetectionModel.java | 2 +-
.../transform/MovingAverageSmoothingFunction.java | 2 +-
.../BasePerformanceEvaluate.java | 2 +-
.../PrecisionByTimePerformanceEvaluation.java | 2 +-
.../RecallByTimePreformanceEvaluation.java | 2 +-
.../apache/pinot/thirdeye/api/DimensionSpec.java | 75 -
.../apache/pinot/thirdeye/api/DimensionType.java | 25 -
.../user/dashboard}/UserDashboardResource.java | 170 +-
.../auto/onboard/AutoOnboardConfiguration.java | 2 +-
.../thirdeye/auto/onboard/AutoOnboardService.java | 2 +-
.../thirdeye/auto/onboard/ConfigGenerator.java | 4 +-
.../{api => common/dimension}/DimensionKey.java | 12 +-
.../{api => common/dimension}/DimensionMap.java | 2 +-
.../{api => common/metric}/MetricSchema.java | 2 +-
.../{api => common/metric}/MetricSpec.java | 2 +-
.../{api => common/metric}/MetricTimeSeries.java | 3 +-
.../{api => common/metric}/MetricType.java | 2 +-
.../{api => common/time}/TimeGranularity.java | 2 +-
.../thirdeye/{api => common/time}/TimeRange.java | 2 +-
.../thirdeye/{api => common/time}/TimeSpec.java | 2 +-
.../checker/DataCompletenessAlgorithm.java | 2 +-
.../checker/DataCompletenessTaskRunner.java | 2 +-
.../checker/DataCompletenessUtils.java | 2 +-
.../checker/Wo4WAvgDataCompletenessAlgorithm.java | 2 +-
.../dashboard/ThirdEyeDashboardApplication.java | 17 +-
.../org/apache/pinot/thirdeye/dashboard/Utils.java | 4 +-
.../dashboard/resources/AnomalyResource.java | 7 +-
.../dashboard/resources/DashboardResource.java | 21 +-
.../dashboard/resources/DetectionJobResource.java | 19 +-
.../dashboard/resources/MetricConfigResource.java | 2 +-
.../dashboard/resources/v2/AnomaliesResource.java | 121 +-
.../dashboard/resources/v2/DataResource.java | 3 +-
.../dashboard/resources/v2/ResourceUtils.java | 2 +-
.../resources/v2/RootCauseMetricResource.java | 2 +-
.../dashboard/resources/v2/TimeSeriesResource.java | 2 +-
.../dashboard/resources/v2/pojo/SearchFilters.java | 24 +-
.../dashboard/views/CompareViewRequest.java | 2 +-
.../thirdeye/dashboard/views/ViewRequest.java | 2 +-
.../views/contributor/ContributorViewHandler.java | 2 +-
.../views/heatmap/HeatMapViewHandler.java | 2 +-
.../views/tabular/TabularViewHandler.java | 2 +-
.../thirdeye/dataframe/util/DataFrameUtils.java | 2 +-
.../pinot/thirdeye/dataframe/util/MetricSlice.java | 10 +-
.../thirdeye/datalayer/dto/AlertSnapshotDTO.java | 2 +-
.../thirdeye/datalayer/dto/AnomalyFunctionDTO.java | 10 +-
.../thirdeye/datalayer/dto/DatasetConfigDTO.java | 2 +-
.../entity/GroupedAnomalyResultsIndex.java | 2 +-
.../datalayer/entity/MergedAnomalyResultIndex.java | 2 +-
.../datalayer/entity/RawAnomalyResultIndex.java | 2 +-
.../datalayer/pojo/AnomalyFunctionBean.java | 2 +-
.../thirdeye/datalayer/pojo/DatasetConfigBean.java | 5 +-
.../datalayer/pojo/DetectionConfigBean.java | 9 +
.../datalayer/pojo/GroupedAnomalyResultsBean.java | 2 +-
.../datalayer/pojo/MergedAnomalyResultBean.java | 2 +-
.../thirdeye/datalayer/pojo/MetricConfigBean.java | 2 +-
.../datalayer/pojo/RawAnomalyResultBean.java | 2 +-
.../thirdeye/datasource/BaseThirdEyeResponse.java | 2 +-
.../pinot/thirdeye/datasource/ThirdEyeRequest.java | 3 +-
.../thirdeye/datasource/ThirdEyeResponse.java | 2 +-
.../pinot/thirdeye/datasource/TimeRangeUtils.java | 7 +-
.../comparison/ThirdEyeRequestGenerator.java | 2 +-
.../comparison/TimeOnTimeComparisonHandler.java | 2 +-
.../comparison/TimeOnTimeComparisonRequest.java | 2 +-
.../comparison/TimeOnTimeResponseParser.java | 2 +-
.../datasource/csv/CSVThirdEyeDataSource.java | 6 +-
.../datasource/csv/CSVThirdEyeResponse.java | 2 +-
.../datasource/pinot/PinotDataSourceMaxTime.java | 2 +-
.../datasource/pinot/PinotThirdEyeDataSource.java | 4 +-
.../datasource/pinot/PinotThirdEyeResponse.java | 2 +-
.../pinot/thirdeye/datasource/pinot/PqlUtils.java | 4 +-
.../timeseries/BaseTimeSeriesResponseParser.java | 2 +-
.../datasource/timeseries/TimeSeriesHandler.java | 2 +-
.../datasource/timeseries/TimeSeriesRequest.java | 2 +-
.../timeseries/TimeSeriesResponseConverter.java | 8 +-
.../pinot/thirdeye/detection/DataProvider.java | 14 +-
.../thirdeye/detection/DefaultDataProvider.java | 83 +-
.../detection/DetectionMigrationResource.java | 441 ++++-
.../thirdeye/detection/DetectionPipeline.java | 6 +-
.../thirdeye/detection/DetectionResource.java | 136 +-
.../pinot/thirdeye/detection/DetectionUtils.java | 63 +-
.../detection/LegacyAnomalyFunctionTranslator.java | 1 +
.../alert/StatefulDetectionAlertFilter.java | 4 +-
.../detection/algorithm/DimensionWrapper.java | 1 +
.../algorithm/LegacyAlertFilterWrapper.java | 1 +
.../algorithm/LegacyAnomalyFunctionAlgorithm.java | 10 +-
.../algorithm/LegacyDimensionWrapper.java | 1 +
.../detection/algorithm/LegacyMergeWrapper.java | 11 +-
.../thirdeye/detection/algorithm/MergeWrapper.java | 20 +-
.../algorithm/RuleBasedFilterWrapper.java | 1 +
.../stage/AnomalyDetectionStageWrapper.java | 4 +-
.../algorithm/stage/AnomalyFilterStageWrapper.java | 1 +
.../algorithm/stage/GrouperStageWrapper.java | 1 +
.../registry/DetectionAlertRegistry.java | 12 +-
.../annotation/registry/DetectionRegistry.java | 31 +-
.../detection/components/RuleBaselineProvider.java | 13 -
.../finetune/GridSearchTuningAlgorithm.java | 1 +
.../finetune/TimeBucketF1ScoreFunction.java | 3 +-
.../detection/onboard/YamlOnboardingTaskInfo.java | 42 +-
.../onboard/YamlOnboardingTaskRunner.java | 138 ++
.../detection/spi/components/BaselineProvider.java | 8 +-
.../thirdeye/detection/spi/model/TimeSeries.java | 7 +-
.../detection/validators/ConfigValidator.java | 58 +-
.../validators/DetectionConfigValidator.java | 101 +
...dator.java => SubscriptionConfigValidator.java} | 78 +-
.../detection/wrapper/AnomalyDetectorWrapper.java | 56 +-
.../detection/wrapper/AnomalyFilterWrapper.java | 1 +
.../yaml/CompositePipelineConfigTranslator.java | 61 +-
.../yaml/YamlDetectionAlertConfigTranslator.java | 96 +-
.../yaml/YamlDetectionConfigTranslator.java | 10 +-
.../yaml/YamlDetectionTranslatorLoader.java | 1 +
.../thirdeye/detection/yaml/YamlResource.java | 635 ++++---
.../detector/email/AnomalyGraphGenerator.java | 2 +-
.../email/filter/PrecisionRecallEvaluator.java | 4 +-
.../detector/function/AnomalyFunction.java | 4 +-
.../detector/function/BaseAnomalyFunction.java | 6 +-
.../detector/metric/transfer/MetricTransfer.java | 2 +-
.../rootcause/impl/MetricAnalysisPipeline.java | 2 +-
.../rootcause/impl/MetricAnalysisPipeline2.java | 2 +-
.../thirdeye/tracking/RequestStatisticsLogger.java | 2 +-
.../apache/pinot/thirdeye/util/IntervalUtils.java | 5 +-
.../apache/pinot/thirdeye/util/NumberUtils.java | 2 +-
.../apache/pinot/thirdeye/util/ThirdEyeUtils.java | 6 +-
.../src/main/resources/scripts/getGraphPnj.js | 4 +-
...HierarchicalAnomaliesEmailContentFormatter.java | 2 +-
...TestMultipleAnomaliesEmailContentFormatter.java | 2 +-
...TestOnboardingNotificationContentFormatter.java | 2 +-
.../thirdeye/alert/feed/TestUnionAnomalyFeed.java | 2 +-
.../anomaly/alert/grouping/AlertGroupKeyTest.java | 2 +-
.../grouping/DimensionalAlertGrouperTest.java | 2 +-
.../alert/grouping/DummyAlertGrouperTest.java | 2 +-
.../HorizontalDimensionalAlertGrouperTest.java | 2 +-
...alAlertGroupAuxiliaryRecipientProviderTest.java | 2 +-
.../detection/TestDetectionJobSchedulerUtils.java | 2 +-
.../datafilter/AverageThresholdDataFilterTest.java | 8 +-
.../function/BackwardAnomalyFunctionUtilsTest.java | 6 +-
.../function/TestBackwardAnoamlyFunctionUtils.java | 6 +-
.../function/TestMinMaxThresholdFunction.java | 2 +-
.../function/TestWeekOverWeekRuleFunction.java | 2 +-
.../AutoOnboardPinotMetricsServiceTest.java | 2 +-
.../auto/onboard/AutoOnboardServiceTest.java | 2 +-
.../dimension}/DimensionMapTest.java | 3 +-
.../metric}/MetricTimeSeriesTest.java | 3 +-
.../checker/DataCompletenessUtilsTest.java | 4 +-
.../dashboard/handler/ContributorTest.java | 2 +-
.../thirdeye/dashboard/handler/HeatMapTest.java | 2 +-
.../thirdeye/dashboard/handler/TabularTest.java | 2 +-
.../resource/v2/UserDashboardResourceTest.java | 42 +-
.../pinot/thirdeye/datalayer/DaoTestUtils.java | 4 +-
.../bao/TestGroupedAnomalyResultsManager.java | 2 +-
.../datalayer/bao/TestRawAnomalyResultManager.java | 2 +-
.../datalayer/dto/DatasetConfigDTOTest.java | 2 +-
.../thirdeye/datasource/TestTimeRangeUtils.java | 4 +-
.../datasource/comparison/TimeOnTimeTest.java | 5 +-
.../datasource/csv/CSVThirdEyeDataSourceTest.java | 4 +-
.../datasource/csv/CSVThirdEyeResponseTest.java | 4 +-
.../thirdeye/datasource/pinot/PqlUtilsTest.java | 4 +-
.../BaseTimeSeriesResponseParserTest.java | 4 +-
.../timeseries/TestTimeSeriesResponseUtils.java | 11 +-
.../datasource/timeseries/TimeSeriesTest.java | 5 +-
.../pinot/thirdeye/detection/DataProviderTest.java | 17 +-
.../detection/DetectionMigrationResourceTest.java | 203 +-
.../detection/DetectionPipelineTaskRunnerTest.java | 4 +-
.../thirdeye/detection/DetectionTestUtils.java | 2 +-
.../components/RuleBaselineProviderTest.java | 2 +-
.../wrapper/AnomalyDetectorWrapperTest.java | 2 +-
.../wrapper/BaselineFillingMergeWrapperTest.java | 6 +-
.../YamlDetectionAlertConfigTranslatorTest.java | 85 +-
.../yaml/YamlDetectionConfigTranslatorTest.java | 2 +
.../thirdeye/detection/yaml/YamlResourceTest.java | 154 +-
.../detector/email/filter/TestUserReportUtils.java | 2 +-
.../metric/transfer/testMetricTransfer.java | 6 +-
.../AnomalyApplicationEndToEndTest.java | 4 +-
.../rootcause/timeseries/BaselineTest.java | 2 +-
.../pinot/thirdeye/tools/DashboardHttpUtils.java | 1 -
.../FetchMetricDataAndExistingAnomaliesTool.java | 2 +-
.../tools/RunAdhocDatabaseQueriesTool.java | 146 +-
.../detection/legacy-anomaly-function-1.json | 27 +
.../detection/legacy-anomaly-function-2.json | 26 +
.../detection/migrated-detection-config-1.json | 48 +
.../detection/migrated-detection-config-2.json | 44 +
.../detection/yaml/alertconfig/alert-config-1.yaml | 3 -
.../detection/yaml/alertconfig/alert-config-2.yaml | 7 +-
.../detection/yaml/alertconfig/alert-config-3.yaml | 6 +-
.../detection/yaml/alertconfig/alert-config-4.yaml | 6 +-
.../thirdeye/detection/yaml/pipeline-config-1.yaml | 1 +
.../thirdeye/detection/yaml/pipeline-config-2.yaml | 1 +
1398 files changed, 22285 insertions(+), 15653 deletions(-)
create mode 100644 DISCLAIMER
copy LICENSE => LICENSE-binary (64%)
create mode 100644 NOTICE-binary
delete mode 100644 docs/High-level-stream.png
delete mode 100644 docs/Low-level-stream.png
delete mode 100644 docs/PlanNode.png
delete mode 100644 docs/ServerSegmentCompletion.dot.png
delete mode 100644 docs/commit-happy-path-1.png
delete mode 100644 docs/commit-happy-path-2.png
delete mode 100644 docs/committer-failed.png
delete mode 100644 docs/controller-failed.png
delete mode 100644 docs/controller-segment-completion.png
delete mode 100644 docs/delayed-server.png
delete mode 100644 docs/expressionTree.jpg
delete mode 100644 docs/expressions_udf.rst
create mode 100644 docs/index_techniques.rst
delete mode 100644 docs/llc.rst
delete mode 100644 docs/multiple-server-failure.png
delete mode 100644 docs/parseTree.png
delete mode 100644 docs/partition_aware_routing.rst
create mode 100644 docs/pluggable_storage.rst
delete mode 100644 docs/schema_timespec.rst
delete mode 100644 docs/segment-consumer-fsm.png
delete mode 100644 docs/segment-creation.png
delete mode 100644 docs/segment-helix-fsm.png
delete mode 100644 docs/zk-setup.png
create mode 100644 licenses-binary/LICENSE-angular.txt
create mode 100644 licenses-binary/LICENSE-antlr.txt
create mode 100644 licenses-binary/LICENSE-args4j.txt
create mode 100644 licenses-binary/LICENSE-azure-data-lake-store-sdk.txt
create mode 100644 licenses-binary/LICENSE-cddl-1.0.txt
create mode 100644 licenses-binary/LICENSE-cddl-1.1.txt
create mode 100644 licenses-binary/LICENSE-codemirror.txt
create mode 100644 licenses-binary/LICENSE-cpl-1.0.txt
create mode 100644 licenses-binary/LICENSE-epl-1.0.txt
create mode 100644 licenses-binary/LICENSE-epl-2.0.txt
create mode 100644 licenses-binary/LICENSE-foundation.txt
create mode 100644 licenses-binary/LICENSE-handlebars.txt
create mode 100644 licenses-binary/LICENSE-jcabi.txt
create mode 100644 licenses-binary/LICENSE-jline.txt
create mode 100644 licenses-binary/LICENSE-jopt-simple.txt
create mode 100644 licenses-binary/LICENSE-jquery-datatables.txt
create mode 100644 licenses-binary/LICENSE-jquery.txt
create mode 100644 licenses-binary/LICENSE-js-beautify.txt
create mode 100644 licenses-binary/LICENSE-mimepull.txt
create mode 100644 licenses-binary/LICENSE-normalize.txt
create mode 100644 licenses-binary/LICENSE-paranamer.txt
create mode 100644 licenses-binary/LICENSE-scala.txt
create mode 100644 licenses-binary/LICENSE-slf4j.txt
create mode 100644 licenses-binary/LICENSE-underscore.txt
create mode 100644 licenses/LICENSE-angular.txt
create mode 100644 licenses/LICENSE-codemirror.txt
create mode 100644 licenses/LICENSE-foundation.txt
create mode 100644 licenses/LICENSE-handlebars.txt
create mode 100644 licenses/LICENSE-jquery-datatables.txt
create mode 100644 licenses/LICENSE-jquery.txt
create mode 100644 licenses/LICENSE-js-beautify.txt
create mode 100644 licenses/LICENSE-normalize.txt
create mode 100644 licenses/LICENSE-underscore.txt
create mode 100644
pinot-common/src/test/java/org/apache/pinot/common/metadata/segment/ColumnPartitionMetadataTest.java
create mode 100644
pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerPeriodicTaskStarterTest.java
delete mode 100644
pinot-core/src/main/java/org/apache/pinot/core/segment/name/DefaultSegmentNameGenerator.java
copy
pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/LimitAstNode.java
=>
pinot-core/src/main/java/org/apache/pinot/core/segment/name/FixedSegmentNameGenerator.java
(59%)
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/segment/name/SimpleSegmentNameGenerator.java
delete mode 100644
pinot-core/src/test/java/org/apache/pinot/core/segment/name/DefaultSegmentNameGeneratorTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/segment/name/SimpleSegmentNameGeneratorTest.java
create mode 100644 pinot-distribution/pinot-source-assembly.xml
create mode 100644
pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/BaseSegmentJob.java
create mode 100644
pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/DefaultControllerRestApi.java
delete mode 100644
pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mapper/HadoopSegmentCreationMapReduceJob.java
create mode 100644
pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mapper/SegmentCreationMapper.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/AdminConsoleIntegrationTest.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/controller/periodic/tasks/SegmentStatusCheckerIntegrationTest.java
create mode 100644 pinot-perf/README.md
create mode 100644
thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
create mode 100644
thirdeye/thirdeye-frontend/app/pods/components/alert-details/template.hbs
delete mode 100644
thirdeye/thirdeye-frontend/app/pods/components/modals/yaml-documentation/component.js
delete mode 100644
thirdeye/thirdeye-frontend/app/pods/components/modals/yaml-documentation/template.hbs
create mode 100644
thirdeye/thirdeye-frontend/app/pods/components/rootcause-callgraph-table/component.js
create mode 100644
thirdeye/thirdeye-frontend/app/pods/components/rootcause-callgraph-table/template.hbs
copy thirdeye/thirdeye-frontend/app/pods/manage/{yaml =>
explore-new}/controller.js (100%)
create mode 100644
thirdeye/thirdeye-frontend/app/pods/manage/explore-new/route.js
copy thirdeye/thirdeye-frontend/app/pods/manage/{yaml =>
explore-new}/template.hbs (63%)
create mode 100644
thirdeye/thirdeye-frontend/app/pods/partials/rootcause/callgraph/template.hbs
create mode 100644
thirdeye/thirdeye-frontend/app/pods/services/rootcause-callgraph-cache/service.js
create mode 100644
thirdeye/thirdeye-frontend/app/shared/callgraphTableColumns.js
create mode 100644
thirdeye/thirdeye-frontend/app/styles/components/rootcause-callgraph.scss
delete mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/api/DimensionSpec.java
delete mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/api/DimensionType.java
rename
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{dashboard/resources/v2
=> api/user/dashboard}/UserDashboardResource.java (79%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{api =>
common/dimension}/DimensionKey.java (93%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{api =>
common/dimension}/DimensionMap.java (99%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{api =>
common/metric}/MetricSchema.java (98%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{api =>
common/metric}/MetricSpec.java (96%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{api =>
common/metric}/MetricTimeSeries.java (99%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{api =>
common/metric}/MetricType.java (97%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{api =>
common/time}/TimeGranularity.java (99%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{api =>
common/time}/TimeRange.java (98%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{api =>
common/time}/TimeSpec.java (97%)
copy
pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/LimitAstNode.java
=>
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/onboard/YamlOnboardingTaskInfo.java
(57%)
create mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/onboard/YamlOnboardingTaskRunner.java
create mode 100644
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/validators/DetectionConfigValidator.java
rename
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/validators/{DetectionAlertConfigValidator.java
=> SubscriptionConfigValidator.java} (57%)
rename thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/{api =>
common/dimension}/DimensionMapTest.java (97%)
rename thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/{api =>
common/metric}/MetricTimeSeriesTest.java (99%)
create mode 100644
thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/legacy-anomaly-function-1.json
create mode 100644
thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/legacy-anomaly-function-2.json
create mode 100644
thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/migrated-detection-config-1.json
create mode 100644
thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/migrated-detection-config-2.json
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
