This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a change to branch
Add-queue-based-throttling-to-ThrottleOnCriticalHeapUsageExecutor
in repository https://gitbox.apache.org/repos/asf/pinot.git
discard f59a6240150 Implement comprehensive queue-based throttling for
ThrottleOnCriticalHeapUsageExecutor
add 8075abf0673 Tenant Rebalance Cancellation (#16886)
add 2bf80df1768 Allow manual scheudling of PeriodicTask even if they have
auto-scheduling turned off (#16934)
add bcb5302577a Update usages of some deprecated methods from
jackson-databind (#16941)
add 3b5ef194466 [ngram index part 1]Add realtime ngram filtering index and
benchmark results. (#16364)
add e76ada7da52 Bump org.checkerframework:checker-qual from 3.51.0 to
3.51.1 (#16945)
add 1a76d8dc9f7 Add support to add a resource level Helix constraint on
state transition messages (#16933)
add edffdecd86a Disable StaleInstancesCleanupTask by default (#16942)
add 0a9e7ac48dd Bump software.amazon.awssdk:bom from 2.34.7 to 2.34.8
(#16944)
add 77b8c8af677 Use v4 as the default raw index version (#16943)
add 47f366de875 Disallow allowableValues and example together in @ApiParam
due to swagger UI bug (#16947)
add 8f28975f356 Bump software.amazon.awssdk:bom from 2.34.8 to 2.34.9
(#16948)
add 0422a2dd31d Bump com.azure:azure-sdk-bom from 1.2.38 to 1.3.0 (#16949)
add ee0e5a32890 Bump org.webjars:swagger-ui from 5.28.1 to 5.29.2 (#16951)
add 3d76866c3a5 Bump com.github.luben:zstd-jni from 1.5.7-4 to 1.5.7-5
(#16952)
add df74d57666d Bump org.apache.maven.enforcer:enforcer-api from 3.6.1 to
3.6.2 (#16950)
add b52b903c04d Cleap up deprecated configs for MinionInstancesCleanupTask
(#16946)
add 777796a3615 Encode column and segment names when constructing metadata
read URL (#16940)
add d0a9a7f47a2 Preserve the Order of the Pre-check Items (#15496)
add a181e360abc Update list of collaborators post 1.4.0 release based on
recent commit history (#16954)
add afbb0fc170c Add new metrics exporting the total max and used memory by
Netty and gRPC (#16939)
add 2f59e6c0fc7 Bump it.unimi.dsi:fastutil from 8.5.17 to 8.5.18 (#16962)
add 6858f18e589 Bump software.amazon.awssdk:bom from 2.34.9 to 2.35.0
(#16960)
add 833cc0122c4 Bump org.ow2.asm:asm from 9.8 to 9.9 (#16963)
add f8642ff8d9a Bump org.codehaus.mojo:exec-maven-plugin from 3.6.0 to
3.6.1 (#16961)
add 3efaa808bdc Reenable the single commit for idealstate update to zk
metadata. (#16819)
add 6d187f22f4a Controller REST API to extract table names for MSE queries
(#16957)
add a4193e6c270 Use 2 bitsets in ScanBasedRegexpLikePredicateEvaluator to
track DictIdToRegexMatcher for REGEX_LIKE predicate (#16922)
add a1455b22319 Rebuild H3 index on segment reload if resolution config is
updated (#16953)
add 9e8caf52571 Resets in-progress pauseless table segments (#16904)
add c823f73af59 Optimise index stats collector for no dict (#16845)
add f73e756d06d Fix Realtime Ingestion Metrics (#16783)
add a439da1490e Bump software.amazon.awssdk:bom from 2.35.0 to 2.35.1
(#16969)
add 5197f7b4598 Bump org.webjars:swagger-ui from 5.29.2 to 5.29.3 (#16970)
add 13647c6c38a Report table's server instances based on ideal state
instead of tags (#16964)
add 608cf9af86c Change default kafka consumer to use Kafka 3 (#16858)
add b3c8db0605a Implement comprehensive queue-based throttling for
ThrottleOnCriticalHeapUsageExecutor
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (f59a6240150)
\
N -- N -- N
refs/heads/Add-queue-based-throttling-to-ThrottleOnCriticalHeapUsageExecutor
(b3c8db0605a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 18 +-
.github/workflows/pinot_tests.yml | 6 +
.mvn/jvm.config | 8 +
config/checkstyle.xml | 14 +
.../broker/api/resources/PinotBrokerRouting.java | 7 +-
.../apache/pinot/broker/grpc/BrokerGrpcServer.java | 10 +
.../requesthandler/BaseBrokerRequestHandler.java | 7 +
.../BaseSingleStageBrokerRequestHandler.java | 4 +
.../java/org/apache/pinot/client/Connection.java | 1 +
.../apache/pinot/client/ExternalViewReader.java | 18 +-
.../apache/pinot/common/metrics/BrokerGauge.java | 11 +-
.../apache/pinot/common/metrics/ServerGauge.java | 29 +-
.../apache/pinot/common/metrics/ServerMeter.java | 4 +-
.../apache/pinot/common/metrics/ServerTimer.java | 5 +-
.../response/PinotBrokerTimeSeriesResponse.java | 2 +-
.../response/encoder/JsonResponseEncoder.java | 2 +-
.../common/utils/config/QueryOptionsUtils.java | 11 +
.../common/utils/helix/IdealStateGroupCommit.java | 1 +
.../common/utils/helix/IdealStateSingleCommit.java | 154 ++++
.../sql/parsers/parser}/TableNameExtractor.java | 2 +-
.../prometheus/ServerPrometheusMetricsTest.java | 5 +-
.../parsers/parser}/TableNameExtractorTest.java | 2 +-
pinot-connectors/pinot-spark-3-connector/pom.xml | 3 +
.../pinot/controller/BaseControllerStarter.java | 18 +
.../apache/pinot/controller/ControllerConf.java | 109 +--
.../controller/api/resources/DebugResource.java | 8 +-
.../api/resources/PinotQueryResource.java | 32 +
.../api/resources/PinotSegmentRestletResource.java | 27 +-
.../api/resources/PinotTableInstances.java | 2 +-
.../api/resources/PinotTableRestletResource.java | 7 +-
.../api/resources/PinotTenantRestletResource.java | 38 +
.../api/resources/ZookeeperResource.java | 20 +-
.../controller/helix/ControllerRequestClient.java | 5 +-
.../controller/helix/RealtimeConsumerMonitor.java | 5 -
.../controller/helix/SegmentStatusChecker.java | 8 -
.../helix/core/PinotHelixResourceManager.java | 23 +-
.../core/cleanup/StaleInstancesCleanupTask.java | 5 +
.../realtime/PinotLLCRealtimeSegmentManager.java | 84 ++-
.../core/rebalance/DefaultRebalancePreChecker.java | 5 +-
.../helix/core/rebalance/RebalanceChecker.java | 26 +-
.../helix/core/rebalance/RebalanceConfig.java | 35 +
.../core/rebalance/TableRebalanceManager.java | 19 +-
.../helix/core/rebalance/TableRebalancer.java | 2 +-
.../rebalance/tenant/TenantRebalanceChecker.java | 121 +--
.../rebalance/tenant/TenantRebalanceConfig.java | 20 +
.../rebalance/tenant/TenantRebalanceContext.java | 47 ++
.../rebalance/tenant/TenantRebalanceObserver.java | 38 -
.../tenant/TenantRebalanceProgressStats.java | 44 +-
.../core/rebalance/tenant/TenantRebalancer.java | 93 +--
.../tenant/ZkBasedTenantRebalanceObserver.java | 290 ++++++--
.../util/ServerSegmentMetadataReader.java | 69 +-
.../pinot/controller/util/TableMetadataReader.java | 2 +-
.../RealtimeSegmentValidationManager.java | 2 +-
.../pinot/controller/ControllerConfTest.java | 53 +-
.../tenant/TenantRebalanceCheckerTest.java | 55 +-
.../rebalance/tenant/TenantRebalancerTest.java | 510 ++++++++++++-
.../resources/memory_estimation/table-config.json | 3 +-
.../manager/realtime/IngestionDelayTracker.java | 470 +++++++-----
.../realtime/RealtimeSegmentDataManager.java | 9 +-
.../manager/realtime/RealtimeTableDataManager.java | 27 +-
.../predicate/PredicateEvaluatorProvider.java | 14 +-
.../RegexpLikePredicateEvaluatorFactory.java | 45 +-
.../core/periodictask/PeriodicTaskScheduler.java | 89 +--
.../query/scheduler/resources/ResourceManager.java | 12 +-
.../apache/pinot/core/transport/QueryServer.java | 4 +
.../pinot/core/transport/grpc/GrpcQueryServer.java | 10 +
.../realtime/IngestionDelayTrackerTest.java | 329 +++++++--
.../periodictask/PeriodicTaskSchedulerTest.java | 34 +-
.../fakestream/FakeStreamMetadataProvider.java | 18 +
.../org/apache/pinot/queries/BaseQueriesTest.java | 13 +
.../pinot/queries/CustomReloadQueriesTest.java | 214 ++++++
.../ForwardIndexHandlerReloadQueriesTest.java | 12 -
.../tests/BaseClusterIntegrationTest.java | 5 +
.../tests/HybridClusterIntegrationTest.java | 16 +-
.../tests/OfflineClusterIntegrationTest.java | 35 +-
...PartialUpsertTableRebalanceIntegrationTest.java | 16 +-
...imeIngestionConsumingTransitionFailureTest.java | 45 ++
...imeIngestionConsumingTransitionFailureTest.java | 45 ++
...sRealtimeIngestionSegmentCommitFailureTest.java | 43 +-
.../ingestion/RealtimeKinesisIntegrationTest.java | 5 +-
...FailureInjectingRealtimeSegmentDataManager.java | 6 +-
.../FailureInjectingRealtimeTableDataManager.java | 23 +-
.../utils/FailureInjectingTableConfig.java | 69 ++
.../FailureInjectingTableDataManagerProvider.java | 24 +-
.../pinot/perf/BenchmarkNgramFilteringIndex.java | 102 +++
.../kafka20/KafkaStreamMetadataProvider.java | 28 +
.../kafka20/KafkaPartitionLevelConsumerTest.java | 26 +-
.../kafka30/KafkaStreamMetadataProvider.java | 28 +
.../kafka30/KafkaPartitionLevelConsumerTest.java | 26 +-
.../kinesis/KinesisPartitionGroupOffset.java | 2 +-
.../kinesis/KinesisStreamMetadataProvider.java | 5 +
.../pulsar/PulsarStreamMetadataProvider.java | 5 +
.../query/mailbox/channel/GrpcMailboxServer.java | 23 +
.../apache/pinot/query/runtime/QueryRunner.java | 7 +-
.../invertedindex/RealtimeNgramFilteringIndex.java | 196 +++++
.../stats/MapColumnPreIndexStatsCollector.java | 29 +-
.../stats/NoDictColumnStatisticsCollector.java | 232 ++++++
.../stats/SegmentPreIndexStatsCollectorImpl.java | 15 +
.../segment/index/loader/ForwardIndexHandler.java | 8 +
.../defaultcolumn/BaseDefaultColumnHandler.java | 41 +-
.../index/loader/invertedindex/H3IndexHandler.java | 44 ++
.../segment/local/utils/TableConfigUtils.java | 4 +-
.../RealtimeNgramFilteringIndexTest.java | 96 +++
.../stats/MapColumnPreIndexStatsCollectorTest.java | 97 ++-
.../stats/NoDictColumnStatisticsCollectorTest.java | 812 +++++++++++++++++++++
.../SegmentPreIndexStatsCollectorImplTest.java | 66 ++
.../index/loader/ForwardIndexHandlerTest.java | 55 +-
.../index/loader/SegmentPreProcessorTest.java | 88 ++-
.../spi/creator/ColumnIndexCreationInfo.java | 17 +-
.../segment/spi/index/ForwardIndexConfig.java | 11 +-
.../spi/partition/PartitionFunctionTest.java | 2 +-
.../server/api/resources/ReingestionResource.java | 5 +-
.../api/resources/SegmentMetadataFetcher.java | 8 +-
.../pinot/server/api/resources/TablesResource.java | 99 +--
.../org/apache/pinot/spi/config/ConfigUtils.java | 5 +-
.../pinot/spi/config/table/IndexingConfig.java | 12 +
.../ThrottleOnCriticalHeapUsageExecutor.java | 165 ++---
.../org/apache/pinot/spi/stream/StreamConfig.java | 2 +-
.../pinot/spi/stream/StreamMetadataProvider.java | 17 +
.../apache/pinot/spi/utils/CommonConstants.java | 21 +-
.../pinot/spi/utils/IngestionConfigUtils.java | 24 +
.../java/org/apache/pinot/spi/utils/JsonUtils.java | 13 +-
.../utils/builder/ControllerRequestURLBuilder.java | 57 +-
.../spi/utils/builder/TableConfigBuilder.java | 7 +
.../pinot/spi/utils/builder/UrlBuilderUtils.java | 75 ++
.../apache/pinot/spi/config/ConfigUtilsTest.java | 2 +-
.../pinot/spi/utils/IngestionConfigUtilsTest.java | 25 +
.../src/main/resources/appAssemblerScriptTemplate | 3 +
pom.xml | 21 +-
129 files changed, 5060 insertions(+), 1347 deletions(-)
create mode 100644 .mvn/jvm.config
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/utils/helix/IdealStateSingleCommit.java
rename {pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client
=>
pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser}/TableNameExtractor.java
(99%)
rename {pinot-clients/pinot-java-client/src/test/java/org/apache/pinot/client
=>
pinot-common/src/test/java/org/apache/pinot/sql/parsers/parser}/TableNameExtractorTest.java
(99%)
delete mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/tenant/TenantRebalanceObserver.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/queries/CustomReloadQueriesTest.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/PauselessDedupRealtimeIngestionConsumingTransitionFailureTest.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/PauselessRealtimeIngestionConsumingTransitionFailureTest.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/realtime/utils/FailureInjectingTableConfig.java
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkNgramFilteringIndex.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeNgramFilteringIndex.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeNgramFilteringIndexTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollectorTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/stats/SegmentPreIndexStatsCollectorImplTest.java
create mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/utils/builder/UrlBuilderUtils.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
