This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a change to branch xiangfu0/data-3221-6-lazy-materialization
in repository https://gitbox.apache.org/repos/asf/pinot.git
omit 9576f825d22 Keep nested and sparse settings distinct during config
sharing
omit d51a3d62695 Use current OpenStruct config constructors in materializer
tests
omit 3a3b0933991 Keep lazy materialization compatible with the segment read
lifecycle
omit ee7356fb5e4 Keep rebased source imports in project order
omit 74f3279936e Serialize v1/v2 index buffer access for lazy
materialization and cover destroy vs in-flight
omit 3dfb91fd9b6 Retain compact index configs for segment-specific loads
omit e465b529ece Reuse resolved index configs for lazy column
materialization
omit 99251704d38 Materialize immutable-segment columns lazily behind an
opt-in instance config (default off)
add 4887783ecb7 Avoid quadratic MutableRoaringBitmap growth when adding
doc id ranges (#19818)
add dd2c3cdfac9 Add size-based table retention (#19770)
add b86760394c3 Fix dependency vulnerabilities in shaded plugins (#19807)
add 16b3e4d3f53 Isolate ingestion offset test trackers and ensure cleanup
add b5397bb0459 [controller] Require unrestricted table scope for
cluster-level BasicAuth access (#18975)
add 3a0ebd94728 Make the default forward index compression codec
configurable (#19797)
add dcda568fbaa Default pinot.server.consuming.segment.consistency.mode to
PROTECTED (#19010)
add b5af5f745ae Prepare t-digest relocation and legacy accuracy baseline
(#19829)
add 3782011c240 Improve MIN/MAX error message for non-numeric columns
(#19653)
add 0ac943c4304 Respect explicit append.uuid.to.segment.name in
SegmentGenerationAndPushTask (#19820)
add a8b6b89b1eb Avoid duplicate row allocations in unordered
single-aggregation group-by merges (#19825)
add fffc79c0cc1 Support Object parameters for column arguments in scalar
function wrapper (#19789)
add de2ae09d7fe Fix MSE group trimming for intermediate aggregation
results (#19800)
add fff3de65d01 Optimize GROUP BY GROUPING SETS/ROLLUP/CUBE via base
aggregation (#19264)
add bf401646428 Remove tdunning t-digest dependency while preserving
legacy bytes (#19795)
add 18ae067f98b Parse-once for JSON-index ingestion: cache the parsed Map
and flatten it directly (#18756)
add c2e17e36e81 Avoid ZooKeeper reads in segment completion requests for
existing FSMs (#19830)
add dea51b0db3d Bump io.opentelemetry:opentelemetry-bom from 1.66.0 to
1.67.0 (#19842)
add 4e30f44bbb0 Bump com.github.luben:zstd-jni from 1.5.7-21 to 1.5.7-22
(#19849)
add 4eb563bd64b Bump org.apache.httpcomponents.client5:httpclient5 from
5.6.4 to 5.6.5 (#19847)
add a6635327b21 Bump actions/upload-artifact from 7.0.1 to 7.0.2 (#19851)
add dd8d255357f Bump io.grpc:grpc-bom from 1.84.0 to 1.84.2 (#19850)
add 730cb2a92c4 Bump org.checkerframework:checker-qual from 4.3.0 to 4.3.1
(#19848)
add 48c055ce022 Bump software.amazon.awssdk:bom from 2.55.11 to 2.55.14
(#19845)
add 479b7741d80 Bump org.jetbrains.kotlin:kotlin-bom from 2.4.20 to 2.4.21
(#19844)
add 58f8ffecf26 Bump commons-codec:commons-codec from 1.22.1 to 1.23.0
(#19841)
add db47c062b04 Emit per-table broker metrics for routing rebuilds on
assignment change (#19784)
add 4212f6f4273 Materialize immutable-segment columns lazily behind an
opt-in instance config (default off)
add f7a53887f08 Reuse resolved index configs for lazy column
materialization
add 465a0072eac Retain compact index configs for segment-specific loads
add 41841930062 Serialize v1/v2 index buffer access for lazy
materialization and cover destroy vs in-flight
add afbf4c6d6af Keep rebased source imports in project order
add 98d0fe7a279 Keep lazy materialization compatible with the segment read
lifecycle
add e87aab86339 Use current OpenStruct config constructors in materializer
tests
add d10860d81a1 Keep nested and sparse settings distinct during config
sharing
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (9576f825d22)
\
N -- N -- N refs/heads/xiangfu0/data-3221-6-lazy-materialization
(d10860d81a1)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/pinot_compatibility_tests.yml | 2 +-
.github/workflows/pinot_integration_tests.yml | 2 +-
...ulti_stage_query_engine_compatibility_tests.yml | 2 +-
.github/workflows/pinot_unit_tests.yml | 2 +-
.github/workflows/pr-flow.yml | 2 +-
LICENSE-binary | 1 -
NOTICE | 4 +
NOTICE-binary | 4 +
compatibility-verifier/README.md | 95 ++
.../tdigest-compatibility/GenerateRankErrors.java | 162 ++
.../tdigest-compatibility/generate-rank-errors.sh | 83 +
helm/pinot/values.yaml | 10 +
.../routing/manager/BaseBrokerRoutingManager.java | 32 +-
.../routing/manager/BrokerRoutingManagerTest.java | 56 +-
.../apache/pinot/common/metrics/BrokerGauge.java | 6 +-
.../apache/pinot/common/metrics/BrokerMeter.java | 15 +
.../apache/pinot/common/metrics/BrokerTimer.java | 3 +
.../pinot/common/metrics/ControllerGauge.java | 5 +
.../apache/pinot/common/metrics/ServerMeter.java | 4 +
.../pinot/common/utils/RoaringBitmapUtils.java | 24 +
.../pinot/common/utils/ServiceStartableUtils.java | 68 +-
.../common/utils/config/QueryOptionsUtils.java | 16 +
.../pinot/common/utils/RoaringBitmapUtilsTest.java | 54 +
.../common/utils/ServiceStartableUtilsTest.java | 152 ++
.../pinot/controller/api/access/AccessControl.java | 17 +-
.../api/access/AuthenticationFilter.java | 57 +-
.../api/access/BaseBasicAuthAccessControl.java | 56 +-
.../api/access/BasicAuthAccessControlFactory.java | 4 +
...PinotControllerPeriodicTaskRestletResource.java | 4 +-
.../resources/PinotIngestionRestletResource.java | 6 +-
.../api/resources/PinotSegmentRestletResource.java | 9 +-
.../PinotSegmentUploadDownloadRestletResource.java | 4 +-
.../api/resources/PinotTableRestletResource.java | 2 +-
.../api/resources/PinotTaskRestletResource.java | 14 +-
.../core/minion/PinotHelixTaskResourceManager.java | 11 +-
.../realtime/BlockingSegmentCompletionFSM.java | 9 +-
.../helix/core/realtime/SegmentCompletionFSM.java | 15 +
.../core/realtime/SegmentCompletionManager.java | 29 +-
.../helix/core/retention/RetentionManager.java | 245 ++-
.../api/PinotSegmentUploadAuthorizationTest.java | 7 +-
.../api/access/AuthenticationFilterTest.java | 277 ++-
.../access/BasicAuthAccessControlFactoryTest.java | 101 +-
...ontrollerClusterBasicAuthAuthorizationTest.java | 108 +-
.../minion/PinotHelixTaskResourceManagerTest.java | 16 +
.../helix/core/realtime/SegmentCompletionTest.java | 57 +
.../helix/core/retention/RetentionManagerTest.java | 943 +++++++++++
.../java/org/apache/pinot/core/auth/Authorize.java | 7 +
.../apache/pinot/core/auth/BasicAuthPrincipal.java | 31 +-
.../pinot/core/auth/FineGrainedAuthUtils.java | 72 +-
.../apache/pinot/core/common/BlockDocIdSet.java | 13 +-
.../apache/pinot/core/common/ObjectSerDeUtils.java | 16 +-
.../core/data/table/ConcurrentIndexedTable.java | 15 +-
.../table/DeterministicConcurrentIndexedTable.java | 11 +-
.../data/table/GroupingSetsBaseIndexedTable.java | 51 +
.../apache/pinot/core/data/table/IndexedTable.java | 124 +-
.../pinot/core/data/table/SimpleIndexedTable.java | 25 +-
.../apache/pinot/core/data/table/TableResizer.java | 43 +-
.../operator/combine/GroupByCombineOperator.java | 146 +-
.../SequentialSortedGroupByCombineOperator.java | 1 +
.../combine/SortedGroupByCombineOperator.java | 1 +
.../pinot/core/operator/docidsets/AndDocIdSet.java | 7 +-
.../pinot/core/operator/docidsets/OrDocIdSet.java | 14 +-
.../filter/SortedIndexBasedFilterOperator.java | 52 +-
.../pinot/core/operator/query/GroupByOperator.java | 54 +
.../streaming/StreamingGroupByCombineOperator.java | 2 +
.../function/ScalarTransformFunctionWrapper.java | 62 +-
.../core/plan/maker/InstancePlanMakerImplV2.java | 194 +++
.../aggregation/function/AggregationFunction.java | 12 +
.../function/AggregationFunctionUtils.java | 16 +-
.../function/MaxAggregationFunction.java | 53 +-
.../function/MinAggregationFunction.java | 53 +-
.../ParentExprMinMaxAggregationFunction.java | 8 +
.../PercentileRawTDigestAggregationFunction.java | 12 +-
.../PercentileSmartTDigestAggregationFunction.java | 43 +-
.../function/PercentileTDigestAccumulator.java | 1585 -----------------
.../PercentileTDigestAggregationFunction.java | 72 +-
...artitionedDistinctCountAggregationFunction.java | 8 +
.../function/funnel/BitmapMergeStrategy.java | 16 +-
.../funnel/FunnelCountAggregationFunction.java | 5 +
.../aggregation/function/funnel/MergeStrategy.java | 7 +
.../function/funnel/PartitionedMergeStrategy.java | 5 +
.../function/funnel/SetMergeStrategy.java | 54 +-
.../FunnelCompleteCountAggregationFunction.java | 1 +
...unnelEventsFunctionEvalAggregationFunction.java | 1 +
.../window/FunnelMatchStepAggregationFunction.java | 1 +
.../window/FunnelMaxStepAggregationFunction.java | 1 +
...FunnelStepDurationStatsAggregationFunction.java | 1 +
.../groupby/DefaultGroupByExecutor.java | 71 +-
.../query/aggregation/groupby/GroupByExecutor.java | 9 +
.../groupby/GroupingSetsGroupKeyGenerator.java | 281 +++-
.../core/query/request/context/QueryContext.java | 72 +
.../aggregator/PercentileTDigestAggregator.java | 2 +-
.../startree/operator/StarTreeFilterOperator.java | 22 +-
.../org/apache/pinot/core/util/GroupByUtils.java | 428 ++++-
.../org/apache/pinot/core/auth/BasicAuthTest.java | 85 +-
.../pinot/core/auth/FineGrainedAuthUtilsTest.java | 159 +-
.../pinot/core/common/ObjectSerDeUtilsTest.java | 43 +-
.../realtime/IngestionDelayTrackerTest.java | 200 ++-
.../pinot/core/data/table/IndexedTableTest.java | 12 +-
.../table/UnboundedConcurrentIndexedTableTest.java | 490 ++++++
.../MixedLayoutGroupByCombineOperatorTest.java | 265 +++
.../StreamingGroupByCombineOperatorTest.java | 5 +
.../ScalarTransformFunctionWrapperTest.java | 47 +
.../plan/maker/InstancePlanMakerImplV2Test.java | 220 +++
.../function/AggregationFunctionUtilsTest.java | 33 +
.../function/MaxAggregationFunctionTest.java | 105 ++
.../function/MinAggregationFunctionTest.java | 105 ++
...ercentileRawTDigestAggregationFunctionTest.java | 122 +-
...centileSmartTDigestAggregationFunctionTest.java | 146 +-
.../PercentileTDigestAggregationFunctionTest.java | 650 ++++---
.../function/funnel/FunnelFinalResultTest.java | 193 +++
.../groupby/EstimateBaseGroupCountTest.java | 97 ++
.../PercentileTDigestAggregatorTest.java | 19 +-
.../v2/PercentileTDigestStarTreeV2Test.java | 2 +-
.../v2/PercentileTDigestWithMVStarTreeV2Test.java | 2 +-
...eAggregatedPercentileTDigestStarTreeV2Test.java | 5 +-
.../pinot/core/util/DeriveGroupingSetsTest.java | 170 ++
.../pinot/queries/FastFilteredCountTest.java | 3 +
.../ObjectParameterScalarFunctionsQueriesTest.java | 195 +++
.../queries/PercentileTDigestMVQueriesTest.java | 20 +-
.../queries/PercentileTDigestQueriesTest.java | 38 +-
.../pinot/queries/SerializedBytesQueriesTest.java | 41 +-
.../resources/data/tdigest-3.3-k1-rank-errors.csv | 299 ++++
...PartialUpsertTableRebalanceIntegrationTest.java | 3 +-
.../tests/SegmentCompletionIntegrationTest.java | 43 +-
.../tests/custom/GroupingSetsQueriesTest.java | 287 ++++
...ureInjectingPinotLLCRealtimeSegmentManager.java | 30 +
.../perf/BenchmarkGroupingSetsQueriesSSE.java | 304 ++++
.../apache/pinot/perf/BenchmarkJsonFlatten.java | 112 ++
.../BenchmarkPercentileTDigestAggregation.java | 2 +-
.../BenchmarkPercentileTDigestCombine.java | 119 +-
...chmarkPercentileTDigestStarTreeAggregation.java | 12 +-
.../BenchmarkPercentileTDigestValueAggregator.java | 2 +-
.../perf/aggregation/TDigestBenchmarkUtils.java | 45 +-
.../SegmentGenerationAndPushTaskExecutor.java | 8 +-
.../SegmentGenerationAndPushTaskGenerator.java | 11 +-
.../MergeRollupTDigestTaskExecutorTest.java | 52 +-
.../SegmentGenerationAndPushTaskExecutorTest.java | 37 +
.../SegmentGenerationAndPushTaskGeneratorTest.java | 151 ++
.../query/runtime/operator/AggregateOperator.java | 16 +-
.../operator/MultistageGroupByExecutor.java | 108 +-
.../runtime/operator/AggregateOperatorTest.java | 198 +++
.../operator/MultistageGroupByExecutorTest.java | 47 +
.../queries/ObjectParameterFunctions.json | 60 +
pinot-segment-local/pom.xml | 4 -
.../src/main/appended-resources/META-INF/NOTICE | 3 +
.../PercentileTDigestValueAggregator.java | 564 +------
.../tdigest/NonFiniteAwareTDigest.java | 779 +++++++++
.../{ => tdigest}/SerializedTDigest.java | 18 +-
.../tdigest/SerializedTDigestInput.java | 165 ++
.../local/customobject/tdigest/TDigest.java | 113 ++
.../customobject/tdigest/TDigestAccumulator.java | 1774 ++++++++++++++++++++
.../local/customobject/tdigest/TDigestCodec.java | 877 ++++++++++
.../indexsegment/mutable/MutableSegmentImpl.java | 13 +
.../realtime/impl/json/MutableJsonIndexImpl.java | 20 +
.../recordtransformer/DataTypeTransformer.java | 89 +-
.../recordtransformer/SanitizationTransformer.java | 8 +-
.../impl/openstruct/OpenStructColumnSplitter.java | 11 +-
.../segment/index/forward/ForwardIndexType.java | 17 +-
.../segment/index/loader/ForwardIndexHandler.java | 7 +-
.../index/openstruct/OpenStructIndexType.java | 7 +-
.../segment/local/utils/CustomSerDeUtils.java | 19 -
.../pinot/segment/local/utils/TDigestUtils.java | 642 -------
.../segment/local/utils/TableConfigUtils.java | 12 +
.../PercentileTDigestValueAggregatorTest.java | 1183 ++++++++++++-
.../tdigest/TDigestAccumulatorTest.java | 1255 ++++++++++++++
.../customobject/tdigest/TDigestCodecTest.java | 1580 +++++++++++++++++
.../indexsegment/mutable/IndexingFailureTest.java | 38 +
.../recordtransformer/DataTypeTransformerTest.java | 132 ++
.../openstruct/OpenStructColumnSplitterTest.java | 2 +-
.../segment/local/segment/index/JsonIndexTest.java | 159 ++
.../forward/ForwardIndexCreatorFactoryTest.java | 52 +
.../ForwardIndexDefaultCompressionTypeTest.java | 100 ++
.../index/openstruct/OpenStructIndexTypeTest.java | 5 +-
...rrentMapPartitionUpsertMetadataManagerTest.java | 2 +
.../segment/local/utils/TDigestTestUtils.java | 109 ++
.../segment/local/utils/TDigestUtilsTest.java | 423 -----
.../segment/local/utils/TableConfigUtilsTest.java | 46 +
.../src/test/resources/tdigest/legacy-3.2.tsv | 92 +
.../src/test/resources/tdigest/legacy-3.3.tsv | 80 +
.../segment/spi/index/ForwardIndexConfig.java | 128 +-
.../SegmentsValidationAndRetentionConfig.java | 69 +-
.../apache/pinot/spi/data/readers/GenericRow.java | 38 +
.../apache/pinot/spi/utils/CommonConstants.java | 35 +
.../ConsumingSegmentConsistencyModeListener.java | 5 +-
.../java/org/apache/pinot/spi/utils/JsonUtils.java | 266 +++
.../spi/utils/builder/TableConfigBuilder.java | 9 +
.../pinot/spi/data/readers/GenericRowTest.java | 36 +
.../org/apache/pinot/spi/utils/JsonUtilsTest.java | 294 ++++
.../spi/utils/builder/TableConfigBuilderTest.java | 41 +
pom.xml | 71 +-
191 files changed, 18909 insertions(+), 4476 deletions(-)
create mode 100644
compatibility-verifier/tdigest-compatibility/GenerateRankErrors.java
create mode 100755
compatibility-verifier/tdigest-compatibility/generate-rank-errors.sh
create mode 100644
pinot-common/src/test/java/org/apache/pinot/common/utils/RoaringBitmapUtilsTest.java
create mode 100644
pinot-common/src/test/java/org/apache/pinot/common/utils/ServiceStartableUtilsTest.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/data/table/GroupingSetsBaseIndexedTable.java
delete mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileTDigestAccumulator.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/data/table/UnboundedConcurrentIndexedTableTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/operator/combine/MixedLayoutGroupByCombineOperatorTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/funnel/FunnelFinalResultTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/EstimateBaseGroupCountTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/util/DeriveGroupingSetsTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/queries/ObjectParameterScalarFunctionsQueriesTest.java
create mode 100644
pinot-core/src/test/resources/data/tdigest-3.3-k1-rank-errors.csv
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkGroupingSetsQueriesSSE.java
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkJsonFlatten.java
create mode 100644
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskGeneratorTest.java
create mode 100644
pinot-query-runtime/src/test/resources/queries/ObjectParameterFunctions.json
create mode 100644
pinot-segment-local/src/main/appended-resources/META-INF/NOTICE
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/tdigest/NonFiniteAwareTDigest.java
rename
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/{
=> tdigest}/SerializedTDigest.java (66%)
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/tdigest/SerializedTDigestInput.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/tdigest/TDigest.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/tdigest/TDigestAccumulator.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/tdigest/TDigestCodec.java
delete mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TDigestUtils.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/customobject/tdigest/TDigestAccumulatorTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/customobject/tdigest/TDigestCodecTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexDefaultCompressionTypeTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TDigestTestUtils.java
delete mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TDigestUtilsTest.java
create mode 100644
pinot-segment-local/src/test/resources/tdigest/legacy-3.2.tsv
create mode 100644
pinot-segment-local/src/test/resources/tdigest/legacy-3.3.tsv
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
