This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a change to branch xiangfu0/data-3221-11-metadata-only-pruning
in repository https://gitbox.apache.org/repos/asf/pinot.git
discard a500a345e7d Honour the data-source contract in getDataSourceMetadata
and cover the pruner's immutable branch
discard 30250766679 Preserve EQ pruning short circuit while caching IN bounds
discard 8389604539b Read pruning bounds once per IN predicate
discard 8c4214e270a DATA-3221 (13): prune segments from column metadata
instead of materializing them
discard cc4d67f14ee Assert the preprocess check leaves the segment schema
unbuilt on the server load path
discard d07e7ff815c Iterate column metadata directly during forward-index
planning
discard a26669b1042 DATA-3221 (12): make the new column-name accessor sorted
and null-safe
discard e25648b7409 DATA-3221 (11): list physical columns without building the
segment schema
discard ae1667b1327 Fix the column metadata mock in index loading tests
discard 0275bdea56b Adapt compression stats fixture to column metadata
iteration
discard 960bdd27a3d Read sparse child settings from column metadata arrays
discard b12f1bdd322 Stub column names in segment metadata test fixtures
discard 16f03fecde5 Stub the column accessors on the bare metadata mocks the
upsert and pruner tests build segments from
discard 2af42065b28 Make the derived column metadata map read-only and the SPI
defaults null-tolerant
discard 2e7479e2786 Adapt lazy loader and config test to column views
discard 755386bb956 Use existing column views instead of extra metadata APIs
discard 2273679ef51 DATA-3221 (10): publish the segment column arrays as one
immutable pair
discard 07bd2d2a264 DATA-3221 (10): hold segment column metadata in sorted
arrays, derive the map on demand
discard 54f539dfff9 Pack min/max words without boxing and tighten the layout
Javadoc
discard 962eb86dece Keep focused packed metadata coverage
discard 7d895786666 DATA-3221 (8+9 follow-up): drop the shared shape, pack
bitsPerElement instead
discard 9c49d3b66d6 DATA-3221 (8+9): shared column shape and primitive min/max
in ColumnMetadataImpl
discard e105b6ba7c3 Preserve sparse manifests through slim metadata and lazy
loading
discard 40d323de9d7 Invalidate the derived schema when the loader registers
virtual columns
discard 9c602b52a81 Use per-segment schema state in lazy metadata tests
discard f046b64ef42 Slim ColumnMetadataImpl to 72 bytes and derive the
per-segment Schema lazily
discard 73bf3edd64e Keep nested and sparse settings distinct during config
sharing
discard 7677d53bb1d Use current OpenStruct config constructors in materializer
tests
discard 11352519dc4 Keep lazy materialization compatible with the segment read
lifecycle
discard 6a9541b967f Keep rebased source imports in project order
discard 6041cd6972b Serialize v1/v2 index buffer access for lazy
materialization and cover destroy vs in-flight
discard 93f193c7199 Retain compact index configs for segment-specific loads
discard 88f4bcf1282 Reuse resolved index configs for lazy column
materialization
discard c8e5a00705a Materialize immutable-segment columns lazily behind an
opt-in instance config (default off)
add f818bae6d75 Fix controller job-limit leak and multicluster test port
allocation (#19773)
add 763ba2cb7e1 Use Node 24 for compatibility workflow tooling (#19796)
add 6f87b7150f4 Fix JDK 25 compilation in Zstandard decompressor (#19798)
add 85d90cb3db0 Add class-level Javadoc to ExprMinMaxObjectTest (#19802)
add ada7648609a Count TEXT_MATCH filters without materializing matching
doc ids (#19760)
add ee2ee576f8b Persist upsert snapshot diagnostics for replica divergence
triage (#19637)
add 94fc1432751 Normalize signed zero in transform test fixtures (#19803)
add 113f9ed26bc Bump org.ow2.asm:asm from 9.10.1 to 9.11 (#19813)
add f59ce704600 Bump org.xerial.snappy:snappy-java from 1.1.10.10 to
1.1.10.11 (#19812)
add ec8d00a5c44 Bump io.prometheus.jmx:collector from 1.6.0 to 1.7.0
(#19811)
add 62e7a2ae2ec Bump io.netty:netty-bom from 4.1.138.Final to
4.1.139.Final (#19810)
add 4d6e2681d11 Bump org.jline:jline from 4.4.6 to 4.4.7 (#19808)
add 4b7b358b9dd Push AND restrictions into composite filter children
(#19408)
add 910f9d5e494 Bump org.roaringbitmap:RoaringBitmap from 1.6.23 to 1.6.24
(#19809)
add 99251704d38 Materialize immutable-segment columns lazily behind an
opt-in instance config (default off)
add e465b529ece Reuse resolved index configs for lazy column
materialization
add 3dfb91fd9b6 Retain compact index configs for segment-specific loads
add 74f3279936e Serialize v1/v2 index buffer access for lazy
materialization and cover destroy vs in-flight
add ee7356fb5e4 Keep rebased source imports in project order
add 3a3b0933991 Keep lazy materialization compatible with the segment read
lifecycle
add d51a3d62695 Use current OpenStruct config constructors in materializer
tests
add 9576f825d22 Keep nested and sparse settings distinct during config
sharing
add 6dc2b588d33 Slim ColumnMetadataImpl to 72 bytes and derive the
per-segment Schema lazily
add 0c07f160880 Use per-segment schema state in lazy metadata tests
add b86cd4bc542 Invalidate the derived schema when the loader registers
virtual columns
add 8131274b0d8 Preserve sparse manifests through slim metadata and lazy
loading
add bd38bf3d5d8 DATA-3221 (8+9): shared column shape and primitive min/max
in ColumnMetadataImpl
add 5f178f149d4 DATA-3221 (8+9 follow-up): drop the shared shape, pack
bitsPerElement instead
add 228c0327954 Keep focused packed metadata coverage
add 8684f9852fd Pack min/max words without boxing and tighten the layout
Javadoc
add 35d55b7539d DATA-3221 (10): hold segment column metadata in sorted
arrays, derive the map on demand
add 44e9b41cc4a DATA-3221 (10): publish the segment column arrays as one
immutable pair
add 44ad73007b7 Use existing column views instead of extra metadata APIs
add 1ab53940c97 Adapt lazy loader and config test to column views
add 913d3a09f5b Make the derived column metadata map read-only and the SPI
defaults null-tolerant
add badf3008d91 Stub the column accessors on the bare metadata mocks the
upsert and pruner tests build segments from
add 6124c5e14c5 Stub column names in segment metadata test fixtures
add f508f703b72 Read sparse child settings from column metadata arrays
add a3058ad209f Adapt compression stats fixture to column metadata
iteration
add a4065ee9b16 Fix the column metadata mock in index loading tests
add 40ab1915b16 DATA-3221 (11): list physical columns without building the
segment schema
add 8f94beee736 DATA-3221 (12): make the new column-name accessor sorted
and null-safe
add ed35aa57389 Iterate column metadata directly during forward-index
planning
add 971f689948e Assert the preprocess check leaves the segment schema
unbuilt on the server load path
add 5f890c3e7e2 DATA-3221 (13): prune segments from column metadata
instead of materializing them
add a4dad38c2e3 Read pruning bounds once per IN predicate
add 31d4e200079 Preserve EQ pruning short circuit while caching IN bounds
add 659220751ea Honour the data-source contract in getDataSourceMetadata
and cover the pruner's immutable branch
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a500a345e7d)
\
N -- N -- N
refs/heads/xiangfu0/data-3221-11-metadata-only-pruning (659220751ea)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/pinot_compatibility_checks.yml | 8 +-
.github/workflows/pinot_compatibility_tests.yml | 4 +-
...ulti_stage_query_engine_compatibility_tests.yml | 4 +-
.../common/utils/config/QueryOptionsUtils.java | 21 +
.../core/util/ControllerZkHelixUtilsTest.java | 6 +
.../apache/pinot/core/common/BlockDocIdSet.java | 95 ++++-
.../data/manager/realtime/ConsumerCoordinator.java | 27 ++
.../realtime/RealtimeSegmentDataManager.java | 21 +-
.../dociditerators/MVScanDocIdIterator.java | 55 +--
.../pinot/core/operator/docidsets/AndDocIdSet.java | 121 +++++-
.../core/operator/docidsets/EmptyDocIdSet.java | 7 +
.../operator/docidsets/ExpressionDocIdSet.java | 5 +
.../core/operator/docidsets/MVScanDocIdSet.java | 5 +
.../core/operator/docidsets/MatchAllDocIdSet.java | 7 +
.../pinot/core/operator/docidsets/NotDocIdSet.java | 47 +++
.../pinot/core/operator/docidsets/OrDocIdSet.java | 90 +++-
.../core/operator/docidsets/SVScanDocIdSet.java | 5 +
.../core/operator/filter/AndFilterOperator.java | 12 +-
.../operator/filter/BaseColumnFilterOperator.java | 5 +-
.../operator/filter/CombinedFilterOperator.java | 7 +-
.../core/operator/filter/FilterOperatorUtils.java | 2 +-
.../core/operator/filter/MapFilterOperator.java | 7 +-
.../operator/filter/TextMatchFilterOperator.java | 8 +-
.../core/plan/maker/InstancePlanMakerImplV2.java | 41 ++
.../function/AggregationFunctionUtils.java | 3 +-
.../core/query/request/context/QueryContext.java | 14 +
.../manager/realtime/ConsumerCoordinatorTest.java | 25 ++
.../realtime/RealtimeSegmentDataManagerTest.java | 84 ++++
.../docidsets/AndDocIdSetPushdownTest.java | 465 +++++++++++++++++++++
.../function/BaseTransformFunctionTest.java | 5 +-
.../plan/maker/AndRestrictionPushdownModeTest.java | 129 ++++++
.../utils/exprminmax/ExprMinMaxObjectTest.java | 3 +
.../AndRestrictionPushdownNullQueriesTest.java | 201 +++++++++
.../queries/AndRestrictionPushdownQueriesTest.java | 109 +++++
.../pinot/queries/TextSearchQueriesTest.java | 18 +
.../tests/custom/GroupByTrimmingTest.java | 2 +-
.../BaseMultiClusterIntegrationTest.java | 8 +-
.../perf/BenchmarkAndRestrictionPushdown.java | 208 +++++++++
.../immutable/ImmutableSegmentImpl.java | 22 +-
.../io/compression/ZstandardDecompressor.java | 2 +-
.../index/readers/text/LuceneTextIndexReader.java | 123 ++++--
.../upsert/BasePartitionUpsertMetadataManager.java | 29 +-
.../pinot/segment/local/upsert/DocIdsSnapshot.java | 130 ++++++
.../upsert/PartitionUpsertMetadataManager.java | 5 +
.../segment/local/utils/LuceneTextIndexUtils.java | 89 ++++
.../readers/text/LuceneTextIndexCountTest.java | 289 +++++++++++++
.../BasePartitionUpsertMetadataManagerTest.java | 58 ++-
...rrentMapPartitionUpsertMetadataManagerTest.java | 3 +-
.../segment/local/upsert/DocIdsSnapshotTest.java | 136 ++++++
.../mutable/ThreadSafeMutableRoaringBitmap.java | 16 +-
.../index/reader/MultiColumnTextIndexReader.java | 19 +
.../segment/spi/index/reader/TextIndexReader.java | 18 +
.../pinot/server/api/resources/TablesResource.java | 30 +-
.../pinot/server/api/TablesResourceTest.java | 54 +++
.../apache/pinot/spi/utils/CommonConstants.java | 25 ++
pom.xml | 12 +-
56 files changed, 2797 insertions(+), 147 deletions(-)
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/operator/docidsets/AndDocIdSetPushdownTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/plan/maker/AndRestrictionPushdownModeTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/queries/AndRestrictionPushdownNullQueriesTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/queries/AndRestrictionPushdownQueriesTest.java
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkAndRestrictionPushdown.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/DocIdsSnapshot.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/readers/text/LuceneTextIndexCountTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/upsert/DocIdsSnapshotTest.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
