This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a change to branch xiangfu0/data-3221-6-lazy-materialization
in repository https://gitbox.apache.org/repos/asf/pinot.git
omit c11b2378f9b Materialize immutable-segment columns lazily behind an
opt-in instance config (default off)
omit e068c8a5d55 Fold PhysicalColumnIndexContainer's IndexTypeMap into a
presence mask plus dense readers
omit caac2497be6 Delegate immutable DataSourceMetadata to ColumnMetadata
instead of snapshotting it
omit e265da96c85 Share equal segment-derived FieldSpec instances through a
weak interner
omit 74e7788dc62 Share parsed time-column names and verify custom defaults
omit e92d1ae4878 Fix metadata test import order
omit 0ffe5f315c8 Canonicalize the default null value and intern per-column
strings at metadata parse time
add 9ee1c681be8 Fix conflicting agent instructions and stale workflow
guidance (#19541)
add a6ea7b0debd Remove obsolete benchmarks and timing-only unit tests
(#19538)
add 66fca21494e Reuse fixed-row buffers for DataTable result construction
(#19533)
add 4290c50ff6b Bump org.mongodb:bson from 5.11.0 to 5.11.1 (#19545)
add 3b8759fe2c6 Bump com.google.cloud:libraries-bom from 26.88.0 to
26.88.1 (#19546)
add b5035a6cea8 Bump software.amazon.awssdk:bom from 2.54.13 to 2.54.16
(#19548)
add d7b8ce3b387 Bump org.apache.fory:fory-json from 1.7.1 to 1.7.2 (#19549)
add 4087449ec96 Bump org.roaringbitmap:RoaringBitmap from 1.6.21 to 1.6.23
(#19550)
add 4df89218b38 Bump io.netty:netty-bom from 4.1.137.Final to
4.1.138.Final (#19551)
add ea6b22dcfe4 Bump bouncycastle.version from 1.85 to 1.86 (#19547)
add 7fd2f1b157f PageCache warmup on Server Restarts and Segment Refresh
(#16033)
add 47a5f8671f6 Return HTTP 400 for DDL table validator rejections (#19544)
add 810d7985b7e Add Jupyter JupySQL tutorial for querying Pinot (#19374)
add 92360684523 Make filter count and bitmap shortcuts three-valued under
null handling (#19440)
add 6ac0a577da5 Reject QUALIFY in the single-stage engine instead of
silently ignoring it (#19553)
add 8228534299a Reuse compiled constant string casts during planning
(#19514)
add a3e25756e17 Read each broker's instance config once per refresh in
ExternalViewReader (#19567)
add 719e486ac1d Bump org.codehaus.mojo:animal-sniffer-annotations from
1.27 to 1.28 (#19559)
add d49d91f4252 Bump software.amazon.awssdk:bom from 2.54.16 to 2.54.17
(#19560)
add f74a76d5909 Bump org.codehaus.mojo:buildnumber-maven-plugin from 3.3.0
to 3.3.1 (#19561)
add ac708ce2b27 Bump org.codehaus.mojo:versions-maven-plugin from 2.21.0
to 2.22.0 (#19562)
add 52095c8eb6a Bump org.codehaus.mojo:javacc-maven-plugin from 3.2.0 to
3.2.1 (#19563)
add d3a4b6e527a Bump org.apache.zookeeper:zookeeper from 3.9.5 to 3.9.6
(#19564)
add 12f9ebb6027 Bump org.codehaus.mojo:exec-maven-plugin from 3.6.3 to
3.6.4 (#19565)
add a979a6a0d8e Bump org.codehaus.mojo:build-helper-maven-plugin from
3.6.1 to 3.6.2 (#19566)
add ee615670b3f Reduce per-segment allocations in broker routing (#19530)
add 4b06328c0cb Fix binary compatibility CI baseline selection (#19569)
add 051049b9be3 Report BrokerResourceMissing instead of a generic internal
error when MSE routing is absent (#19558)
add 81f1a03abf8 Bump org.apache.fory:fory-json from 1.7.2 to 1.7.3 (#19573)
add 2a07d3ec173 Bump at.yawk.lz4:lz4-java from 1.11.2 to 1.11.3 (#19574)
add 8e454baabaa fix(readers): don't leak column readers or skip segment
destroy on close() failure (#19572)
add 3b760adb71a Optimise the OPEN_STRUCT per-row consumption path, and
follow-up fixes from #18643 review (#19093)
add 22470be7320 Add a cluster config to rewrite exact DISTINCTCOUNT and
PERCENTILE into approximations (#19510)
add 42cd1d67b6f Tolerate concurrent deletes when sizing consumer and
segment directories (#19580)
add c1a26e91c1e Restore the default when an adaptive routing metric export
cluster config is removed (#19578)
add b319ffd5785 Share FieldSpecs and repeated values across segment
metadata loads (#19473)
add 087e51cef0a Delegate immutable DataSourceMetadata to ColumnMetadata
instead of snapshotting it
add 6876c57f43b Fold PhysicalColumnIndexContainer's IndexTypeMap into a
presence mask plus dense readers
add b4f8c648793 Materialize immutable-segment columns lazily behind an
opt-in instance config (default off)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (c11b2378f9b)
\
N -- N -- N refs/heads/xiangfu0/data-3221-6-lazy-materialization
(b4f8c648793)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.claude/agents/code-reviewer.md | 2 +-
.claude/skills/bench-compare/SKILL.md | 2 +-
.claude/skills/flaky-analyze/SKILL.md | 2 +-
.claude/skills/precommit/SKILL.md | 2 +-
.claude/skills/quickstart/SKILL.md | 2 +-
.claude/skills/review-architecture/SKILL.md | 2 +-
.claude/skills/review-concurrency-state/SKILL.md | 2 +-
.claude/skills/review-config-backcompat/SKILL.md | 2 +-
.claude/skills/review-correctness-nulls/SKILL.md | 2 +-
.claude/skills/review-naming-api/SKILL.md | 2 +-
.claude/skills/review-performance/SKILL.md | 2 +-
.claude/skills/review-process-scope/SKILL.md | 2 +-
.claude/skills/review-testing/SKILL.md | 2 +-
.claude/skills/run-test/SKILL.md | 2 +-
.github/workflows/pinot_compatibility_checks.yml | 28 +-
AGENTS.md | 91 +---
CLAUDE.md | 91 +---
contrib/jupyter-jupysql/README.md | 76 +++
contrib/jupyter-jupysql/pinot_jupysql_eda.ipynb | 215 +++++++++
.../jupyter-jupysql/requirements.txt | 13 +-
kb/agents/code-reviewer.md | 47 +-
kb/claude.md | 2 +-
kb/code-review-principles.md | 106 ++--
kb/plugin-modules.md | 69 +++
kb/skills/README.md | 151 +-----
kb/skills/bench-compare.md | 86 ++--
kb/skills/flaky-analyze.md | 32 +-
kb/skills/precommit.md | 63 +--
kb/skills/quickstart.md | 42 +-
kb/skills/review-architecture.md | 17 +-
kb/skills/review-concurrency-state.md | 24 +-
kb/skills/review-config-backcompat.md | 12 +-
kb/skills/review-correctness-nulls.md | 20 +-
kb/skills/review-naming-api.md | 13 +-
kb/skills/review-performance.md | 18 +-
kb/skills/review-process-scope.md | 19 +-
kb/skills/review-testing.md | 64 ++-
kb/skills/run-test.md | 26 +-
.../broker/broker/helix/BaseBrokerStarter.java | 10 +
.../apache/pinot/broker/querylog/QueryLogger.java | 6 +
.../ApproximateFunctionOverrideProvider.java | 161 +++++++
.../requesthandler/BaseBrokerRequestHandler.java | 10 +
.../BaseSingleStageBrokerRequestHandler.java | 98 ++--
.../MultiStageBrokerRequestHandler.java | 18 +
.../instanceselector/BalancedInstanceSelector.java | 16 +-
.../ReplicaGroupInstanceSelector.java | 15 +-
.../routing/manager/BaseBrokerRoutingManager.java | 19 +-
.../pinot/broker/querylog/QueryLoggerTest.java | 1 +
.../ApproximateFunctionOverrideProviderTest.java | 134 ++++++
.../broker/requesthandler/QueryOverrideTest.java | 56 ++-
.../instanceselector/InstanceSelectorTest.java | 62 +++
.../routing/manager/BrokerRoutingManagerTest.java | 103 ++++
.../apache/pinot/client/ExternalViewReader.java | 20 +-
.../client/admin/PageCacheWarmupAdminClient.java | 91 ++++
.../pinot/client/admin/PinotAdminClient.java | 11 +
.../pinot/client/ExternalViewReaderTest.java | 103 +++-
.../apache/pinot/common/metrics/BrokerMeter.java | 5 +
.../pinot/common/metrics/ControllerMeter.java | 3 +
.../apache/pinot/common/metrics/ServerMeter.java | 4 +
.../pinot/common/pagecache/WarmupQueryUtils.java | 497 +++++++++++++++++++
.../pinot/common/response/BrokerResponse.java | 13 +
.../response/broker/BrokerResponseNative.java | 19 +-
.../response/broker/BrokerResponseNativeV2.java | 18 +-
.../response/broker/CursorResponseNative.java | 5 +-
.../org/apache/pinot/common/utils/FileUtils.java | 22 +
.../common/utils/config/QueryOptionsUtils.java | 8 +
.../utils/config/QueryWorkloadConfigUtils.java | 58 +--
.../common/utils/config/TableConfigSerDeUtils.java | 13 +-
.../pinot/common/utils/helix/HelixHelper.java | 52 ++
.../apache/pinot/sql/parsers/CalciteSqlParser.java | 10 +
.../sql/parsers/parser/TableNameExtractor.java | 5 +
.../common/pagecache/WarmupQueryUtilsTest.java | 318 ++++++++++++
.../pinot/common/utils/FALFInternerTest.java | 71 +--
.../pinot/sql/parsers/CalciteSqlParserTest.java | 37 ++
.../sql/parsers/parser/TableNameExtractorTest.java | 14 +
.../apache/pinot/controller/ControllerConf.java | 13 +
.../pinot/controller/api/resources/Constants.java | 1 +
.../resources/PageCacheWarmupRestletResource.java | 290 +++++++++++
.../api/resources/PinotDdlRestletResource.java | 3 +-
.../helix/core/PinotHelixResourceManager.java | 6 +-
.../util/PageCacheWarmupControllerExecutor.java | 378 +++++++++++++++
.../api/PinotDdlRestletResourceTest.java | 39 ++
.../pinot/controller/helix/ControllerTest.java | 1 +
.../helix/core/retention/RetentionManagerTest.java | 79 ---
.../PageCacheWarmupControllerExecutorTest.java | 331 +++++++++++++
.../common/datatable/BaseDataTableBuilder.java | 9 +-
.../core/common/datatable/DataTableBuilder.java | 1 +
.../core/operator/filter/AndFilterOperator.java | 80 +++-
.../operator/filter/BaseColumnFilterOperator.java | 66 ++-
.../core/operator/filter/BaseFilterOperator.java | 65 ++-
.../operator/filter/BitmapBasedFilterOperator.java | 45 +-
.../core/operator/filter/BitmapCollection.java | 127 ++++-
.../operator/filter/ExpressionFilterOperator.java | 25 +-
.../core/operator/filter/FilterOperatorUtils.java | 43 +-
.../filter/InvertedIndexFilterOperator.java | 24 +-
.../core/operator/filter/MapFilterOperator.java | 10 +
.../core/operator/filter/NotFilterOperator.java | 33 +-
.../core/operator/filter/OrFilterOperator.java | 77 ++-
.../filter/RangeIndexBasedFilterOperator.java | 35 +-
.../filter/SortedIndexBasedFilterOperator.java | 19 +-
...istinctCountSmartSketchAggregationFunction.java | 77 ++-
.../DistinctCountSmartHLLAggregationFunction.java | 19 +-
...stinctCountSmartHLLPlusAggregationFunction.java | 19 +-
.../DistinctCountSmartULLAggregationFunction.java | 19 +-
.../PercentileSmartTDigestAggregationFunction.java | 69 ++-
.../apache/pinot/core/startree/StarTreeUtils.java | 46 +-
.../routing/stats/ServerRoutingStatsManager.java | 14 +-
.../core/common/datatable/DataTableSerDeTest.java | 66 +++
.../operator/filter/AndFilterOperatorTest.java | 111 +++++
.../core/operator/filter/BitmapCollectionTest.java | 91 ++++
.../filter/InvertedIndexFilterOperatorTest.java | 64 ++-
.../operator/filter/NotFilterOperatorTest.java | 81 +++-
.../core/operator/filter/OrFilterOperatorTest.java | 68 +++
.../function/AnyValueAggregationFunctionTest.java | 5 +-
...stinctCountSmartHLLAggregationFunctionTest.java | 126 +++++
...centileSmartTDigestAggregationFunctionTest.java | 76 +++
.../pinot/core/startree/v2/BaseStarTreeV2Test.java | 4 +-
.../stats/ServerRoutingStatsManagerTest.java | 41 +-
.../queries/NullHandlingEnabledQueriesTest.java | 210 ++++++++
.../queries/StarTreeNullHandlingQueriesTest.java | 210 ++++++++
...ApproximateFunctionOverrideIntegrationTest.java | 202 ++++++++
.../tests/NullHandlingIntegrationTest.java | 6 +-
.../tests/PageCacheWarmupIntegrationTest.java | 247 ++++++++++
pinot-perf/README.md | 19 +
pinot-perf/pom.xml | 20 -
.../pinot/perf/BenchmarkConstantCastPlanning.java | 145 ++++++
.../perf/BenchmarkFixedIntArrayOffHeapIdMap.java | 59 ---
.../perf/BenchmarkGroovyExpressionEvaluation.java | 190 --------
.../org/apache/pinot/perf/BenchmarkJsonKeyMap.java | 172 -------
.../perf/BenchmarkOffHeapDictionaryMemory.java | 133 ------
.../apache/pinot/perf/BenchmarkQueryEngine.java | 152 ------
.../pinot/perf/BenchmarkRoaringBitmapCreation.java | 217 ---------
.../pinot/perf/BenchmarkRoaringBitmapMapping.java | 328 -------------
.../pinot/perf/BenchmarkScalarCastPlanning.java | 151 ++++++
.../pinot/perf/ForwardIndexWriterBenchmark.java | 120 -----
.../java/org/apache/pinot/perf/LazyDataList.java | 163 -------
.../org/apache/pinot/perf/RawIndexBenchmark.java | 293 ------------
.../pinot/perf/StringDictionaryPerfTest.java | 189 --------
.../pinot/perf/aggregation/BenchmarkSumQuery.java | 121 -----
.../pinot/plugin/filesystem/HadoopPinotFSTest.java | 16 +-
.../PinotApproximateAggregateRewriteRule.java | 191 ++++++++
.../calcite/rel/rules/PinotQueryRuleSets.java | 4 +
.../apache/pinot/calcite/rex/PinotRexExecutor.java | 130 +++++
.../org/apache/pinot/query/QueryEnvironment.java | 31 ++
.../apache/pinot/query/context/PlannerContext.java | 14 +
.../apache/pinot/query/routing/WorkerManager.java | 15 +-
.../pinot/calcite/rex/PinotRexExecutorTest.java | 532 +++++++++++++++++++++
.../query/ApproximateAggregateRewriteTest.java | 162 +++++++
.../pinot/query/ConstantCastPlanningTest.java | 186 +++++++
.../pinot/query/routing/WorkerManagerTest.java | 178 ++++++-
.../resources/queries/WindowFunctionPlans.json | 15 +
.../test/resources/queries/WindowFunctions.json | 44 +-
.../immutable/ImmutableSegmentImpl.java | 8 +
.../indexsegment/mutable/MutableSegmentImpl.java | 14 +-
.../creator/impl/SegmentColumnarIndexCreator.java | 25 +-
.../impl/openstruct/OpenStructColumnSplitter.java | 57 ++-
.../stats/NoDictColumnStatisticsCollector.java | 18 +
.../openstruct/ImmutableOpenStructDataSource.java | 123 +++++
.../segment/index/openstruct/MutableKeyColumn.java | 23 +-
.../index/openstruct/MutableOpenStructIndex.java | 86 +++-
.../index/openstruct/OpenStructIndexType.java | 33 ++
.../segment/readers/PinotSegmentRecordReader.java | 49 +-
.../segment/store/SegmentLocalFSDirectory.java | 8 +-
.../mutable/MutableSegmentImplOpenStructTest.java | 106 ++++
.../openstruct/OpenStructColumnSplitterTest.java | 15 +-
.../segment/index/SegmentMetadataImplTest.java | 5 +
.../ImmutableOpenStructDataSourceTest.java | 389 ++++++++++++++-
.../openstruct/MutableOpenStructIndexTest.java | 155 +++++-
.../index/openstruct/OpenStructIndexTypeTest.java | 30 ++
.../index/vector/VectorSearchBenchmark.java | 389 ---------------
.../readers/PinotSegmentRecordReaderCloseTest.java | 90 ++++
.../pinot/segment/spi/AggregationFunctionType.java | 23 +-
.../apache/pinot/segment/spi/ColumnMetadata.java | 10 -
.../apache/pinot/segment/spi/SegmentMetadata.java | 6 -
.../spi/datasource/OpenStructDataSource.java | 26 +
.../spi/index/metadata/ColumnMetadataImpl.java | 6 +-
.../spi/index/metadata/ColumnMetadataImplTest.java | 43 ++
.../pinot/server/api/resources/TablesResource.java | 77 +++
.../pinot/server/starter/ServerInstance.java | 8 +
.../server/starter/helix/BaseServerStarter.java | 13 +-
.../server/warmup/PageCacheWarmupQueryUtils.java | 99 ++++
.../warmup/PageCacheWarmupServerQueryExecutor.java | 334 +++++++++++++
.../PageCacheWarmupServerQueryExecutorTest.java | 166 +++++++
.../spi/config/table/PageCacheWarmupConfig.java | 112 +++++
.../spi/config/table/PageCacheWarmupRequest.java | 65 +++
.../apache/pinot/spi/config/table/TableConfig.java | 21 +-
.../apache/pinot/spi/utils/CommonConstants.java | 46 +-
.../utils/builder/ControllerRequestURLBuilder.java | 4 +
.../spi/utils/builder/TableConfigBuilder.java | 9 +-
.../pinot/tools/admin/PinotAdministrator.java | 2 +
.../GeneratePageCacheWarmupQueriesCommand.java | 335 +++++++++++++
.../pinot/tools/pagecache/WarmupQuerySource.java | 214 +++++++++
.../pagecache/QueryLogWarmupQuerySourceTest.java | 91 ++++
pom.xml | 30 +-
194 files changed, 10727 insertions(+), 3756 deletions(-)
create mode 100644 contrib/jupyter-jupysql/README.md
create mode 100644 contrib/jupyter-jupysql/pinot_jupysql_eda.ipynb
copy docker/images/pinot-superset/.dockerignore =>
contrib/jupyter-jupysql/requirements.txt (86%)
create mode 100644 kb/plugin-modules.md
create mode 100644
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/ApproximateFunctionOverrideProvider.java
create mode 100644
pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/ApproximateFunctionOverrideProviderTest.java
create mode 100644
pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/admin/PageCacheWarmupAdminClient.java
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/pagecache/WarmupQueryUtils.java
create mode 100644
pinot-common/src/test/java/org/apache/pinot/common/pagecache/WarmupQueryUtilsTest.java
create mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PageCacheWarmupRestletResource.java
create mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/util/PageCacheWarmupControllerExecutor.java
create mode 100644
pinot-controller/src/test/java/org/apache/pinot/controller/util/PageCacheWarmupControllerExecutorTest.java
create mode 100644
pinot-core/src/test/java/org/apache/pinot/queries/StarTreeNullHandlingQueriesTest.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ApproximateFunctionOverrideIntegrationTest.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/PageCacheWarmupIntegrationTest.java
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkConstantCastPlanning.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkGroovyExpressionEvaluation.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkJsonKeyMap.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkOffHeapDictionaryMemory.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkQueryEngine.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkRoaringBitmapCreation.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkRoaringBitmapMapping.java
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkScalarCastPlanning.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/ForwardIndexWriterBenchmark.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/LazyDataList.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/RawIndexBenchmark.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/StringDictionaryPerfTest.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/aggregation/BenchmarkSumQuery.java
create mode 100644
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotApproximateAggregateRewriteRule.java
create mode 100644
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rex/PinotRexExecutor.java
create mode 100644
pinot-query-planner/src/test/java/org/apache/pinot/calcite/rex/PinotRexExecutorTest.java
create mode 100644
pinot-query-planner/src/test/java/org/apache/pinot/query/ApproximateAggregateRewriteTest.java
create mode 100644
pinot-query-planner/src/test/java/org/apache/pinot/query/ConstantCastPlanningTest.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplOpenStructTest.java
delete mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/vector/VectorSearchBenchmark.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentRecordReaderCloseTest.java
create mode 100644
pinot-server/src/main/java/org/apache/pinot/server/warmup/PageCacheWarmupQueryUtils.java
create mode 100644
pinot-server/src/main/java/org/apache/pinot/server/warmup/PageCacheWarmupServerQueryExecutor.java
create mode 100644
pinot-server/src/test/java/org/apache/pinot/server/warmup/PageCacheWarmupServerQueryExecutorTest.java
create mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/PageCacheWarmupConfig.java
create mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/PageCacheWarmupRequest.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/GeneratePageCacheWarmupQueriesCommand.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/pagecache/WarmupQuerySource.java
create mode 100644
pinot-tools/src/test/java/org/apache/pinot/tools/pagecache/QueryLogWarmupQuerySourceTest.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
