>From Ritik Raj <[email protected]>: Ritik Raj has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21352?usp=email )
Change subject: [NO ISSUE][COLUMN][ALGEBRA] Columnar dictionary encoding + aggregation optimizations (squashed) ...................................................................... [NO ISSUE][COLUMN][ALGEBRA] Columnar dictionary encoding + aggregation optimizations (squashed) Squashed feature branch: adaptive dictionary encoding + predicate pushdown for columnar storage, plus a set of decomposable-aggregation optimizer rules. All new optimizer rules are flag-gated OFF by default and SET-toggleable per request. Storage / columnar: - Adaptive dictionary encoding + IN-list (set-membership) and multi-column AND dictionary predicate pushdown for columnar datasets. Optimizer rules (flag-gated OFF; SET-able via SqlppCompilationProvider.getCompilerOptions whitelist): - Code-domain GROUP BY (IntroduceColumnarGroupByRule): fused columnar partial GROUP BY over per-component dictionary codes; SUM/MIN/MAX/COUNT/AVG, BIGINT + DOUBLE measures, multi-key. - Incremental rollup (IntroduceIncrementalRollupRule): hierarchical GROUPING SETS subset-chain computes each coarser level from the finer level's output; fires over CTE/JOIN; DISTINCT-safe (bails on non-decomposable). - Eager aggregation (IntroduceEagerAggregationRule): push a partial GROUP BY below a key-preserving inner join; CBO cost gate (collapse + fact-shuffle) guarantees no regression (bails when estimates absent). Flags (default OFF, SET-toggleable): org.apache.asterix.column.dictionary.enabled / .predicate.pushdown.enabled org.apache.asterix.column.dictionary.group.enabled org.apache.asterix.opt.incremental.rollup.enabled org.apache.asterix.opt.eager.aggregation.enabled Tests: CodeGroupBy*, IncrementalRollup*, EagerAggregation* runtime tests (correctness + plan-shape + gated perf). No on-disk format change; default behavior unchanged (all new paths OFF). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Change-Id: I131c098d241c9aefc7cfb85d729dea7a0ad41ca0 --- A asterixdb/asterix-algebra/EAGER-AGGREGATION-DESIGN.md A asterixdb/asterix-algebra/INCREMENTAL-ROLLUP-DESIGN.md A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/ColumnarPartialGroupByPOperator.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/SqlppCompilationProvider.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceEagerAggregationRule.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceIncrementalRollupRule.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/pushdown/ColumnarGroupByEligibility.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/pushdown/ColumnarGroupKeyFieldResolver.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/pushdown/IntroduceColumnarGroupByRule.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/CitiRollupPerfTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/CodeGroupByAvgTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/CodeGroupByBigintTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/CodeGroupByBroadenPerfTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/CodeGroupByPerfTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/CodeGroupBySqlTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/DictionaryEndToEndPerfTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/EagerAggregationTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/IncrementalRollupOverCteTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/IncrementalRollupPerfTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/IncrementalRollupTest.java A asterixdb/asterix-column/DICTIONARY-ENCODING.md A asterixdb/asterix-column/DICTIONARY-PERF.md A asterixdb/asterix-column/DICTIONARY-PUSHDOWN-NOTES.md A asterixdb/asterix-column/STRING-DICTIONARY-NOTES.md M asterixdb/asterix-column/pom.xml M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/assembler/AbstractValueAssembler.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/bytes/dictionary/DictionaryFormat.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/bytes/dictionary/LongDictionaryValuesReader.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/bytes/dictionary/LongDictionaryValuesWriter.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/bytes/dictionary/StringDictionaryValuesReader.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/bytes/dictionary/StringDictionaryValuesWriter.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/bytes/stream/in/PushbackByteInputStream.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/filter/FilterAccessorProvider.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/filter/dictionary/DictionaryCodeInterval.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/filter/dictionary/DictionaryColumnPredicate.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/filter/dictionary/DictionaryComparisonOp.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/filter/dictionary/DictionaryFilterDescriptor.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/filter/dictionary/DictionaryPredicatePushdown.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/filter/iterable/evaluator/ColumnIterableFilterEvaluatorFactory.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/filter/iterable/evaluator/DictionaryColumnIterableFilterEvaluator.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/ColumnAssembler.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/AggKind.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/CodeGroupByConfig.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/CodeGroupByTable.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/CodeResolver.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/ColumnGroupByAggregator.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/ColumnarGroupBySearchOperatorDescriptor.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/ColumnarGroupBySearchOperatorNodePushable.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/CompositeKey.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/GroupKeyOrdinalResolver.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/GroupKeySlot.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/PartialGroupTupleEmitter.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/query/group/ResolvedKey.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/tuple/QueryColumnTupleReference.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/values/IColumnValuesReader.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/values/reader/AbstractColumnValuesReader.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/values/reader/ColumnValueReaderFactory.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/values/reader/value/LongDictionaryValueReader.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/values/reader/value/StringDictionaryValueReader.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/values/writer/ColumnValuesWriterFactory.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/values/writer/LongDictionaryColumnValuesWriter.java A asterixdb/asterix-column/src/main/java/org/apache/asterix/column/values/writer/StringDictionaryColumnValuesWriter.java A asterixdb/asterix-column/src/test/java/org/apache/asterix/column/encdec/DictionaryEncodingBenchmark.java A asterixdb/asterix-column/src/test/java/org/apache/asterix/column/encdec/DictionaryGroupByBenchmarkTest.java A asterixdb/asterix-column/src/test/java/org/apache/asterix/column/encdec/DictionaryPushdownTest.java A asterixdb/asterix-column/src/test/java/org/apache/asterix/column/encdec/LongDictionaryValuesCodecTest.java A asterixdb/asterix-column/src/test/java/org/apache/asterix/column/encdec/StringDictionaryValuesCodecTest.java A asterixdb/asterix-column/src/test/java/org/apache/asterix/column/test/group/AdmDoubleSumFieldAggregatorFactory.java A asterixdb/asterix-column/src/test/java/org/apache/asterix/column/test/group/CodeGroupByDataflowTest.java A asterixdb/asterix-column/src/test/java/org/apache/asterix/column/test/group/CodeGroupBySpikeTest.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/ColumnarPartialGroupByDataSource.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/filter/ColumnFilterBuilder.java A asterixdb/docs/superpowers/plans/2026-06-16-broaden-code-group-by.md A asterixdb/docs/superpowers/specs/2026-06-16-broaden-code-group-by-design.md A docs/superpowers/plans/2026-06-14-code-group-by-stage0.md A docs/superpowers/plans/2026-06-14-code-group-by-stage1-impl.md A docs/superpowers/plans/2026-06-14-code-group-by-stage1a-spike.md A docs/superpowers/plans/2026-06-14-dictionary-in-pushdown-followups.md A docs/superpowers/plans/2026-06-14-dictionary-in-pushdown.md A docs/superpowers/specs/2026-06-14-code-group-by-design-options.md A docs/superpowers/specs/2026-06-14-code-group-by-stage0-1-spec.md A docs/superpowers/specs/2026-06-14-dictionary-setmembership-and-codegroupby-design.md M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/base/PhysicalOperatorTag.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/api/IColumnTupleIterator.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/lsm/tuples/AbstractColumnTupleReference.java 87 files changed, 16,092 insertions(+), 10 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/52/21352/1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21352?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: asterixdb Gerrit-Branch: lumina Gerrit-Change-Id: I131c098d241c9aefc7cfb85d729dea7a0ad41ca0 Gerrit-Change-Number: 21352 Gerrit-PatchSet: 1 Gerrit-Owner: Ritik Raj <[email protected]>
