This is an automated email from the ASF dual-hosted git repository.
saurabhd336 pushed a change to branch release-1.0.0-rc
in repository https://gitbox.apache.org/repos/asf/pinot.git
discard d26cf0c480 [maven-release-plugin] prepare for next development
iteration
omit f29f33b710 [maven-release-plugin] prepare release release-1.0.0-rc0
add 1c61e4cebf Adding rule to skip SINGLE_VALUE agg function (#11383)
add decc4eba2f Properly handle shutdown of TableDataManager (#11380)
add 624478781d disable broker query killing test (#11367)
add c64040b017 add additional lucene index configs (#11354)
add a5a1641e60 Remove noop (#11205)
add dd22f8bca1 Add DECIMAL data type to orc record reader (#11377)
add 1060b4c785 Fixing the stale pinot ServerInstance in
_tableTenantServersMap (#11386)
add 534f158c50 Fix the thread safety issue for mutable forward index
(#11392)
add 1783f2a10a Add extract(quarter/dow/doy) support (#11388)
add 430734e950 UI: fix message shown after scheduling task (#11391)
add 6c89906dd6 Fix for issue when using MinIO as deep store (#11196)
add 2c023895a2 Fix intermediate stage routing for multi-stage engine
(#11393)
add 76e3899bcf Remove dedicated executor for exchange (#11394)
add 2236237f30 Upgrade jacoco plugin to use the new includeCurrentProject
flag (#11381)
add bfa4e9a36b Clean up ScalarTransformFunctionWrapper. (#11385)
add 4a263ea7d9 DISTINCT COUNT/SUM/AVG NULL support. (#11348)
add 3974b1681a Integration test exceptions fix (#11396)
add adbfd23edc Tests for CASE/WHEN query with NULL support (#11395)
add 5915aae4fa add configuration to fail server startup on non-good status
checker (#11347)
add afd5530230 allow passing freshness checker after an idle threshold
(#11345)
add 027943b6e4 [multistage] Retain timestampAdd and timestampDiff
functions during SqlNode to RexNode conversion (#11400)
add e8da9eb053 Fix RawStringDistinctExecutor integer overflow (#11403)
add 800220b860 VarianceAggregationFunction NULL support. (#11365)
add 32e47c7b00 Minor change to keep all distinct executor consistent
(#11405)
add e0d0aeb3d4 [multistage] Make join operator more resilient (#11401)
add ce7208add7 Split current current QueryConfig to
QueryRunnerServerConfig and QueryRuntimeConfig (#11410)
add f31ce3835c ForwardIndexHandler should only refer to columns in schema
(#11374)
add e6efb9c504 [Multi-stage] Minor fix on executor service (#11416)
add c17807325a [minor] typo (#11419)
add b670c552f5 Fix pipeline breaker error handling (#11411)
add 8ca6435209 Add broker validation for hybrid tableConfig creation
(#7908)
add 4ebed46eb1 [Multi-stage] Refactor query dispatcher to remove pipeline
breaker (#11422)
add c092b105e1 Fix a bug in SortOperator that prevents error block to be
propogated (#11425)
add 11276c6cd8 [Multi-stage] Limit groups for group-by executor (#11424)
add 37e70ecb77 [Multi-stage] Fix minor indentation error for physical plan
(#11423)
add d91e5db76c Add SchemaConformingTransformer to transform records with
varying keys to fit a table's schema without dropping fields. (#11210)
add 2b4036292e [logging] fix consume rate logging bug to respect 1 minute
threshold (#11421)
add 5faa792762 Support partition parallelism for partitioned table scan
(#11266)
add 399f033ec3 [Multi-stage] Remove PhysicalPlanContext and clean up
executor logic (#11439)
add 94d3b7dfe9 Use fromValue instead of valueOf to parse sse config
(#11428)
add 2e1e475697 Skip computation incase of Null Literal for case/when
transformation (#11444)
add c1693f8715 Allow adding exception to TransferableBlock without
converting it to DataBlock (#11456)
add 9e0752c267 [multistage]use all servers if no server instance found for
intermediate stage (#11452)
add 7a3d89cb19 Add array functions to v2 fn registry (#11455)
add 6c69be463f Fix the data type handling in multi-stage engine (#11453)
add cacb0b12e8 fixing canonicalize function name (#11460)
add 513f79df24 improve error message to suggest multi-stage engine when
possible (#11451)
add d7aed2e863 Funnel Count - Multiple Strategies (no partitioning
requisites) (#11092)
add fcb7172408 [Flaky-test] Fix UpsertTableSegmentPreloadIntegrationTest
(#11458)
add c793d505cd Upgrade hadoop version to 3.2.4 (#11369)
add 0817c5e848 Turn on expand identifier in SqlValidator config (#11457)
add f14700a1e2 only emit throttling metrics when throttling is allowed
(#11443)
add 5bee13a68a sql between function should be inclusive (#11467)
add b3b0b9ea8a Many places in the code, the entire response String is
read, then converted into JSON. Changing that to directly populate JSON using
the Stream to reduce the memory overhead. (#11465)
add dea0cd3c04 [multistage] fix the logic of parsing float literal (#11466)
add 86a6b78935 Validate ArrayToMv function presence only in leaf stage
(#11454)
add ff3dcbc3d9 Upgrading Apache Helix to latest version (#11450)
add a29e93b92f Disallow tenant tag override with upsert tables (#11468)
add a022e4194d Fix partition info in segment ZK metadata for changes in
stream partition count (#11476)
add fac2d9ff06 Use operator name instead of explain string in operatorId
(#11480)
add 43e785dfb3 Reuse broker/server hostname for multistage queryrunner
hostname (#11473)
add 2f48060d37 override leaf query limit to 0 if set explicitly from
outside (#11479)
add 385878711c Fix some misuse of full class name (#11481)
add 0ae7b8b565 create h2 tables for result comparison in
TPCHQueryIntegrationTest (#11389)
add d8d1e92cb9 Stream leaf stage blocks (#11472)
add c48d7f8087 set reader spec config and add test (#10731)
add a485778a8e [multistage] Evaluate literal expression during query
parsing (#11438)
add b2eb24a9a7 Fix return type for arrayLength (#11498)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (d26cf0c480)
\
N -- N -- N refs/heads/release-1.0.0-rc (b2eb24a9a7)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/pinot_tests.yml | 4 +-
contrib/pinot-fmpp-maven-plugin/pom.xml | 6 +-
pinot-broker/pom.xml | 5 +-
.../broker/broker/helix/BaseBrokerStarter.java | 15 +-
.../broker/broker/helix/ClusterChangeMediator.java | 6 +-
.../MultiStageBrokerRequestHandler.java | 75 +--
.../pinot/broker/routing/BrokerRoutingManager.java | 86 +--
.../instanceselector/BaseInstanceSelector.java | 37 +-
.../routing/instanceselector/InstanceSelector.java | 5 +
.../routing/instanceselector/SegmentStates.java | 8 +-
.../MultiStageBrokerRequestHandlerTest.java | 8 +-
pinot-clients/pinot-java-client/pom.xml | 5 +-
.../java/org/apache/pinot/client/BrokerCache.java | 4 +-
.../client/JsonAsyncHttpPinotClientTransport.java | 7 +-
pinot-clients/pinot-jdbc-client/pom.xml | 8 +-
.../response/ControllerResponseFuture.java | 11 +-
.../response/ControllerTenantBrokerResponse.java | 12 +-
.../client/controller/response/SchemaResponse.java | 9 +-
.../client/controller/response/TableResponse.java | 12 +-
pinot-clients/pom.xml | 6 +-
pinot-common/pom.xml | 43 +-
.../pinot/common/datablock/DataBlockUtils.java | 730 +++++++++++++-------
.../pinot/common/datablock/MetadataBlock.java | 14 +-
.../pinot/common/exception/QueryException.java | 6 +-
.../pinot/common/function/FunctionRegistry.java | 25 +
.../common/function/TransformFunctionType.java | 12 +-
.../function/scalar/ComparisonFunctions.java | 2 +-
.../common/function/scalar/ObjectFunctions.java | 22 +-
.../apache/pinot/common/request/BrokerRequest.java | 14 +-
.../apache/pinot/common/request/DataSource.java | 245 +++----
.../apache/pinot/common/request/Expression.java | 182 ++---
.../pinot/common/request/ExpressionType.java | 7 +-
.../org/apache/pinot/common/request/Function.java | 92 ++-
.../apache/pinot/common/request/Identifier.java | 59 +-
.../pinot/common/request/InstanceRequest.java | 42 +-
.../java/org/apache/pinot/common/request/Join.java | 307 ++++-----
.../org/apache/pinot/common/request/JoinType.java | 4 +-
.../org/apache/pinot/common/request/Literal.java | 102 ++-
.../apache/pinot/common/request/PinotQuery.java | 471 ++++++-------
.../apache/pinot/common/request/QuerySource.java | 9 +-
.../common/request/context/LiteralContext.java | 16 +
.../pinot/common/response/ProcessingException.java | 2 +-
.../org/apache/pinot/common/utils/DataSchema.java | 250 ++++---
.../common/utils/config/QueryOptionsUtils.java | 20 +-
.../pinot/common/utils/request/RequestUtils.java | 31 +-
.../apache/pinot/sql/parsers/CalciteSqlParser.java | 2 +-
.../sql/parsers/rewriter/ExprMinMaxRewriter.java | 18 +-
pinot-common/src/main/proto/plan.proto | 20 +-
.../apache/pinot/common/utils/DataSchemaTest.java | 35 -
.../common/utils/request/RequestUtilsTest.java | 29 +-
.../pinot/sql/parsers/CalciteSqlCompilerTest.java | 54 +-
pinot-common/src/thrift/query.thrift | 1 +
pinot-compatibility-verifier/pom.xml | 5 +-
pinot-connectors/pinot-flink-connector/pom.xml | 8 +-
.../connector/flink/sink/FlinkSegmentWriter.java | 4 +-
pinot-connectors/pinot-spark-2-connector/pom.xml | 5 +-
pinot-connectors/pinot-spark-3-connector/pom.xml | 5 +-
pinot-connectors/pinot-spark-common/pom.xml | 5 +-
pinot-connectors/pom.xml | 6 +-
pinot-controller/pom.xml | 5 +-
.../api/resources/PinotQueryResource.java | 15 +
.../helix/core/PinotHelixResourceManager.java | 170 ++---
.../realtime/PinotLLCRealtimeSegmentManager.java | 13 +-
.../main/resources/app/pages/TaskQueueTable.tsx | 4 +-
.../src/main/resources/app/requests/index.ts | 2 +-
.../main/resources/app/utils/PinotMethodUtils.ts | 4 +-
pinot-core/pom.xml | 5 +-
.../core/common/datablock/DataBlockBuilder.java | 289 +++-----
.../core/data/manager/BaseTableDataManager.java | 76 ++-
.../core/data/manager/InstanceDataManager.java | 6 -
.../manager/offline/DimensionTableDataManager.java | 10 +-
.../manager/offline/OfflineTableDataManager.java | 1 +
.../core/data/manager/realtime/IdleTimer.java | 71 ++
.../realtime/LLRealtimeSegmentDataManager.java | 60 +-
.../realtime/RealtimeConsumptionRateManager.java | 11 +-
.../manager/realtime/RealtimeTableDataManager.java | 29 +-
.../core/function/scalar/SketchFunctions.java | 5 +-
.../function/BaseBinaryGeoTransformFunction.java | 7 +-
.../transform/function/ScalarFunctions.java | 28 +
.../transform/function/StDistanceFunction.java | 2 +-
.../core/operator/InstanceResponseOperator.java | 4 +-
.../operator/blocks/InstanceResponseBlock.java | 22 +-
.../blocks/results/AggregationResultsBlock.java | 26 +-
.../operator/blocks/results/BaseResultsBlock.java | 14 +-
.../blocks/results/DistinctResultsBlock.java | 20 +-
.../blocks/results/ExceptionResultsBlock.java | 12 +-
.../blocks/results/ExplainResultsBlock.java | 24 +-
.../blocks/results/GroupByResultsBlock.java | 35 +-
.../blocks/results/MetadataResultsBlock.java | 12 +-
.../operator/blocks/results/ResultsBlockUtils.java | 8 +-
.../blocks/results/SelectionResultsBlock.java | 29 +-
.../core/operator/combine/BaseCombineOperator.java | 4 -
.../combine/BaseSingleBlockCombineOperator.java | 8 +-
.../operator/combine/GroupByCombineOperator.java | 2 +-
...xValueBasedSelectionOrderByCombineOperator.java | 6 +-
.../core/operator/query/AggregationOperator.java | 11 +-
.../query/DictionaryBasedDistinctOperator.java | 2 +-
.../core/operator/query/DistinctOperator.java | 10 +-
.../operator/query/EmptySelectionOperator.java | 9 +-
.../operator/query/FastFilteredCountOperator.java | 11 +-
.../query/FilteredAggregationOperator.java | 9 +-
.../operator/query/FilteredGroupByOperator.java | 4 +-
.../pinot/core/operator/query/GroupByOperator.java | 18 +-
.../query/LinearSelectionOrderByOperator.java | 5 +-
.../query/NonScanBasedAggregationOperator.java | 11 +-
.../core/operator/query/SelectionOnlyOperator.java | 5 +-
.../operator/query/SelectionOrderByOperator.java | 4 +-
.../streaming/BaseStreamingCombineOperator.java | 41 +-
.../streaming/StreamingGroupByCombineOperator.java | 2 +-
.../StreamingInstanceResponseOperator.java | 69 +-
.../streaming/StreamingSelectionOnlyOperator.java | 4 +-
.../transform/function/CaseTransformFunction.java | 25 +-
.../function/ExtractTransformFunction.java | 14 +-
.../function/LiteralTransformFunction.java | 12 +-
.../function/ScalarTransformFunctionWrapper.java | 40 +-
.../pinot/core/plan/AggregationPlanNode.java | 10 +-
.../apache/pinot/core/plan/CombinePlanNode.java | 39 +-
.../apache/pinot/core/plan/DistinctPlanNode.java | 2 +-
.../apache/pinot/core/plan/GroupByPlanNode.java | 6 +-
.../apache/pinot/core/plan/SelectionPlanNode.java | 2 +-
.../plan/StreamingInstanceResponsePlanNode.java | 14 +-
.../core/plan/maker/InstancePlanMakerImplV2.java | 20 +-
.../apache/pinot/core/plan/maker/PlanMaker.java | 6 +-
.../aggregation/DoubleAggregationResultHolder.java | 8 +-
.../aggregation/IntAggregateResultHolder.java | 8 +-
.../aggregation/ObjectAggregationResultHolder.java | 4 +-
.../function/AggregationFunctionFactory.java | 17 +-
.../BaseDistinctAggregateAggregationFunction.java | 319 +++++++--
.../function/DistinctAvgAggregationFunction.java | 4 +-
.../function/DistinctAvgMVAggregationFunction.java | 2 +-
.../function/DistinctCountAggregationFunction.java | 4 +-
.../DistinctCountMVAggregationFunction.java | 2 +-
.../function/DistinctSumAggregationFunction.java | 4 +-
.../function/DistinctSumMVAggregationFunction.java | 2 +-
.../function/FunnelCountAggregationFunction.java | 511 --------------
.../function/VarianceAggregationFunction.java | 84 ++-
.../function/funnel/AggregationStrategy.java | 167 +++++
.../funnel/BitmapAggregationStrategy.java} | 25 +-
.../function/funnel/BitmapMergeStrategy.java | 52 ++
.../funnel/BitmapResultExtractionStrategy.java | 85 +++
.../function/funnel/DictIdsWrapper.java} | 30 +-
.../funnel/FunnelCountAggregationFunction.java | 188 ++++++
.../FunnelCountAggregationFunctionFactory.java | 272 ++++++++
.../FunnelCountSortedAggregationFunction.java | 129 ++++
.../function/funnel/MergeStrategy.java} | 25 +-
.../function/funnel/PartitionedMergeStrategy.java} | 36 +-
.../function/funnel/ResultExtractionStrategy.java | 52 ++
.../function/funnel/SetMergeStrategy.java | 52 ++
.../funnel/SetResultExtractionStrategy.java | 94 +++
.../function/funnel/SortedAggregationResult.java | 67 ++
.../funnel/SortedAggregationStrategy.java} | 25 +-
.../funnel/ThetaSketchAggregationStrategy.java | 73 ++
.../function/funnel/ThetaSketchMergeStrategy.java | 62 ++
.../ThetaSketchResultExtractionStrategy.java} | 39 +-
...eRawBigDecimalSingleColumnDistinctExecutor.java | 2 +-
.../BaseRawBytesSingleColumnDistinctExecutor.java | 2 +-
.../BaseRawStringSingleColumnDistinctExecutor.java | 2 +-
.../RawIntSingleColumnDistinctOnlyExecutor.java | 2 +-
.../pinot/core/query/executor/QueryExecutor.java | 43 +-
.../executor/ResultsBlockStreamer.java} | 26 +-
.../query/executor/ServerQueryExecutorV1Impl.java | 29 +-
.../core/query/request/ServerQueryRequest.java | 7 +-
.../apache/pinot/core/routing/RoutingManager.java | 9 +-
.../pinot/core/transport/grpc/GrpcQueryServer.java | 3 +-
.../grpc/GrpcResultsBlockStreamer.java} | 33 +-
.../pinot/core/common/datablock/DataBlockTest.java | 56 +-
.../core/common/datablock/DataBlockTestUtils.java | 23 +-
.../core/data/manager/realtime/IdleTimerTest.java | 123 ++++
.../blocks/results/ResultsBlockUtilsTest.java | 8 +-
.../combine/CombineErrorOperatorsTest.java | 21 +-
.../function/ArrayBaseTransformFunctionTest.java | 4 +-
.../function/LiteralTransformFunctionTest.java | 19 +
.../query/scheduler/PrioritySchedulerTest.java | 5 +-
.../selection/SelectionOperatorServiceTest.java | 13 +-
.../apache/pinot/queries/AllNullQueriesTest.java | 2 +-
.../pinot/queries/BaseFunnelCountQueriesTest.java | 41 +-
.../pinot/queries/BigDecimalQueriesTest.java | 2 +-
.../org/apache/pinot/queries/ExprMinMaxTest.java | 10 +-
...Test.java => FunnelCountQueriesBitmapTest.java} | 34 +-
...> FunnelCountQueriesPartitionedSortedTest.java} | 22 +-
...java => FunnelCountQueriesPartitionedTest.java} | 28 +-
...tedTest.java => FunnelCountQueriesSetTest.java} | 34 +-
...java => FunnelCountQueriesThetaSketchTest.java} | 34 +-
.../queries/NullHandlingEnabledQueriesTest.java | 329 ++++++++++
.../apache/pinot/queries/TransformQueriesTest.java | 2 +-
pinot-distribution/pom.xml | 11 +-
pinot-integration-test-base/pom.xml | 5 +-
.../tests/BaseClusterIntegrationTest.java | 38 +-
.../tests/ClusterIntegrationTestUtils.java | 37 +-
pinot-integration-tests/pom.xml | 5 +-
.../tests/BaseClusterIntegrationTestSet.java | 20 -
.../tests/BaseRealtimeClusterIntegrationTest.java | 2 +-
.../tests/MultiStageEngineIntegrationTest.java | 93 ++-
.../tests/OfflineClusterIntegrationTest.java | 8 +-
...flineClusterMemBasedBrokerQueryKillingTest.java | 4 +-
.../UpsertTableSegmentPreloadIntegrationTest.java | 100 +--
.../integration/tests/custom/TimestampTest.java | 269 ++++++--
.../pinot/integration/tests/tpch/Constants.java | 28 +-
.../apache/pinot/integration/tests/tpch/README.md | 43 ++
.../tests/{ => tpch}/TPCHQueryIntegrationTest.java | 126 ++--
.../pinot/integration/tests/tpch/TblToAvro.java | 233 +++++++
.../src/test/resources/tpch/1.sql | 2 +-
.../src/test/resources/tpch/10.sql | 4 +-
.../src/test/resources/tpch/11.sql | 4 +-
.../src/test/resources/tpch/12.sql | 4 +-
.../src/test/resources/tpch/14.sql | 4 +-
.../src/test/resources/tpch/15.sql | 4 +-
.../src/test/resources/tpch/22.sql | 4 +-
.../src/test/resources/tpch/3.sql | 4 +-
.../src/test/resources/tpch/4.sql | 4 +-
.../src/test/resources/tpch/5.sql | 4 +-
.../src/test/resources/tpch/6.sql | 4 +-
.../src/test/resources/tpch/{7.sql => 7-h2.sql} | 6 +-
.../src/test/resources/tpch/7.sql | 6 +-
.../src/test/resources/tpch/{8.sql => 8-h2.sql} | 6 +-
.../src/test/resources/tpch/8.sql | 6 +-
.../src/test/resources/tpch/{9.sql => 9-h2.sql} | 2 +-
.../src/test/resources/tpch/9.sql | 2 +-
pinot-minion/pom.xml | 5 +-
pinot-perf/README.md | 4 +-
pinot-perf/pom.xml | 5 +-
.../pinot-batch-ingestion-common/pom.xml | 6 +-
.../pinot-batch-ingestion-hadoop/pom.xml | 17 +-
.../pinot-batch-ingestion-spark-2.4/pom.xml | 10 +-
.../pinot-batch-ingestion-spark-3.2/pom.xml | 6 +-
.../pinot-batch-ingestion-standalone/pom.xml | 6 +-
pinot-plugins/pinot-batch-ingestion/pom.xml | 6 +-
.../pinot-environment/pinot-azure/pom.xml | 6 +-
pinot-plugins/pinot-environment/pom.xml | 6 +-
pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 5 +-
pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 6 +-
pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 9 +-
pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 9 +-
.../apache/pinot/plugin/filesystem/S3PinotFS.java | 6 +-
pinot-plugins/pinot-file-system/pom.xml | 6 +-
.../pinot-input-format/pinot-avro-base/pom.xml | 5 +-
.../pinot-input-format/pinot-avro/pom.xml | 5 +-
.../pinot-input-format/pinot-clp-log/pom.xml | 5 +-
.../inputformat/clplog/CLPLogRecordExtractor.java | 4 +-
.../pinot-confluent-avro/pom.xml | 5 +-
pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 5 +-
.../inputformat/csv/CSVRecordExtractorConfig.java | 2 +-
.../pinot-input-format/pinot-json/pom.xml | 5 +-
pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 21 +-
.../plugin/inputformat/orc/ORCRecordReader.java | 12 +
.../inputformat/orc/ORCRecordExtractorTest.java | 22 +-
.../pinot-input-format/pinot-parquet/pom.xml | 31 +-
.../pinot-input-format/pinot-protobuf/pom.xml | 5 +-
.../pinot-input-format/pinot-thrift/pom.xml | 5 +-
pinot-plugins/pinot-input-format/pom.xml | 6 +-
.../pinot-metrics/pinot-dropwizard/pom.xml | 5 +-
pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 5 +-
pinot-plugins/pinot-metrics/pom.xml | 6 +-
.../pinot-minion-builtin-tasks/pom.xml | 5 +-
.../SegmentGenerationAndPushTaskExecutor.java | 2 +
.../SegmentGenerationAndPushTaskGeneratorTest.java | 58 +-
.../src/test/resources/dummyTable.json | 8 +
pinot-plugins/pinot-minion-tasks/pom.xml | 6 +-
.../pinot-segment-uploader-default/pom.xml | 6 +-
pinot-plugins/pinot-segment-uploader/pom.xml | 6 +-
.../pinot-segment-writer-file-based/pom.xml | 6 +-
pinot-plugins/pinot-segment-writer/pom.xml | 6 +-
.../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 6 +-
.../pinot-kafka-base/pom.xml | 6 +-
.../pinot-stream-ingestion/pinot-kinesis/pom.xml | 12 +-
.../pinot-stream-ingestion/pinot-pulsar/pom.xml | 6 +-
pinot-plugins/pinot-stream-ingestion/pom.xml | 6 +-
pinot-plugins/pom.xml | 11 +-
pinot-query-planner/pom.xml | 5 +-
.../apache/calcite/rel/hint/PinotHintOptions.java | 14 +
.../rules/PinotAggregateLiteralAttachmentRule.java | 7 +-
.../rel/rules/PinotEvaluateLiteralRule.java | 225 +++++++
.../calcite/rel/rules/PinotQueryRuleSets.java | 3 +
.../apache/calcite/rel/rules/PinotRuleUtils.java | 7 +
.../rules/PinotSingleValueAggregateRemoveRule.java | 59 ++
.../calcite/sql2rel/PinotConvertletTable.java | 85 +++
.../org/apache/pinot/query/QueryEnvironment.java | 4 +-
.../query/parser/CalciteRexExpressionParser.java | 29 +-
.../query/planner/PhysicalExplainPlanVisitor.java | 2 +-
.../query/planner/logical/LiteralHintUtils.java | 48 +-
.../planner/logical/PinotLogicalQueryPlanner.java | 26 +-
.../planner/logical/RelToPlanNodeConverter.java | 79 +--
.../pinot/query/planner/logical/RexExpression.java | 115 +---
.../query/planner/logical/RexExpressionUtils.java | 128 +++-
.../planner/physical/DispatchablePlanMetadata.java | 9 +
.../planner/physical/MailboxAssignmentVisitor.java | 60 +-
.../planner/physical/PinotDispatchPlanner.java | 18 +-
.../query/planner/plannode/AbstractPlanNode.java | 1 +
.../query/planner/plannode/AggregateNode.java | 3 +-
.../pinot/query/planner/plannode/FilterNode.java | 3 +-
.../pinot/query/planner/plannode/JoinNode.java | 10 +-
.../pinot/query/planner/plannode/ProjectNode.java | 4 +-
.../pinot/query/planner/plannode/ValueNode.java | 14 +-
.../pinot/query/planner/plannode/WindowNode.java | 14 +-
.../planner/serde/ProtoSerializationUtils.java | 12 +
.../validation/ArrayToMvValidationVisitor.java | 179 +++++
.../apache/pinot/query/routing/WorkerManager.java | 108 ++-
.../org/apache/pinot/query/validate/Validator.java | 3 +-
.../apache/pinot/query/QueryCompilationTest.java | 76 +--
.../pinot/query/QueryEnvironmentTestBase.java | 146 ++--
.../query/queries/ResourceBasedQueryPlansTest.java | 2 +
.../query/testutils/MockRoutingManagerFactory.java | 4 +-
.../src/test/resources/queries/AggregatePlans.json | 22 +
.../test/resources/queries/BasicQueryPlans.json | 52 ++
.../resources/queries/LiteralEvaluationPlans.json | 161 +++++
.../resources/queries/ValidationErrorPlan.json | 26 +
pinot-query-runtime/pom.xml | 5 +-
.../pinot/query/mailbox/GrpcSendingMailbox.java | 6 +-
.../query/mailbox/InMemorySendingMailbox.java | 6 +-
.../apache/pinot/query/mailbox/MailboxIdUtils.java | 11 +-
.../apache/pinot/query/mailbox/MailboxService.java | 62 +-
.../pinot/query/mailbox/ReceivingMailbox.java | 41 +-
.../apache/pinot/query/mailbox/SendingMailbox.java | 3 +-
.../query/mailbox/channel/ChannelManager.java | 6 +-
.../query/mailbox/channel/GrpcMailboxServer.java | 6 +-
.../apache/pinot/query/runtime/QueryRunner.java | 267 ++++----
.../query/runtime/blocks/TransferableBlock.java | 60 +-
.../runtime/blocks/TransferableBlockUtils.java | 8 -
.../runtime/executor/ExecutorServiceUtils.java | 78 +++
.../runtime/executor/OpChainSchedulerService.java | 195 ++----
.../runtime/executor/RoundRobinScheduler.java | 263 --------
.../query/runtime/operator/AggregateOperator.java | 88 +--
.../operator/BaseMailboxReceiveOperator.java | 68 +-
.../query/runtime/operator/FilterOperator.java | 42 +-
.../query/runtime/operator/HashJoinOperator.java | 191 ++++--
.../LeafStageTransferableBlockOperator.java | 461 +++++++------
.../runtime/operator/MailboxReceiveOperator.java | 43 +-
.../runtime/operator/MailboxSendOperator.java | 46 +-
.../query/runtime/operator/MultiStageOperator.java | 3 +-
.../operator/MultistageAggregationExecutor.java | 4 +-
.../operator/MultistageGroupByExecutor.java | 94 ++-
.../pinot/query/runtime/operator/OpChain.java | 59 +-
.../query/runtime/operator/OperatorStats.java | 8 +-
.../pinot/query/runtime/operator/SetOperator.java | 9 +-
.../pinot/query/runtime/operator/SortOperator.java | 20 +-
.../operator/SortedMailboxReceiveOperator.java | 59 +-
.../query/runtime/operator/TransformOperator.java | 21 +-
.../runtime/operator/WindowAggregateOperator.java | 83 +--
.../runtime/operator/exchange/BlockExchange.java | 101 +--
.../operator/exchange/BroadcastExchange.java | 7 +-
.../runtime/operator/exchange/HashExchange.java | 8 +-
.../runtime/operator/exchange/RandomExchange.java | 12 +-
.../operator/exchange/SingletonExchange.java | 7 +-
.../runtime/operator/operands/FilterOperand.java | 164 +++--
.../runtime/operator/operands/FunctionOperand.java | 75 ++-
.../runtime/operator/operands/LiteralOperand.java | 12 +-
.../operator/operands/ReferenceOperand.java | 23 +-
.../operator/operands/TransformOperand.java | 69 +-
.../operator/operands/TransformOperandFactory.java | 80 +++
.../runtime/operator/utils/AggregationUtils.java | 70 +-
.../query/runtime/operator/utils/AsyncStream.java | 73 ++
.../utils/BlockingMultiStreamConsumer.java | 252 +++++++
.../runtime/operator/utils/BlockingStream.java | 41 +-
.../query/runtime/operator/utils/TypeUtils.java | 89 ++-
.../runtime/plan/OpChainExecutionContext.java | 36 +-
.../query/runtime/plan/PhysicalPlanContext.java | 97 ---
.../query/runtime/plan/PhysicalPlanVisitor.java | 90 ++-
.../plan/pipeline/PipelineBreakerExecutor.java | 84 ++-
.../plan/pipeline/PipelineBreakerOperator.java | 95 ++-
.../plan/pipeline/PipelineBreakerResult.java | 9 +-
.../plan/server/ServerPlanRequestContext.java | 13 +-
.../plan/server/ServerPlanRequestUtils.java | 58 +-
.../plan/server/ServerPlanRequestVisitor.java | 10 +-
.../apache/pinot/query/service/QueryConfig.java | 78 ---
.../pinot/query/service/SubmissionService.java | 11 +-
.../query/service/dispatch/QueryDispatcher.java | 285 ++++----
.../pinot/query/service/server/QueryServer.java | 20 +-
.../apache/pinot/query/QueryServerEnclosure.java | 48 +-
.../pinot/query/mailbox/MailboxServiceTest.java | 145 ++--
.../runtime/blocks/TransferableBlockUtilsTest.java | 29 +-
.../executor/OpChainSchedulerServiceTest.java | 136 +---
.../runtime/executor/RoundRobinSchedulerTest.java | 185 ------
.../runtime/operator/AggregateOperatorTest.java | 48 +-
.../query/runtime/operator/FilterOperatorTest.java | 161 ++---
.../runtime/operator/HashJoinOperatorTest.java | 439 ++++++-------
.../LeafStageTransferableBlockOperatorTest.java | 375 ++++-------
.../runtime/operator/LiteralValueOperatorTest.java | 11 +-
.../operator/MailboxReceiveOperatorTest.java | 62 +-
.../runtime/operator/MailboxSendOperatorTest.java | 72 +-
.../pinot/query/runtime/operator/OpChainTest.java | 96 ++-
.../query/runtime/operator/OperatorTestUtil.java | 17 +-
.../query/runtime/operator/SortOperatorTest.java | 26 +-
.../operator/SortedMailboxReceiveOperatorTest.java | 83 +--
.../runtime/operator/TransformOperatorTest.java | 134 ++--
.../operator/WindowAggregateOperatorTest.java | 274 +++-----
.../operator/exchange/BlockExchangeTest.java | 54 +-
.../operator/exchange/BroadcastExchangeTest.java | 4 +-
.../operator/exchange/HashExchangeTest.java | 4 +-
.../operator/exchange/RandomExchangeTest.java | 4 +-
.../operator/exchange/SingletonExchangeTest.java | 10 +-
.../plan/pipeline/PipelineBreakerExecutorTest.java | 81 +--
.../runtime/{ => queries}/QueryRunnerTest.java | 194 +++---
.../runtime/{ => queries}/QueryRunnerTestBase.java | 60 +-
.../runtime/queries/ResourceBasedQueriesTest.java | 39 +-
.../service/dispatch/QueryDispatcherTest.java | 143 ++--
.../query/service/server/QueryServerTest.java | 41 +-
.../src/test/resources/log4j2.xml | 40 +-
.../src/test/resources/queries/Case.json | 28 +-
.../src/test/resources/queries/QueryHints.json | 12 +
pinot-segment-local/pom.xml | 5 +-
.../DistinctCountThetaSketchValueAggregator.java | 4 +-
.../local/data/manager/TableDataManager.java | 4 +-
.../immutable/ImmutableSegmentLoader.java | 2 +-
.../forward/FixedByteMVMutableForwardIndex.java | 7 +-
.../forward/FixedByteSVMutableForwardIndex.java | 5 +-
.../invertedindex/RealtimeLuceneTextIndex.java | 4 +-
.../recordtransformer/CompositeTransformer.java | 13 +-
.../SchemaConformingTransformer.java | 545 +++++++++++++++
.../creator/impl/text/LuceneTextIndexCreator.java | 13 +-
.../segment/index/loader/ForwardIndexHandler.java | 5 +
.../segment/index/text/TextIndexConfigBuilder.java | 11 +
.../local/segment/index/text/TextIndexType.java | 3 +-
.../pinot/segment/local/utils/IngestionUtils.java | 7 +
.../segment/local/utils/TableConfigUtils.java | 22 +
.../invertedindex/LuceneMutableTextIndexTest.java | 3 +-
.../NativeAndLuceneMutableTextIndexTest.java | 6 +-
.../SchemaConformingTransformerTest.java | 731 +++++++++++++++++++++
.../segment/store/FilePerIndexDirectoryTest.java | 8 +-
.../store/SingleFileIndexDirectoryTest.java | 8 +-
.../segment/local/utils/TableConfigUtilsTest.java | 51 ++
pinot-segment-spi/pom.xml | 5 +-
.../pinot/segment/spi/index/TextIndexConfig.java | 53 +-
.../spi/partition/ModuloPartitionFunction.java | 2 +-
.../segment/spi/index/TextIndexConfigTest.java | 14 +-
pinot-server/pom.xml | 5 +-
.../org/apache/pinot/server/conf/ServerConf.java | 7 +-
.../server/starter/helix/BaseServerStarter.java | 54 +-
.../FreshnessBasedConsumptionStatusChecker.java | 26 +-
.../starter/helix/HelixInstanceDataManager.java | 82 ++-
.../SegmentOnlineOfflineStateModelFactory.java | 127 ++--
.../pinot/server/worker/WorkerQueryServer.java | 28 +-
...FreshnessBasedConsumptionStatusCheckerTest.java | 137 +++-
pinot-spi/pom.xml | 5 +-
.../apache/pinot/spi/config/table/FieldConfig.java | 2 +
.../config/table/ingestion/IngestionConfig.java | 15 +
.../SchemaConformingTransformerConfig.java | 78 +++
.../apache/pinot/spi/data/readers/GenericRow.java | 7 +
.../pinot/spi/stream/StreamDataDecoderImpl.java | 7 +
.../org/apache/pinot/spi/utils/BooleanUtils.java | 31 +-
.../apache/pinot/spi/utils/CommonConstants.java | 90 ++-
.../java/org/apache/pinot/spi/utils/JsonUtils.java | 5 +
pinot-tools/pom.xml | 11 +-
.../tools/admin/command/StartBrokerCommand.java | 3 +-
.../tools/admin/command/StartServerCommand.java | 5 +-
.../admin/command/StartServiceManagerCommand.java | 15 +-
.../apache/pinot/tools/utils/PinotConfigUtils.java | 7 +-
.../batch/tpch/customer/rawdata/customer.avro | Bin 1741 -> 608 bytes
.../batch/tpch/lineitem/lineitem_schema.json | 6 +-
.../batch/tpch/lineitem/rawdata/lineitem.avro | Bin 1721 -> 1615 bytes
.../examples/batch/tpch/nation/rawdata/nation.avro | Bin 1668 -> 2529 bytes
.../examples/batch/tpch/orders/orders_schema.json | 2 +-
.../examples/batch/tpch/orders/rawdata/orders.avro | Bin 1301 -> 616 bytes
.../examples/batch/tpch/part/rawdata/part.avro | Bin 1502 -> 632 bytes
.../batch/tpch/partsupp/rawdata/partsupp.avro | Bin 1322 -> 1017 bytes
.../examples/batch/tpch/region/rawdata/region.avro | Bin 580 -> 630 bytes
.../batch/tpch/supplier/rawdata/supplier.avro | Bin 1466 -> 533 bytes
pom.xml | 107 ++-
457 files changed, 12758 insertions(+), 8339 deletions(-)
copy
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/OfflineTableDataManager.java
=>
pinot-common/src/test/java/org/apache/pinot/common/utils/request/RequestUtilsTest.java
(61%)
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IdleTimer.java
delete mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FunnelCountAggregationFunction.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/AggregationStrategy.java
copy
pinot-core/src/main/java/org/apache/pinot/core/query/{distinct/raw/RawIntSingleColumnDistinctOnlyExecutor.java
=> aggregation/function/funnel/BitmapAggregationStrategy.java} (53%)
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/BitmapMergeStrategy.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/BitmapResultExtractionStrategy.java
copy
pinot-core/src/main/java/org/apache/pinot/core/{data/manager/offline/OfflineTableDataManager.java
=> query/aggregation/function/funnel/DictIdsWrapper.java} (62%)
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/FunnelCountAggregationFunction.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/FunnelCountAggregationFunctionFactory.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/FunnelCountSortedAggregationFunction.java
copy
pinot-core/src/main/java/org/apache/pinot/core/{data/manager/offline/OfflineTableDataManager.java
=> query/aggregation/function/funnel/MergeStrategy.java} (63%)
copy
pinot-core/src/main/java/org/apache/pinot/core/{operator/blocks/results/MetadataResultsBlock.java
=> query/aggregation/function/funnel/PartitionedMergeStrategy.java} (51%)
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ResultExtractionStrategy.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/SetMergeStrategy.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/SetResultExtractionStrategy.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/SortedAggregationResult.java
copy
pinot-core/src/main/java/org/apache/pinot/core/query/{distinct/raw/RawIntSingleColumnDistinctOnlyExecutor.java
=> aggregation/function/funnel/SortedAggregationStrategy.java} (53%)
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchAggregationStrategy.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchMergeStrategy.java
copy
pinot-core/src/main/java/org/apache/pinot/core/{operator/blocks/results/MetadataResultsBlock.java
=> query/aggregation/function/funnel/ThetaSketchResultExtractionStrategy.java}
(51%)
copy
pinot-core/src/main/java/org/apache/pinot/core/{data/manager/offline/OfflineTableDataManager.java
=> query/executor/ResultsBlockStreamer.java} (66%)
copy
pinot-core/src/main/java/org/apache/pinot/core/{plan/StreamingInstanceResponsePlanNode.java
=> transport/grpc/GrpcResultsBlockStreamer.java} (50%)
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/IdleTimerTest.java
copy
pinot-core/src/test/java/org/apache/pinot/queries/{FunnelCountQueriesNonSortedTest.java
=> FunnelCountQueriesBitmapTest.java} (58%)
rename
pinot-core/src/test/java/org/apache/pinot/queries/{FunnelCountQueriesSortedTest.java
=> FunnelCountQueriesPartitionedSortedTest.java} (79%)
copy
pinot-core/src/test/java/org/apache/pinot/queries/{FunnelCountQueriesNonSortedTest.java
=> FunnelCountQueriesPartitionedTest.java} (66%)
copy
pinot-core/src/test/java/org/apache/pinot/queries/{FunnelCountQueriesNonSortedTest.java
=> FunnelCountQueriesSetTest.java} (60%)
rename
pinot-core/src/test/java/org/apache/pinot/queries/{FunnelCountQueriesNonSortedTest.java
=> FunnelCountQueriesThetaSketchTest.java} (58%)
copy
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/OfflineTableDataManager.java
=>
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/tpch/Constants.java
(52%)
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/tpch/README.md
rename
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/{ =>
tpch}/TPCHQueryIntegrationTest.java (54%)
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/tpch/TblToAvro.java
copy pinot-integration-tests/src/test/resources/tpch/{7.sql => 7-h2.sql} (87%)
copy pinot-integration-tests/src/test/resources/tpch/{8.sql => 8-h2.sql} (86%)
copy pinot-integration-tests/src/test/resources/tpch/{9.sql => 9-h2.sql} (91%)
create mode 100644
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/resources/dummyTable.json
create mode 100644
pinot-query-planner/src/main/java/org/apache/calcite/rel/rules/PinotEvaluateLiteralRule.java
create mode 100644
pinot-query-planner/src/main/java/org/apache/calcite/rel/rules/PinotSingleValueAggregateRemoveRule.java
create mode 100644
pinot-query-planner/src/main/java/org/apache/calcite/sql2rel/PinotConvertletTable.java
create mode 100644
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/validation/ArrayToMvValidationVisitor.java
create mode 100644
pinot-query-planner/src/test/resources/queries/LiteralEvaluationPlans.json
create mode 100644
pinot-query-planner/src/test/resources/queries/ValidationErrorPlan.json
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/ExecutorServiceUtils.java
delete mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/RoundRobinScheduler.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/operands/TransformOperandFactory.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/utils/AsyncStream.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/utils/BlockingMultiStreamConsumer.java
copy
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/OfflineTableDataManager.java
=>
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/utils/BlockingStream.java
(51%)
delete mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/PhysicalPlanContext.java
delete mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/QueryConfig.java
delete mode 100644
pinot-query-runtime/src/test/java/org/apache/pinot/query/runtime/executor/RoundRobinSchedulerTest.java
rename pinot-query-runtime/src/test/java/org/apache/pinot/query/runtime/{ =>
queries}/QueryRunnerTest.java (61%)
rename pinot-query-runtime/src/test/java/org/apache/pinot/query/runtime/{ =>
queries}/QueryRunnerTestBase.java (91%)
copy pinot-perf/README.md => pinot-query-runtime/src/test/resources/log4j2.xml
(57%)
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/SchemaConformingTransformer.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/recordtransformer/SchemaConformingTransformerTest.java
create mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/SchemaConformingTransformerConfig.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
