This is an automated email from the ASF dual-hosted git repository.
jlli pushed a change to branch deep.store.dir.structure
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
omit 8878df5 Update LICENSE & NOTICE (#6944)
omit 9adbe55 Include complex-type-handling quickstarts in the assembler
(#6961)
omit 958b4d4 Enabling compatibility tests in the script (#6959)
omit 1209bcc Use stored types instead of derived types in Forward Indices.
(#6960)
omit 71e156c Make Pinot Broker/Server/Minion can start by just passing a
config file (#5446)
omit f15225f fixed showing all pinot servers in tenant details (#6934)
omit 0d6ca90 fixed UI crashing when server or brokers are offline (#6935)
omit 4d834ea Update JSON datatype test cases + minor JSON index bug fix.
(#6951)
omit 08b909c Move TableDataManager to segment-local to unblock partial
upsert feature (#6947)
omit c5bca8a fixed Pinot ui can't show all the pinot servers (#6956)
omit 5188d4e Add NETTY_CONNECTION_SEND_REQUEST_LATENCY to broker timer
metrics (#6936)
omit 16f94a1 Rename some complex type configs (#6953)
omit d63d6c5 Fix NPE issue when consumed kafka message is null or the
record value is null. (#6950)
omit f4ae3e0 Add collectionToJsonMode to schema inference (#6946)
omit e400566 Add the complex-type support to decoder/reader (#6945)
omit 7b44624 Fix the exception thrown in the case that a specified table
name does not exist (#6328) (#6765)
omit 6f4790f add command to infer pinot schema from json data (#6930)
omit fe322f5 Adding a new Controller API to retrieve ingestion status for
realtime… (#6890)
omit 57c1603 Fix CAST transform function for chained transforms (#6941)
omit 1b0bb72 Update Quickstart to take tmp dir as a parameter (#5445)
omit 9becc57 Implement getRecord for immutableSegmentImpl (#6924)
omit 98891a3 Fix rules to include DateTime column (#6937) (#6938)
omit 9fb6d4d Convert collections not unnested to json (#6918)
omit 08ed07c Simplify the schema check in SegmentConverter (#6931)
omit 5dac15b Clean up generated files for recommendation engine in case of
failure (#6926)
omit 5b7023a Add support for Long in Modulo partition function. (#6929)
omit c0983f7 Update time boundary only when segment is available on server
(#6925)
omit 6e7fed9 Enhance PinotSegmentRecordReader to preserve null values
(#6922)
omit 9c95b4c add complex-type support to avro-to-pinot schema inference
(#6928)
omit 59e342d Mitigate calcite NPE bug. (#6908)
omit e379292 Add correct yaml files for real time data(#6787) (#6916)
omit 1a38329 Add complex-type transformation to offline segment creation
(#6914)
omit e1d6ca4 Compare state of rate limiting with ignoring cases (#6917)
omit 9035323 Fix log4j2 for Quickstart. (#6919)
omit 07246b7 Fix update tableConfigs call for realtime (#6915)
omit 8102e43 Core Pinot Environment Provider Implementation Logic to fetch
Failure… (#6842)
omit 244d64d update default presto pinot image configs (#6913)
omit 7b4d9b5 initialize server tls regardless of uploader https settings
(#6911)
omit 086c3f6 Allow overriding delimiter in Complex type config (#6905)
omit 428e2a8 Update the javadoc for PinotSegmentRestletResource (#6903)
omit 046b255 Add the complex data type transformer for complex type
handling (#6845)
omit dce7479 Move maxBurstQps broker metric in createRateLimiter method
(#6898)
omit 9b44c21 Add config File support(#6787) (#6901)
omit 5faf33d Adjust threadCpuTime calculation (#6886)
omit d76d5e2 Add segment size rule to Recommendation Engine (#6869)
omit 185b249 Enhance JSON index to support nested array (#6877)
omit f064811 TableConfigs toJsonObject fix (#6900)
omit ff8700f Add debug endpoint for tables. (#6897)
omit f09600d Remove realtime segment metrics if it's destroyed (#6887)
omit d83f188 Allow multiple alias for the same selection column. (#6891)
omit d2961e7 [fix] update table config show error msg on exception (#6895)
omit bd4239f JSON column datatype support. (#6878)
omit d6f4415 fix integration tests for kafka ports (#6889)
omit 32e055b Validate DISTINCT query when convert Calcite tree into
PinotQuery (#6883)
omit 9b87787 Fix flaky test (#6893)
omit 2d86114 filter out javaagent from generator.sh JAVA_OPTS (#6884)
omit 3c40c78 add shell-script for quickstart auth (#6892)
omit bb25194 Update helm to override zk url when zk is disabled (#6888)
omit d0c678f Updating QueryRunner to return object of report summary
(#6885)
omit 4f0a7d2 Allow empty string in MV column (#6879)
omit 73426bc Add Zstandard compression support with JMH
benchmarking(#6804) (#6876)
omit fe596b6 Make zookeeper, kafka, controller, broker use random port for
integration tests (#6872)
This update removed existing revisions from the reference, leaving the
reference pointing at a previous point in the repository history.
* -- * -- N refs/heads/deep.store.dir.structure (fe10fc1)
\
O -- O -- O (8878df5)
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/pinot_tests.yml | 2 -
.github/workflows/scripts/.pinot_quickstart.sh | 100 +-
LICENSE | 73 +
LICENSE-binary | 109 +-
NOTICE-binary | 115 +-
compatibility-verifier/checkoutAndBuild.sh | 4 +-
compatibility-verifier/compCheck.sh | 250 +-
.../config/FeatureTest2-schema.json | 94 -
.../config/data/RealtimeFeatureTest1-data-00.csv | 11 -
... => feature-test-1-realtime-stream-config.json} | 2 +-
...-realtime.json => feature-test-1-realtime.json} | 8 +-
.../sample-test-suite/config/kafka/topic1.cfg | 18 +-
.../queries/feature-test-2-sql-realtime.queries | 62 -
.../feature-test-2-sql-realtime.results | 63 -
.../sample-test-suite/kafka-stream-produce-op.yaml | 13 +-
.../sample-test-suite/kafka-topic-create-op.yaml | 9 +-
.../sample-test-suite/offline-table-create-op.yaml | 10 +-
.../sample-test-suite/offline-table-delete-op.yaml | 10 +-
.../sample-test-suite/post-broker-rollback.yaml | 53 +-
.../post-controller-rollback.yaml | 63 +-
.../sample-test-suite/post-server-rollback.yaml | 53 +-
.../sample-test-suite/post-server-upgrade.yaml | 53 +-
.../sample-test-suite/pre-broker-upgrade.yaml | 53 +-
.../sample-test-suite/pre-controller-upgrade.yaml | 68 +-
.../sample-test-suite/pre-server-upgrade.yaml | 53 +-
.../sample-test-suite/query-op.yaml | 9 +-
.../realtime-table-create-op.yaml | 10 +-
.../realtime-table-delete-op.yaml | 10 +-
.../sample-test-suite/sample.yaml | 57 +
.../sample-test-suite/segment-create-op.yaml | 13 +-
.../sample-test-suite/segment-delete-op.yaml | 10 +-
.../pinot-presto/etc/catalog/pinot.properties | 7 +
.../etc/catalog/pinot_quickstart.properties | 6 +
docker/images/pinot/bin/generator.sh | 10 +-
kubernetes/helm/pinot/templates/_helpers.tpl | 4 +-
kubernetes/helm/pinot/values.yaml | 10 +-
...haus-animal-sniffer.txt => LICENSE-angular.txt} | 2 +-
licenses-binary/LICENSE-aopalliance.txt | 2 +
...s-animal-sniffer.txt => LICENSE-codemirror.txt} | 4 +-
licenses-binary/LICENSE-cpl-1.0.txt | 88 +
licenses-binary/LICENSE-edl-1.0.txt | 13 -
...ENSE-stax2.txt => LICENSE-esotericsoftware.txt} | 9 +-
licenses-binary/LICENSE-foundation.txt | 22 +
...ENSE-google-auth.txt => LICENSE-fusesource.txt} | 5 +-
...s-animal-sniffer.txt => LICENSE-handlebars.txt} | 4 +-
licenses-binary/LICENSE-jcabi.txt | 2 +-
licenses-binary/LICENSE-jcraft.txt | 32 +
licenses-binary/LICENSE-jline.txt | 32 +
...NSE-stax2.txt => LICENSE-jquery-datatables.txt} | 4 +-
.../LICENSE-jquery-requestAnimationFrame.txt | 22 +
licenses-binary/LICENSE-jquery-sizzle.txt | 36 +
licenses-binary/LICENSE-jquery.txt | 20 +
...eactive-streams.txt => LICENSE-js-beautify.txt} | 8 +-
licenses-binary/LICENSE-mimepull.txt | 28 +
.../{LICENSE-azure.txt => LICENSE-normalize.txt} | 16 +-
licenses-binary/LICENSE-paranamer.txt | 28 +
...SE-google-protobuf.txt => LICENSE-protobuf.txt} | 0
licenses-binary/LICENSE-py4j.txt | 26 +
licenses-binary/LICENSE-reflections.txt | 13 -
.../{LICENSE-scala-lang.txt => LICENSE-scala.txt} | 0
licenses-binary/LICENSE-underscore.txt | 23 +
licenses-binary/LICENSE-xmlenc.txt | 27 +
licenses-binary/LICENSE-zstd-jni.txt | 26 -
.../LICENSE-angular.txt | 2 +-
.../LICENSE-codemirror.txt | 4 +-
licenses/LICENSE-foundation.txt | 22 +
.../LICENSE-handlebars.txt | 12 +-
.../LICENSE-jquery-datatables.txt | 4 +-
licenses/LICENSE-jquery-requestAnimationFrame.txt | 22 +
licenses/LICENSE-jquery-sizzle.txt | 36 +
licenses/LICENSE-jquery.txt | 20 +
.../LICENSE-js-beautify.txt | 8 +-
.../LICENSE-normalize.txt | 16 +-
licenses/LICENSE-underscore.txt | 23 +
.../broker/broker/BrokerAdminApiApplication.java | 6 +-
...okerResourceOnlineOfflineStateModelFactory.java | 2 +-
.../broker/broker/helix/HelixBrokerStarter.java | 34 +-
.../HelixExternalViewBasedQueryQuotaManager.java | 47 +-
.../requesthandler/BaseBrokerRequestHandler.java | 45 +-
.../routing/timeboundary/TimeBoundaryManager.java | 13 +-
.../broker/broker/HelixBrokerStarterTest.java | 8 +-
...elixExternalViewBasedQueryQuotaManagerTest.java | 32 +-
.../broker/requesthandler/QueryValidationTest.java | 47 +
.../routing/segmentpruner/SegmentPrunerTest.java | 2 +-
.../timeboundary/TimeBoundaryManagerTest.java | 30 +-
pinot-common/pom.xml | 4 -
.../pinot/common/exception/QueryException.java | 13 +-
.../pinot/common/metrics/AbstractMetrics.java | 60 +-
.../apache/pinot/common/metrics/BrokerTimer.java | 4 -
.../apache/pinot/common/metrics/ServerTimer.java | 4 +-
.../response/broker/BrokerResponseNative.java | 2 -
.../org/apache/pinot/common/utils/DataSchema.java | 9 -
.../common/utils/FileUploadDownloadClient.java | 3 -
.../apache/pinot/common/utils/PinotDataType.java | 86 +-
.../org/apache/pinot/common/utils/ZkStarter.java | 18 +-
.../pinot/common/utils/request/RequestUtils.java | 6 +-
.../apache/pinot/sql/parsers/CalciteSqlParser.java | 35 -
.../apache/pinot/common/data/FieldSpecTest.java | 12 -
.../pinot/common/http/MultiGetRequestTest.java | 4 -
.../pinot/common/utils/PinotDataTypeTest.java | 98 +-
.../common/utils/config/TableConfigSerDeTest.java | 9 +-
.../pinot/sql/parsers/CalciteSqlCompilerTest.java | 76 -
.../apache/pinot/controller/ControllerConf.java | 24 +-
.../api/ControllerAdminApiApplication.java | 6 +-
.../pinot/controller/api/debug/TableDebugInfo.java | 207 -
.../api/resources/PinotBrokerRestletResource.java | 9 +-
.../api/resources/PinotSegmentRestletResource.java | 21 +-
.../api/resources/PinotTableRestletResource.java | 40 +-
.../api/resources/TableConfigsRestletResource.java | 18 +-
.../api/resources/TableDebugResource.java | 256 -
.../helix/core/PinotHelixResourceManager.java | 11 -
.../controller/recommender/RecommenderDriver.java | 31 +-
.../data/generator/StringGenerator.java | 26 +-
.../controller/recommender/io/ConfigManager.java | 11 -
.../controller/recommender/io/InputManager.java | 114 +-
.../realtime/provisioning/MemoryEstimator.java | 163 +-
.../controller/recommender/rules/AbstractRule.java | 7 -
.../recommender/rules/RulesToExecute.java | 16 +-
.../recommender/rules/impl/BloomFilterRule.java | 12 +-
.../recommender/rules/impl/FlagQueryRule.java | 13 +-
.../NoDictionaryOnHeapDictionaryJointRule.java | 2 +-
.../rules/impl/PinotTablePartitionRule.java | 68 +-
.../rules/impl/RealtimeProvisioningRule.java | 41 +-
.../recommender/rules/impl/SegmentSizeRule.java | 157 -
.../io/configs/SegmentSizeRecommendations.java | 84 -
.../rules/io/params/PartitionRuleParams.java | 12 +
.../io/params/RealtimeProvisioningRuleParams.java | 11 -
.../rules/io/params/RecommenderConstants.java | 19 +-
.../rules/io/params/SegmentSizeRuleParams.java | 81 -
.../utils/QueryInvertedSortedIndexRecommender.java | 6 +-
.../util/ConsumingSegmentInfoReader.java | 48 -
.../pinot/controller/util/FileIngestionHelper.java | 2 +-
.../app/components/Homepage/ClusterConfig.tsx | 1 -
.../app/components/Homepage/InstanceTable.tsx | 1 -
.../src/main/resources/app/pages/HomePage.tsx | 10 +-
.../src/main/resources/app/pages/Tenants.tsx | 5 +-
.../pinot/controller/ControllerTestUtils.java | 6 +-
.../api/ConsumingSegmentInfoReaderTest.java | 76 +-
.../api/TableConfigsRestletResourceTest.java | 14 +-
...ControllerPeriodicTaskStarterStatelessTest.java | 3 +-
.../pinot/controller/helix/ControllerTest.java | 19 +-
.../helix/PinotControllerModeStatelessTest.java | 9 +-
.../core/util/SegmentDeletionManagerTest.java | 3 -
.../controller/recommender/TestConfigEngine.java | 68 +-
.../realtime/provisioning/MemoryEstimatorTest.java | 5 +-
.../rules/impl/PinotTablePartitionRuleTest.java | 75 -
.../rules/impl/SegmentSizeRuleTest.java | 89 -
.../recommenderInput/BloomFilterInput.json | 7 +-
.../BloomFilterInputWithDateTimeColumn.json | 64 -
.../recommenderInput/DataSizeCalculationInput.json | 2 -
.../recommenderInput/EmptyQueriesInput.json | 4 +-
.../resources/recommenderInput/FlagQueryInput.json | 22 +-
.../resources/recommenderInput/InvalidInput1.json | 5 +-
.../resources/recommenderInput/InvalidInput2.json | 4 +-
.../recommenderInput/KafkaPartitionRuleInput.json | 4 +-
.../recommenderInput/KafkaPartitionRuleInput2.json | 7 +-
...NoDictionaryOnHeapDictionaryJointRuleInput.json | 5 -
.../PinotTablePartitionRuleInput.json | 7 +-
.../RealtimeProvisioningInput_dateTimeColumn.json | 5 +-
.../RealtimeProvisioningInput_timeColumn.json | 4 +-
.../recommenderInput/SegmentSizeRuleInput.json | 167 -
...gmentSizeRuleInput_noNeedToGenerateSegment.json | 168 -
.../SegmentSizeRuleInput_realtimeOnlyTable.json | 150 -
...put_ruleIsDisableButItNeedsToBeSilentlyRun.json | 150 -
.../recommenderInput/SortedInvertedIndexInput.json | 7 +-
.../VariedLengthDictionaryInput.json | 2 +-
.../core/data/manager/BaseTableDataManager.java | 4 +-
.../core/data/manager/InstanceDataManager.java | 1 -
.../core}/data/manager/SegmentDataManager.java | 2 +-
.../pinot/core}/data/manager/TableDataManager.java | 4 +-
.../manager/config}/TableDataManagerConfig.java | 9 +-
.../manager/offline/DimensionTableDataManager.java | 9 +-
.../offline/ImmutableSegmentDataManager.java | 2 +-
.../manager/offline/TableDataManagerProvider.java | 13 +-
.../realtime/HLRealtimeSegmentDataManager.java | 1 -
.../realtime/LLRealtimeSegmentDataManager.java | 41 +-
.../realtime/RealtimeSegmentDataManager.java | 11 +-
.../manager/realtime/RealtimeTableDataManager.java | 4 +-
.../apache/pinot/core/minion/SegmentConverter.java | 33 +-
.../apache/pinot/core/minion/SegmentPurger.java | 8 +-
.../core/minion/segment/MapperRecordReader.java | 7 +-
.../core/minion/segment/ReducerRecordReader.java | 5 +
.../core/operator/InstanceResponseOperator.java | 47 +-
.../operator/blocks/IntermediateResultsBlock.java | 9 -
.../core/operator/combine/BaseCombineOperator.java | 30 +-
.../operator/combine/CombineOperatorUtils.java | 3 +-
...xValueBasedSelectionOrderByCombineOperator.java | 6 +-
.../combine/SelectionOnlyCombineOperator.java | 2 +-
.../StreamingSelectionOnlyCombineOperator.java | 2 +-
.../transform/function/BaseTransformFunction.java | 24 +-
.../transform/function/CastTransformFunction.java | 194 +-
.../query/executor/ServerQueryExecutorV1Impl.java | 4 +-
.../core/query/reduce/BrokerReduceService.java | 6 +-
.../query/reduce/SelectionDataTableReducer.java | 2 +-
.../core/query/request/context/QueryContext.java | 22 +-
.../BrokerRequestToQueryContextConverter.java | 12 +-
.../query/selection/SelectionOperatorUtils.java | 34 +-
.../processing/framework/SegmentMapper.java | 2 -
.../pinot/core/transport/AsyncQueryResponse.java | 6 +-
.../core/transport/InstanceRequestHandler.java | 27 +-
.../apache/pinot/core/transport/QueryRouter.java | 2 +-
.../pinot/core/transport/ServerChannels.java | 18 +-
.../pinot/core/transport/ServerResponse.java | 17 +-
.../core}/upsert/TableUpsertMetadataManager.java | 7 +-
.../data/manager/BaseTableDataManagerTest.java | 4 +-
.../offline/DimensionTableDataManagerTest.java | 4 +-
.../realtime/LLRealtimeSegmentDataManagerTest.java | 4 +-
.../indexsegment}/IntermediateSegmentTest.java | 3 +-
.../MutableSegmentImplAggregateMetricsTest.java | 3 +-
.../MutableSegmentImplNullValueVectorTest.java | 3 +-
.../mutable/MutableSegmentImplTest.java | 3 +-
.../mutable/MutableSegmentImplTestUtils.java | 3 +-
.../mutable/MutableSegmentImplUpsertTest.java | 5 +-
.../operator/ThreadCpuTimeMeasurementTest.java | 65 -
.../function/BaseTransformFunctionTest.java | 30 -
.../function/CastTransformFunctionTest.java | 30 +-
.../core/query/executor/QueryExecutorTest.java | 4 +-
.../BrokerRequestToQueryContextConverterTest.java | 38 +-
.../framework/SegmentProcessingFrameworkTest.java | 18 +-
.../pinot/core/transport/QueryRoutingTest.java | 11 +-
.../apache/pinot/core/util/SchemaUtilsTest.java | 4 +-
.../pinot/core/util/TableConfigUtilsTest.java | 86 +-
.../queries/FSTBasedRegexpLikeQueriesTest.java | 4 +-
.../apache/pinot/queries/H3IndexQueriesTest.java | 2 +-
...ueriesTest.java => JsonMatchPredicateTest.java} | 116 +-
.../apache/pinot/queries/JsonMatchQueriesTest.java | 209 -
.../NoDictionaryCompressionQueriesTest.java | 375 -
.../queries/SegmentWithNullValueVectorTest.java | 4 +-
.../pinot/queries/TextSearchQueriesTest.java | 4 +-
.../data/test_null_value_vector_data.json | 0
.../data/test_null_value_vector_pinot_schema.json | 0
.../src/test/resources/data/test_upsert_data.json | 0
.../test/resources/data/test_upsert_schema.json | 0
pinot-distribution/pinot-assembly.xml | 6 -
.../org/apache/pinot/compat/tests/QueryOp.java | 7 +-
.../org/apache/pinot/compat/tests/SegmentOp.java | 4 +-
.../tests/BaseClusterIntegrationTest.java | 35 +-
.../tests/BaseClusterIntegrationTestSet.java | 13 +-
.../tests/BasicAuthBatchIntegrationTest.java | 18 +-
.../tests/BasicAuthRealtimeIntegrationTest.java | 5 +-
.../pinot/integration/tests/ClusterTest.java | 46 +-
...tlyOnceKafkaRealtimeClusterIntegrationTest.java | 4 +-
...ridClusterIntegrationTestCommandLineRunner.java | 4 +-
.../IngestionConfigHybridIntegrationTest.java | 2 +-
.../tests/JsonPathClusterIntegrationTest.java | 2 +-
.../tests/LLCRealtimeClusterIntegrationTest.java | 9 -
.../LuceneRealtimeClusterIntegrationTest.java | 2 +-
.../tests/MapTypeClusterIntegrationTest.java | 2 +-
.../tests/NullHandlingIntegrationTest.java | 18 -
.../tests/OfflineClusterIntegrationTest.java | 60 +-
...rDownloadLLCRealtimeClusterIntegrationTest.java | 8 -
.../tests/SegmentCompletionIntegrationTest.java | 12 +-
.../SegmentWriterUploaderIntegrationTest.java | 2 +-
.../tests/ServerStarterIntegrationTest.java | 6 +-
.../UpsertTableSegmentUploadIntegrationTest.java | 16 -
.../ControllerLeaderLocatorIntegrationTest.java | 3 +-
.../pinot/minion/MinionAdminApiApplication.java | 6 +-
.../org/apache/pinot/minion/MinionStarter.java | 19 +-
.../BenchmarkNoDictionaryIntegerCompression.java | 168 -
.../perf/BenchmarkNoDictionaryLongCompression.java | 169 -
.../BenchmarkNoDictionaryStringCompression.java | 181 -
.../pinot-environment/pinot-azure/pom.xml | 53 -
.../plugin/provider/AzureEnvironmentProvider.java | 164 -
.../provider/AzureEnvironmentProviderTest.java | 147 -
.../mock-imds-response-without-computenode.json | 118 -
.../mock-imds-response-without-faultDomain.json | 118 -
.../src/test/resources/mock-imds-response.json | 118 -
pinot-plugins/pinot-environment/pom.xml | 55 -
.../plugin/inputformat/avro/AvroSchemaUtil.java | 18 -
.../pinot/plugin/inputformat/avro/AvroUtils.java | 170 +-
.../plugin/inputformat/avro/AvroUtilsTest.java | 69 +-
.../test/resources/fake_avro_nested_schema.avsc | 70 -
.../protobuf/ProtoBufRecordExtractor.java | 22 +-
.../RealtimeToOfflineSegmentsTaskExecutorTest.java | 4 +-
.../filebased/FileBasedSegmentWriterTest.java | 20 +-
.../kafka09/server/KafkaDataServerStartable.java | 35 +-
.../kafka20/KafkaPartitionLevelConsumer.java | 7 +-
.../kafka20/server/KafkaDataServerStartable.java | 5 -
pinot-plugins/pom.xml | 1 -
.../indexsegment/immutable/EmptyIndexSegment.java | 3 +-
.../immutable/ImmutableSegmentImpl.java | 21 +-
.../indexsegment/mutable/MutableSegmentImpl.java | 7 +-
.../io/compression/ChunkCompressorFactory.java | 6 -
.../local/io/compression/ZstandardCompressor.java | 42 -
.../io/compression/ZstandardDecompressor.java | 43 -
.../local/partition/ModuloPartitionFunction.java | 6 +-
.../converter/RealtimeSegmentConverter.java | 19 +-
.../converter/RealtimeSegmentRecordReader.java | 82 +
.../RealtimeSegmentSegmentCreationDataSource.java | 28 +-
.../stats/RealtimeSegmentStatsContainer.java | 21 +-
.../local/realtime/impl/json/MutableJsonIndex.java | 63 +-
.../recordtransformer/ComplexTypeTransformer.java | 395 -
.../RecordReaderSegmentCreationDataSource.java | 7 -
.../impl/SegmentIndexCreationDriverImpl.java | 13 +-
.../impl/inv/json/BaseJsonIndexCreator.java | 8 +-
.../index/column/PhysicalColumnIndexContainer.java | 7 +-
.../ColumnMinMaxValueGenerator.java | 8 +-
.../defaultcolumn/BaseDefaultColumnHandler.java | 4 +-
.../readers/json/ImmutableJsonIndexReader.java | 119 +-
.../readers/MultiplePinotSegmentRecordReader.java | 170 +-
.../segment/readers/PinotSegmentColumnReader.java | 72 +-
.../segment/readers/PinotSegmentRecordReader.java | 211 +-
.../segment/readers/sort/PinotSegmentSorter.java | 81 +-
.../pinot/segment/local/utils/IngestionUtils.java | 22 -
.../segment/local/utils/TableConfigUtils.java | 3 -
.../local/partition/PartitionFunctionTest.java | 29 +-
.../ComplexTypeTransformerTest.java | 348 -
.../ExpressionTransformerTest.java | 10 +-
.../recordtransformer/RecordTransformerTest.java | 17 +-
.../segment/local/segment/index/JsonIndexTest.java | 37 +-
.../SegmentGenerationWithFilterRecordsTest.java | 2 +-
.../forward/FixedByteChunkSVForwardIndexTest.java | 10 -
.../forward/VarByteChunkSVForwardIndexTest.java | 14 -
.../index/loader/SegmentPreProcessorTest.java | 2 +-
.../segment/local/utils/IngestionUtilsTest.java | 14 +-
.../spi/compression/ChunkCompressionType.java | 2 +-
.../spi/creator/SegmentGeneratorConfig.java | 14 -
.../api/resources/SegmentMetadataFetcher.java | 2 +-
.../server/api/resources/TableSizeResource.java | 4 +-
.../pinot/server/api/resources/TablesResource.java | 4 +-
.../server/starter/helix/AdminApiApplication.java | 5 +-
.../starter/helix/HelixInstanceDataManager.java | 6 +-
.../server/starter/helix/HelixServerStarter.java | 83 +-
.../SegmentOnlineOfflineStateModelFactory.java | 4 +-
.../apache/pinot/server/api/AccessControlTest.java | 2 +-
.../apache/pinot/server/api/BaseResourceTest.java | 4 +-
.../org/apache/pinot/spi/config/TableConfigs.java | 25 +-
.../apache/pinot/spi/config/table/FieldConfig.java | 12 -
.../apache/pinot/spi/config/table/TableStatus.java | 70 -
.../config/table/ingestion/ComplexTypeConfig.java | 70 -
.../config/table/ingestion/IngestionConfig.java | 12 +-
.../java/org/apache/pinot/spi/data/FieldSpec.java | 9 -
.../java/org/apache/pinot/spi/data/Schema.java | 1 -
.../spi/data/readers/BaseRecordExtractor.java | 4 +-
.../apache/pinot/spi/data/readers/GenericRow.java | 49 -
.../pinot/spi/data/readers/RecordReader.java | 12 +-
.../apache/pinot/spi/env/PinotConfiguration.java | 14 +-
.../PinotEnvironmentProvider.java | 42 -
.../PinotEnvironmentProviderFactory.java | 93 -
.../pinot/spi/metrics/PinotMetricsRegistry.java | 1 -
.../spi/stream/StreamDataServerStartable.java | 5 -
.../apache/pinot/spi/utils/CommonConstants.java | 20 +-
.../java/org/apache/pinot/spi/utils/JsonUtils.java | 359 +-
.../java/org/apache/pinot/spi/utils/NetUtils.java | 56 -
.../pinot/spi/utils/PinotReflectionUtils.java | 20 +-
.../PinotEnvironmentProviderFactoryTest.java | 68 -
.../pinot/spi/utils/IngestionConfigUtilsTest.java | 8 +-
.../org/apache/pinot/spi/utils/JsonUtilsTest.java | 305 +-
pinot-spi/src/test/resources/json_util_test.json | 50 -
pinot-tools/pom.xml | 30 -
.../pinot/tools/BatchQuickstartWithMinion.java | 11 +-
.../org/apache/pinot/tools/GenericQuickstart.java | 2 +-
.../apache/pinot/tools/GitHubEventsQuickstart.java | 2 +-
.../org/apache/pinot/tools/HybridQuickstart.java | 19 +-
.../org/apache/pinot/tools/JoinQuickStart.java | 17 +-
.../apache/pinot/tools/JsonIndexQuickStart.java | 27 +-
.../OfflineComplexTypeHandlingQuickStart.java | 100 -
.../org/apache/pinot/tools/QuickStartBase.java | 35 -
.../java/org/apache/pinot/tools/Quickstart.java | 16 +-
.../RealtimeComplexTypeHandlingQuickStart.java | 117 -
.../pinot/tools/RealtimeJsonIndexQuickStart.java | 21 +-
.../org/apache/pinot/tools/RealtimeQuickStart.java | 19 +-
.../org/apache/pinot/tools/SegmentDumpTool.java | 7 +-
.../org/apache/pinot/tools/UpsertQuickStart.java | 17 +-
.../pinot/tools/admin/PinotAdministrator.java | 2 -
.../admin/command/AbstractBaseAdminCommand.java | 3 +
.../pinot/tools/admin/command/AddTableCommand.java | 2 +-
.../admin/command/AvroSchemaToPinotSchema.java | 45 +-
.../tools/admin/command/JsonToPinotSchema.java | 173 -
.../command/LaunchDataIngestionJobCommand.java | 2 +-
.../tools/admin/command/QuickStartCommand.java | 62 +-
.../command/RealtimeProvisioningHelperCommand.java | 6 +-
.../tools/admin/command/StartBrokerCommand.java | 14 +-
.../admin/command/StartControllerCommand.java | 15 +-
.../tools/admin/command/StartKafkaCommand.java | 2 +-
.../tools/admin/command/StartMinionCommand.java | 2 +-
.../tools/admin/command/StartServerCommand.java | 19 +-
.../admin/command/StartServiceManagerCommand.java | 16 +-
.../tools/admin/command/StartZookeeperCommand.java | 2 -
.../anonymizer/PinotDataAndQueryAnonymizer.java | 1 -
.../pinot/tools/perf/PerfBenchmarkDriver.java | 9 +-
.../org/apache/pinot/tools/perf/QueryRunner.java | 134 +-
.../converter/PinotSegmentToAvroConverter.java | 6 +-
.../converter/PinotSegmentToCsvConverter.java | 1 -
.../converter/PinotSegmentToJsonConverter.java | 1 -
.../pinot/tools/service/PinotServiceManager.java | 28 +-
.../PinotServiceManagerAdminApiApplication.java | 5 +-
.../pinot/tools/utils/KafkaStarterUtils.java | 23 +-
.../apache/pinot/tools/utils/PinotConfigUtils.java | 19 +-
.../src/main/resources/conf/pinot-controller.conf | 42 -
.../src/main/resources/conf/pinot-minion.conf | 39 -
.../src/main/resources/conf/pinot-server.conf | 42 -
.../src/main/resources/conf/quickstart-log4j2.xml | 76 +-
...bEvents_offline_complexTypeHandling_schema.json | 64 -
...s_offline_complexTypeHandling_table_config.json | 29 -
.../ingestionJobComplexTypeHandlingSpec.yaml | 139 -
.../batch/githubEvents/ingestionJobSpec.yaml | 2 +-
.../githubEvents_data.json | 0
.../githubEvents_data.json | 9996 --------------------
...eHandling_meetupRsvp_realtime_table_config.json | 40 -
.../complexTypeHandling_meetupRsvp_schema.json | 54 -
pom.xml | 6 -
402 files changed, 3128 insertions(+), 21044 deletions(-)
mode change 100755 => 100644 compatibility-verifier/checkoutAndBuild.sh
delete mode 100644
compatibility-verifier/sample-test-suite/config/FeatureTest2-schema.json
delete mode 100644
compatibility-verifier/sample-test-suite/config/data/RealtimeFeatureTest1-data-00.csv
rename
compatibility-verifier/sample-test-suite/config/{feature-test-2-realtime-stream-config.json
=> feature-test-1-realtime-stream-config.json} (79%)
rename
compatibility-verifier/sample-test-suite/config/{feature-test-2-realtime.json
=> feature-test-1-realtime.json} (87%)
rename pinot-tools/src/main/resources/conf/pinot-broker.conf =>
compatibility-verifier/sample-test-suite/config/kafka/topic1.cfg (67%)
delete mode 100644
compatibility-verifier/sample-test-suite/config/queries/feature-test-2-sql-realtime.queries
delete mode 100644
compatibility-verifier/sample-test-suite/config/query-results/feature-test-2-sql-realtime.results
copy docker/images/pinot-presto/etc/catalog/pinot.properties =>
compatibility-verifier/sample-test-suite/kafka-stream-produce-op.yaml (63%)
copy docker/images/pinot-presto/etc/catalog/pinot.properties =>
compatibility-verifier/sample-test-suite/kafka-topic-create-op.yaml (75%)
copy docker/images/pinot-presto/etc/catalog/pinot.properties =>
compatibility-verifier/sample-test-suite/offline-table-create-op.yaml (76%)
copy docker/images/pinot-presto/etc/catalog/pinot.properties =>
compatibility-verifier/sample-test-suite/offline-table-delete-op.yaml (76%)
copy docker/images/pinot-presto/etc/catalog/pinot.properties =>
compatibility-verifier/sample-test-suite/query-op.yaml (76%)
copy docker/images/pinot-presto/etc/catalog/pinot.properties =>
compatibility-verifier/sample-test-suite/realtime-table-create-op.yaml (75%)
copy docker/images/pinot-presto/etc/catalog/pinot.properties =>
compatibility-verifier/sample-test-suite/realtime-table-delete-op.yaml (75%)
create mode 100644 compatibility-verifier/sample-test-suite/sample.yaml
copy docker/images/pinot-presto/etc/catalog/pinot.properties =>
compatibility-verifier/sample-test-suite/segment-create-op.yaml (66%)
copy docker/images/pinot-presto/etc/catalog/pinot.properties =>
compatibility-verifier/sample-test-suite/segment-delete-op.yaml (77%)
copy licenses-binary/{LICENSE-codehaus-animal-sniffer.txt =>
LICENSE-angular.txt} (94%)
create mode 100644 licenses-binary/LICENSE-aopalliance.txt
copy licenses-binary/{LICENSE-codehaus-animal-sniffer.txt =>
LICENSE-codemirror.txt} (92%)
create mode 100644 licenses-binary/LICENSE-cpl-1.0.txt
delete mode 100644 licenses-binary/LICENSE-edl-1.0.txt
copy licenses-binary/{LICENSE-stax2.txt => LICENSE-esotericsoftware.txt} (64%)
create mode 100644 licenses-binary/LICENSE-foundation.txt
rename licenses-binary/{LICENSE-google-auth.txt => LICENSE-fusesource.txt}
(91%)
copy licenses-binary/{LICENSE-codehaus-animal-sniffer.txt =>
LICENSE-handlebars.txt} (95%)
create mode 100644 licenses-binary/LICENSE-jcraft.txt
create mode 100644 licenses-binary/LICENSE-jline.txt
copy licenses-binary/{LICENSE-stax2.txt => LICENSE-jquery-datatables.txt} (83%)
create mode 100644 licenses-binary/LICENSE-jquery-requestAnimationFrame.txt
create mode 100644 licenses-binary/LICENSE-jquery-sizzle.txt
create mode 100644 licenses-binary/LICENSE-jquery.txt
copy licenses-binary/{LICENSE-reactive-streams.txt => LICENSE-js-beautify.txt}
(75%)
create mode 100644 licenses-binary/LICENSE-mimepull.txt
copy licenses-binary/{LICENSE-azure.txt => LICENSE-normalize.txt} (61%)
create mode 100644 licenses-binary/LICENSE-paranamer.txt
rename licenses-binary/{LICENSE-google-protobuf.txt => LICENSE-protobuf.txt}
(100%)
create mode 100644 licenses-binary/LICENSE-py4j.txt
delete mode 100644 licenses-binary/LICENSE-reflections.txt
rename licenses-binary/{LICENSE-scala-lang.txt => LICENSE-scala.txt} (100%)
create mode 100644 licenses-binary/LICENSE-underscore.txt
create mode 100644 licenses-binary/LICENSE-xmlenc.txt
delete mode 100644 licenses-binary/LICENSE-zstd-jni.txt
copy licenses-binary/LICENSE-codehaus-animal-sniffer.txt =>
licenses/LICENSE-angular.txt (94%)
rename licenses-binary/LICENSE-codehaus-animal-sniffer.txt =>
licenses/LICENSE-codemirror.txt (92%)
create mode 100644 licenses/LICENSE-foundation.txt
copy licenses-binary/LICENSE-azure.txt => licenses/LICENSE-handlebars.txt (87%)
rename licenses-binary/LICENSE-stax2.txt =>
licenses/LICENSE-jquery-datatables.txt (83%)
create mode 100644 licenses/LICENSE-jquery-requestAnimationFrame.txt
create mode 100644 licenses/LICENSE-jquery-sizzle.txt
create mode 100644 licenses/LICENSE-jquery.txt
rename licenses-binary/LICENSE-reactive-streams.txt =>
licenses/LICENSE-js-beautify.txt (75%)
rename licenses-binary/LICENSE-azure.txt => licenses/LICENSE-normalize.txt
(61%)
create mode 100644 licenses/LICENSE-underscore.txt
delete mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/api/debug/TableDebugInfo.java
delete mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/TableDebugResource.java
delete mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/SegmentSizeRule.java
delete mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/io/configs/SegmentSizeRecommendations.java
delete mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/io/params/SegmentSizeRuleParams.java
delete mode 100644
pinot-controller/src/test/java/org/apache/pinot/controller/recommender/rules/impl/PinotTablePartitionRuleTest.java
delete mode 100644
pinot-controller/src/test/java/org/apache/pinot/controller/recommender/rules/impl/SegmentSizeRuleTest.java
delete mode 100644
pinot-controller/src/test/resources/recommenderInput/BloomFilterInputWithDateTimeColumn.json
delete mode 100644
pinot-controller/src/test/resources/recommenderInput/SegmentSizeRuleInput.json
delete mode 100644
pinot-controller/src/test/resources/recommenderInput/SegmentSizeRuleInput_noNeedToGenerateSegment.json
delete mode 100644
pinot-controller/src/test/resources/recommenderInput/SegmentSizeRuleInput_realtimeOnlyTable.json
delete mode 100644
pinot-controller/src/test/resources/recommenderInput/SegmentSizeRuleInput_ruleIsDisableButItNeedsToBeSilentlyRun.json
rename {pinot-segment-local/src/main/java/org/apache/pinot/segment/local =>
pinot-core/src/main/java/org/apache/pinot/core}/data/manager/SegmentDataManager.java
(97%)
rename {pinot-segment-local/src/main/java/org/apache/pinot/segment/local =>
pinot-core/src/main/java/org/apache/pinot/core}/data/manager/TableDataManager.java
(97%)
rename
{pinot-segment-local/src/main/java/org/apache/pinot/segment/local/data/manager
=>
pinot-core/src/main/java/org/apache/pinot/core/data/manager/config}/TableDataManagerConfig.java
(91%)
rename {pinot-segment-local/src/main/java/org/apache/pinot/segment/local =>
pinot-core/src/main/java/org/apache/pinot/core}/upsert/TableUpsertMetadataManager.java
(85%)
rename
{pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable
=>
pinot-core/src/test/java/org/apache/pinot/core/indexsegment}/IntermediateSegmentTest.java
(98%)
rename {pinot-segment-local/src/test/java/org/apache/pinot/segment/local =>
pinot-core/src/test/java/org/apache/pinot/core}/indexsegment/mutable/MutableSegmentImplAggregateMetricsTest.java
(98%)
rename {pinot-segment-local/src/test/java/org/apache/pinot/segment/local =>
pinot-core/src/test/java/org/apache/pinot/core}/indexsegment/mutable/MutableSegmentImplNullValueVectorTest.java
(97%)
rename {pinot-segment-local/src/test/java/org/apache/pinot/segment/local =>
pinot-core/src/test/java/org/apache/pinot/core}/indexsegment/mutable/MutableSegmentImplTest.java
(98%)
rename {pinot-segment-local/src/test/java/org/apache/pinot/segment/local =>
pinot-core/src/test/java/org/apache/pinot/core}/indexsegment/mutable/MutableSegmentImplTestUtils.java
(96%)
rename {pinot-segment-local/src/test/java/org/apache/pinot/segment/local =>
pinot-core/src/test/java/org/apache/pinot/core}/indexsegment/mutable/MutableSegmentImplUpsertTest.java
(95%)
delete mode 100644
pinot-core/src/test/java/org/apache/pinot/core/operator/ThreadCpuTimeMeasurementTest.java
rename
pinot-core/src/test/java/org/apache/pinot/queries/{JsonDataTypeQueriesTest.java
=> JsonMatchPredicateTest.java} (71%)
delete mode 100644
pinot-core/src/test/java/org/apache/pinot/queries/JsonMatchQueriesTest.java
delete mode 100644
pinot-core/src/test/java/org/apache/pinot/queries/NoDictionaryCompressionQueriesTest.java
copy {pinot-segment-local =>
pinot-core}/src/test/resources/data/test_null_value_vector_data.json (100%)
copy {pinot-segment-local =>
pinot-core}/src/test/resources/data/test_null_value_vector_pinot_schema.json
(100%)
rename {pinot-segment-local =>
pinot-core}/src/test/resources/data/test_upsert_data.json (100%)
copy {pinot-segment-local =>
pinot-core}/src/test/resources/data/test_upsert_schema.json (100%)
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkNoDictionaryIntegerCompression.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkNoDictionaryLongCompression.java
delete mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkNoDictionaryStringCompression.java
delete mode 100644 pinot-plugins/pinot-environment/pinot-azure/pom.xml
delete mode 100644
pinot-plugins/pinot-environment/pinot-azure/src/main/java/org/apache/pinot/plugin/provider/AzureEnvironmentProvider.java
delete mode 100644
pinot-plugins/pinot-environment/pinot-azure/src/test/java/org/apache/pinot/plugin/provider/AzureEnvironmentProviderTest.java
delete mode 100644
pinot-plugins/pinot-environment/pinot-azure/src/test/resources/mock-imds-response-without-computenode.json
delete mode 100644
pinot-plugins/pinot-environment/pinot-azure/src/test/resources/mock-imds-response-without-faultDomain.json
delete mode 100644
pinot-plugins/pinot-environment/pinot-azure/src/test/resources/mock-imds-response.json
delete mode 100644 pinot-plugins/pinot-environment/pom.xml
delete mode 100644
pinot-plugins/pinot-input-format/pinot-avro-base/src/test/resources/fake_avro_nested_schema.avsc
delete mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/compression/ZstandardCompressor.java
delete mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/compression/ZstandardDecompressor.java
create mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentRecordReader.java
delete mode 100644
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/ComplexTypeTransformer.java
delete mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/recordtransformer/ComplexTypeTransformerTest.java
delete mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/TableStatus.java
delete mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/ComplexTypeConfig.java
delete mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/environmentprovider/PinotEnvironmentProvider.java
delete mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/environmentprovider/PinotEnvironmentProviderFactory.java
delete mode 100644
pinot-spi/src/test/java/org/apache/pinot/spi/environmentprovider/PinotEnvironmentProviderFactoryTest.java
delete mode 100644 pinot-spi/src/test/resources/json_util_test.json
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/OfflineComplexTypeHandlingQuickStart.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/QuickStartBase.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/RealtimeComplexTypeHandlingQuickStart.java
delete mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/JsonToPinotSchema.java
delete mode 100644 pinot-tools/src/main/resources/conf/pinot-controller.conf
delete mode 100644 pinot-tools/src/main/resources/conf/pinot-minion.conf
delete mode 100644 pinot-tools/src/main/resources/conf/pinot-server.conf
delete mode 100644
pinot-tools/src/main/resources/examples/batch/githubEvents/githubEvents_offline_complexTypeHandling_schema.json
delete mode 100644
pinot-tools/src/main/resources/examples/batch/githubEvents/githubEvents_offline_complexTypeHandling_table_config.json
delete mode 100644
pinot-tools/src/main/resources/examples/batch/githubEvents/ingestionJobComplexTypeHandlingSpec.yaml
rename
pinot-tools/src/main/resources/examples/batch/githubEvents/{rawdata_json_index
=> rawdata}/githubEvents_data.json (100%)
delete mode 100644
pinot-tools/src/main/resources/examples/batch/githubEvents/rawdata_complexTypeHandling/githubEvents_data.json
delete mode 100644
pinot-tools/src/main/resources/examples/stream/meetupRsvp/complexTypeHandling_meetupRsvp_realtime_table_config.json
delete mode 100644
pinot-tools/src/main/resources/examples/stream/meetupRsvp/complexTypeHandling_meetupRsvp_schema.json
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
