This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a change to branch pinot-ingestion-refactor
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
discard 3ffb018 Update Readme to refer to docs
discard c975433 Adding docs for pinot spark usage
discard e325428 Add linking
discard 12f1e2c upgrade version
discard 146d54a Adding support for retry and back-off
discard 45abb1b update with new API
discard 4fe4904 Adding config to control spark job push parallism
discard 171532c Fixing parent directory creation issue
discard 24d9549 don't use default conf
discard 89c4522 fixing class: com.databricks.backend.daemon.data.client.DBFS
not serializable issue
discard 32f1fa3 Ensure segment push job is Serializable
discard fcbbd47 Make spark job an option to parallel push segments
discard 78d0b5b Address comments
discard 0152259 Move segment tars recursively and override existed files
discard 5281119 Initial refactor
discard 511f1a9 Make temp directory with uuid appended
discard 1715be3 Make PushLocation Serializable
discard 2dadcd6 Use SparkContext.getOrCreate() to use shared SparkContext if
possible.
discard 8c72408 Initial commit for pinot-spark
add 1ba6205 Removing segment generation config from RecordReader
interface (#4866)
add 63ceb0c Adding pinot-spi module and moving record reader interface
(#4869)
add 2610755 Initial commit for pinot-spark
add 79132b4 Use SparkContext.getOrCreate() to use shared SparkContext if
possible.
add 9cb3710 Make PushLocation Serializable
add 8639e48 Make temp directory with uuid appended
add d8bb62d Initial refactor
add d55aa93 Move segment tars recursively and override existed files
add 581f6b5 Address comments
add db3640a Make spark job an option to parallel push segments
add 94c1d07 Ensure segment push job is Serializable
add af0bed1 fixing class: com.databricks.backend.daemon.data.client.DBFS
not serializable issue
add 5743dcf don't use default conf
add 991aa3a Fixing parent directory creation issue
add 66812d0 Adding config to control spark job push parallism
add 3c416c6 update with new API
add 3f3b9e7 Adding support for retry and back-off
add ac52a7a upgrade version
add 4bf8fb7 Add linking
add 64cc728 Adding docs for pinot spark usage
add aba5c00 Update Readme to refer to docs
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (3ffb018)
\
N -- N -- N refs/heads/pinot-ingestion-refactor (aba5c00)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../broker/api/resources/PinotClientRequest.java | 2 +-
.../routing/HelixExternalViewBasedRouting.java | 4 +-
.../HelixExternalViewBasedTimeBoundaryService.java | 2 +-
.../pinot/broker/routing/TimeBoundaryService.java | 2 +-
.../broker/broker/HelixBrokerStarterTest.java | 4 +-
.../requesthandler/BrokerRequestOptionsTest.java | 2 +-
...ixExternalViewBasedTimeBoundaryServiceTest.java | 4 +-
pinot-common/pom.xml | 4 ++
.../common/assignment/InstancePartitions.java | 2 +-
.../apache/pinot/common/config/BaseJsonConfig.java | 2 +-
.../pinot/common/config/PinotTaskConfig.java | 2 +-
.../common/config/SegmentPartitionConfig.java | 2 +-
.../apache/pinot/common/config/TableConfig.java | 2 +-
.../pinot/common/data/StarTreeIndexSpec.java | 2 +-
.../pinot/common/lineage/SegmentMergeLineage.java | 2 +-
.../pinot/common/metadata/ZKMetadataProvider.java | 2 +-
.../metadata/instance/InstanceZKMetadata.java | 8 +--
.../segment/LLCRealtimeSegmentZKMetadata.java | 8 +--
.../metadata/segment/OfflineSegmentZKMetadata.java | 8 +--
.../segment/RealtimeSegmentZKMetadata.java | 8 +--
.../metadata/segment/SegmentPartitionMetadata.java | 2 +-
.../common/metadata/segment/SegmentZKMetadata.java | 10 +--
.../SegmentZKMetadataCustomMapModifier.java | 2 +-
.../protocols/SegmentCompletionProtocol.java | 2 +-
.../request/transform/TransformExpressionTree.java | 2 +-
.../pinot/common/response/ServerInstance.java | 2 +-
.../response/broker/BrokerResponseNative.java | 2 +-
.../common/restlet/resources/ResourceUtils.java | 2 +-
.../pinot/common/segment/SegmentMetadata.java | 4 +-
.../apache/pinot/common/utils/AvroSchemaUtil.java | 84 ++++++++++++++++++++++
.../org/apache/pinot/common/utils/DataSchema.java | 3 +-
.../common/utils/FileUploadDownloadClient.java | 1 +
.../org/apache/pinot/common/utils/SchemaUtils.java | 4 +-
.../pinot/common/utils/helix/HelixHelper.java | 1 -
.../pinot/common/utils/time/TimeConverter.java | 2 +-
.../pql/parsers/pql2/ast/FunctionCallAstNode.java | 2 +-
.../org/apache/pinot/startree/hll/HllConfig.java | 2 +-
.../pinot/common/config/IndexingConfigTest.java | 2 +-
.../pinot/common/config/QuotaConfigTest.java | 2 +-
.../org/apache/pinot/common/config/TenantTest.java | 2 +-
.../pinot/common/data/DateTimeFormatSpecTest.java | 24 ++++---
.../common/data/DateTimeGranularitySpecTest.java | 1 +
.../apache/pinot/common/data/FieldSpecTest.java | 27 ++++---
.../org/apache/pinot/common/data/SchemaTest.java | 11 ++-
.../metadata/MetadataEqualsHashCodeTest.java | 10 +--
.../segment/ColumnPartitionMetadataTest.java | 2 +-
.../pinot/common/query/gen/AvroQueryGenerator.java | 7 +-
.../apache/pinot/common/utils/DataSchemaTest.java | 2 +-
.../pinot/common/utils/EqualityUtilsTest.java | 1 +
.../common/utils/FileUploadDownloadClientTest.java | 1 +
.../pinot/common/utils/time/TimeConverterTest.java | 4 +-
.../realtime/impl/kafka/KafkaConsumerFactory.java | 2 +-
.../impl/kafka/KafkaHighLevelStreamConfig.java | 2 +-
.../impl/kafka/KafkaLowLevelStreamConfig.java | 2 +-
.../impl/kafka/KafkaStreamLevelConsumer.java | 6 +-
.../realtime/impl/kafka2/KafkaConsumerFactory.java | 2 +-
.../kafka2/KafkaPartitionLevelStreamConfig.java | 2 +-
.../impl/kafka2/KafkaStreamLevelConsumer.java | 4 +-
.../impl/kafka2/KafkaStreamLevelStreamConfig.java | 2 +-
.../impl/kafka/KafkaAvroMessageDecoder.java | 4 +-
.../impl/kafka/KafkaJSONMessageDecoder.java | 10 +--
.../impl/kafka/KafkaJSONMessageDecoderTest.java | 8 +--
.../api/events/DefaultMetadataEventNotifier.java | 2 +-
.../api/events/MetadataEventNotifier.java | 2 +-
.../PinotInstanceAssignmentRestletResource.java | 2 +-
.../resources/PinotInstanceRestletResource.java | 2 +-
.../api/resources/PinotSchemaRestletResource.java | 4 +-
.../api/resources/PinotSegmentRestletResource.java | 2 +-
.../api/resources/PinotTableInstances.java | 2 +-
.../api/resources/PinotTableRestletResource.java | 2 +-
.../controller/api/resources/PinotTableSchema.java | 2 +-
.../api/resources/PinotTenantRestletResource.java | 2 +-
.../api/resources/PinotVersionRestletResource.java | 2 +-
.../controller/api/resources/PqlQueryResource.java | 2 +-
.../api/resources/ServerTableSizeReader.java | 2 +-
.../resources/WebApplicationExceptionMapper.java | 2 +-
.../helix/core/PinotHelixResourceManager.java | 2 +-
.../helix/core/minion/ClusterInfoProvider.java | 2 +-
.../controller/util/AutoAddInvertedIndex.java | 8 +--
...PinotInstanceAssignmentRestletResourceTest.java | 6 +-
.../api/PinotInstanceRestletResourceTest.java | 2 +-
.../api/PinotSchemaRestletResourceTest.java | 6 +-
.../api/PinotSegmentRestletResourceTest.java | 2 +-
.../api/PinotTableRestletResourceTest.java | 2 +-
.../api/PinotTenantRestletResourceTest.java | 2 +-
.../api/SegmentCompletionProtocolDeserTest.java | 2 +-
.../controller/api/ServerTableSizeReaderTest.java | 2 +-
.../pinot/controller/api/TableSizeReaderTest.java | 2 +-
.../pinot/controller/api/TableViewsTest.java | 2 +-
.../controller/helix/ControllerTenantTest.java | 2 +-
.../pinot/controller/helix/ControllerTest.java | 8 +--
pinot-core/pom.xml | 4 ++
.../apache/pinot/core/common/BlockMetadata.java | 2 +-
.../org/apache/pinot/core/common/BlockValSet.java | 2 +-
.../apache/pinot/core/common/DataBlockCache.java | 4 +-
.../pinot/core/common/DataSourceMetadata.java | 2 +-
.../core/common/RowBasedBlockValueFetcher.java | 2 +-
.../core/data/aggregator/AvgValueAggregator.java | 2 +-
.../core/data/aggregator/CountValueAggregator.java | 2 +-
.../DistinctCountHLLValueAggregator.java | 2 +-
.../core/data/aggregator/MaxValueAggregator.java | 2 +-
.../aggregator/MinMaxRangeValueAggregator.java | 2 +-
.../core/data/aggregator/MinValueAggregator.java | 2 +-
.../aggregator/PercentileEstValueAggregator.java | 2 +-
.../PercentileTDigestValueAggregator.java | 2 +-
.../core/data/aggregator/SumValueAggregator.java | 2 +-
.../core/data/aggregator/ValueAggregator.java | 2 +-
.../data/aggregator/ValueAggregatorFactory.java | 2 +-
.../data/function/FunctionExpressionEvaluator.java | 2 +-
.../manager/offline/OfflineTableDataManager.java | 2 +-
.../realtime/HLRealtimeSegmentDataManager.java | 4 +-
.../realtime/LLRealtimeSegmentDataManager.java | 4 +-
.../manager/realtime/RealtimeTableDataManager.java | 4 +-
.../pinot/core/data/readers/AvroRecordReader.java | 13 ++--
.../pinot/core/data/readers/CSVRecordReader.java | 19 ++---
.../core/data/readers/CSVRecordReaderConfig.java | 1 +
.../core/data/readers/GenericRowRecordReader.java | 12 ++--
.../pinot/core/data/readers/JSONRecordReader.java | 14 ++--
.../readers/MultiplePinotSegmentRecordReader.java | 18 ++---
.../data/readers/PinotSegmentColumnReader.java | 2 +-
.../data/readers/PinotSegmentRecordReader.java | 18 ++---
.../core/data/readers/RecordReaderFactory.java | 5 +-
.../pinot/core/data/readers/RecordReaderUtils.java | 10 +--
.../core/data/readers/ThriftRecordReader.java | 19 ++---
.../data/readers/ThriftRecordReaderConfig.java | 1 +
.../core/data/readers/sort/PinotSegmentSorter.java | 4 +-
.../recordtransformer/CompositeTransformer.java | 4 +-
.../recordtransformer/DataTypeTransformer.java | 6 +-
.../recordtransformer/ExpressionTransformer.java | 6 +-
.../recordtransformer/NullValueTransformer.java | 8 +--
.../core/data/recordtransformer/PinotDataType.java | 4 +-
.../data/recordtransformer/RecordTransformer.java | 2 +-
.../recordtransformer/SanitizationTransformer.java | 8 +--
.../data/recordtransformer/TimeTransformer.java | 8 +--
.../pinot/core/indexsegment/IndexSegment.java | 2 +-
.../pinot/core/indexsegment/IndexSegmentUtils.java | 2 +-
.../generator/SegmentGeneratorConfig.java | 16 ++---
.../immutable/ImmutableSegmentImpl.java | 6 +-
.../immutable/ImmutableSegmentLoader.java | 5 +-
.../core/indexsegment/mutable/MutableSegment.java | 2 +-
.../indexsegment/mutable/MutableSegmentImpl.java | 10 +--
.../pinot/core/minion/BackfillDateTimeColumn.java | 27 ++++---
.../pinot/core/minion/RawIndexConverter.java | 6 +-
.../apache/pinot/core/minion/SegmentConverter.java | 2 +-
.../apache/pinot/core/minion/SegmentPurger.java | 18 ++---
.../minion/rollup/MergeRollupSegmentConverter.java | 4 +-
.../core/minion/rollup/RollupRecordAggregator.java | 6 +-
.../rollup/aggregate/MaxValueAggregator.java | 2 +-
.../rollup/aggregate/SumValueAggregator.java | 2 +-
.../minion/rollup/aggregate/ValueAggregator.java | 2 +-
.../minion/segment/DefaultRecordPartitioner.java | 2 +-
.../core/minion/segment/MapperRecordReader.java | 12 ++--
.../core/minion/segment/RecordAggregator.java | 2 +-
.../core/minion/segment/RecordPartitioner.java | 2 +-
.../core/minion/segment/RecordTransformer.java | 2 +-
.../core/minion/segment/ReducerRecordReader.java | 12 ++--
.../operator/CombineGroupByOrderByOperator.java | 2 +-
.../core/operator/blocks/BlockMetadataImpl.java | 2 +-
.../core/operator/blocks/MultiValueBlock.java | 2 +-
.../core/operator/blocks/SingleValueBlock.java | 2 +-
.../dociditerators/SVScanDocIdIterator.java | 2 +-
.../core/operator/docvalsets/MultiValueSet.java | 2 +-
.../operator/docvalsets/ProjectionBlockValSet.java | 2 +-
.../core/operator/docvalsets/SingleValueSet.java | 2 +-
.../operator/docvalsets/TransformBlockValSet.java | 2 +-
.../predicate/EqualsPredicateEvaluatorFactory.java | 6 +-
.../predicate/InPredicateEvaluatorFactory.java | 6 +-
.../NotEqualsPredicateEvaluatorFactory.java | 6 +-
.../predicate/NotInPredicateEvaluatorFactory.java | 6 +-
.../predicate/PredicateEvaluatorProvider.java | 2 +-
.../predicate/RangePredicateEvaluatorFactory.java | 6 +-
.../RegexpLikePredicateEvaluatorFactory.java | 2 +-
.../operator/query/SelectionOrderByOperator.java | 4 +-
.../transform/TransformResultMetadata.java | 2 +-
.../transform/function/BaseTransformFunction.java | 2 +-
.../DateTimeConversionTransformFunction.java | 2 +-
.../function/ValueInTransformFunction.java | 2 +-
.../datetime/BaseDateTimeTransformer.java | 6 +-
.../datetime/DateTimeTransformerFactory.java | 6 +-
.../datetime/EpochToEpochTransformer.java | 4 +-
.../datetime/EpochToSDFTransformer.java | 4 +-
.../datetime/SDFToEpochTransformer.java | 4 +-
.../transformer/datetime/SDFToSDFTransformer.java | 4 +-
.../core/plan/maker/BrokerRequestPreProcessor.java | 2 +-
.../core/query/aggregation/DistinctTable.java | 2 +-
.../function/AvgAggregationFunction.java | 2 +-
.../function/DistinctAggregationFunction.java | 5 +-
.../function/DistinctCountAggregationFunction.java | 2 +-
.../DistinctCountHLLAggregationFunction.java | 2 +-
.../DistinctCountHLLMVAggregationFunction.java | 2 +-
.../DistinctCountMVAggregationFunction.java | 2 +-
.../function/MinMaxRangeAggregationFunction.java | 2 +-
.../function/PercentileEstAggregationFunction.java | 2 +-
.../PercentileTDigestAggregationFunction.java | 2 +-
.../function/customobject/SerializedHLL.java | 2 +-
.../NoDictionaryMultiColumnGroupKeyGenerator.java | 2 +-
.../NoDictionarySingleColumnGroupKeyGenerator.java | 2 +-
.../groupby/utils/ValueToIdMapFactory.java | 2 +-
.../core/query/pruner/AbstractSegmentPruner.java | 4 +-
.../query/pruner/ColumnValueSegmentPruner.java | 3 +-
.../core/query/reduce/GroupByDataTableReducer.java | 2 +-
.../query/selection/SelectionOperatorUtils.java | 2 +-
.../converter/RealtimeSegmentConverter.java | 8 +--
.../converter/RealtimeSegmentRecordReader.java | 19 ++---
.../converter/stats/RealtimeColumnStatistics.java | 2 +-
.../RealtimeSegmentSegmentCreationDataSource.java | 4 +-
.../core/realtime/impl/RealtimeSegmentConfig.java | 2 +-
.../dictionary/BytesOffHeapMutableDictionary.java | 4 +-
.../dictionary/BytesOnHeapMutableDictionary.java | 4 +-
.../impl/dictionary/MutableDictionaryFactory.java | 2 +-
.../dictionary/StringOffHeapMutableDictionary.java | 2 +-
.../dictionary/StringOnHeapMutableDictionary.java | 2 +-
.../stream/AvroRecordToPinotRowGenerator.java | 8 +--
.../pinot/core/realtime/stream/OffsetCriteria.java | 2 +-
.../realtime/stream/SimpleAvroMessageDecoder.java | 5 +-
.../pinot/core/realtime/stream/StreamConfig.java | 2 +-
.../realtime/stream/StreamConsumerFactory.java | 2 +-
.../realtime/stream/StreamDecoderProvider.java | 2 +-
.../core/realtime/stream/StreamLevelConsumer.java | 3 +-
.../core/realtime/stream/StreamMessageDecoder.java | 4 +-
.../RecordReaderSegmentCreationDataSource.java | 6 +-
.../segment/creator/SegmentCreationDataSource.java | 2 +-
.../pinot/core/segment/creator/SegmentCreator.java | 4 +-
.../creator/SegmentPreIndexStatsCollector.java | 2 +-
.../core/segment/creator/StatsCollectorConfig.java | 4 +-
.../creator/impl/SegmentColumnarIndexCreator.java | 12 ++--
.../creator/impl/SegmentDictionaryCreator.java | 4 +-
.../impl/SegmentIndexCreationDriverImpl.java | 10 +--
.../inv/OffHeapBitmapInvertedIndexCreator.java | 2 +-
.../stats/AbstractColumnStatisticsCollector.java | 2 +-
.../stats/BytesColumnPredIndexStatsCollector.java | 2 +-
.../stats/SegmentPreIndexStatsCollectorImpl.java | 6 +-
.../pinot/core/segment/index/ColumnMetadata.java | 16 ++---
.../core/segment/index/SegmentMetadataImpl.java | 6 +-
.../index/column/PhysicalColumnIndexContainer.java | 2 +-
.../index/data/source/ColumnDataSource.java | 2 +-
.../segment/index/loader/SegmentPreProcessor.java | 2 +-
.../loader/bloomfilter/BloomFilterHandler.java | 3 +-
.../ColumnMinMaxValueGenerator.java | 4 +-
.../defaultcolumn/BaseDefaultColumnHandler.java | 8 +--
.../defaultcolumn/DefaultColumnHandlerFactory.java | 2 +-
.../defaultcolumn/V1DefaultColumnHandler.java | 2 +-
.../defaultcolumn/V3DefaultColumnHandler.java | 4 +-
.../index/readers/BaseImmutableDictionary.java | 2 +-
.../segment/index/readers/BytesDictionary.java | 2 +-
.../index/readers/OnHeapStringDictionary.java | 2 +-
.../segment/index/readers/StringDictionary.java | 2 +-
.../name/NormalizedDateSegmentNameGenerator.java | 2 +-
.../virtualcolumn/DocIdVirtualColumnProvider.java | 2 +-
.../SingleStringVirtualColumnProvider.java | 2 +-
.../VirtualColumnProviderFactory.java | 6 +-
.../apache/pinot/core/startree/MetricBuffer.java | 6 +-
.../core/startree/OffHeapStarTreeBuilder.java | 8 +--
.../pinot/core/startree/StarTreeBuilder.java | 2 +-
.../pinot/core/startree/StarTreeBuilderConfig.java | 2 +-
.../apache/pinot/core/startree/hll/HllUtil.java | 2 +-
.../startree/v2/builder/BaseSingleTreeBuilder.java | 4 +-
.../v2/store/StarTreeDimensionDataSource.java | 2 +-
.../startree/v2/store/StarTreeLoaderUtils.java | 4 +-
.../v2/store/StarTreeMetricDataSource.java | 2 +-
.../core/transport/InstanceRequestHandler.java | 2 +-
.../org/apache/pinot/core/util/ArrayCopyUtils.java | 2 +-
.../java/org/apache/pinot/core/util/AvroUtils.java | 17 ++---
.../apache/pinot/core/util/trace/TraceContext.java | 2 +-
.../apache/pinot/core/common/DataFetcherTest.java | 10 +--
.../core/common/RealtimeNoDictionaryTest.java | 4 +-
.../function/FunctionExpressionEvaluatorTest.java | 2 +-
.../realtime/LLRealtimeSegmentDataManagerTest.java | 2 +-
.../readers/BackfillDateTimeRecordReaderTest.java | 21 +++---
.../core/data/readers/JSONRecordReaderTest.java | 2 +-
.../MultiplePinotSegmentRecordReaderTest.java | 15 ++--
.../data/readers/PinotSegmentRecordReaderTest.java | 17 ++---
.../pinot/core/data/readers/PinotSegmentUtil.java | 9 +--
.../data/readers/RecordReaderSampleDataTest.java | 6 +-
.../pinot/core/data/readers/RecordReaderTest.java | 7 +-
.../core/data/readers/RecordReaderUtilsTest.java | 12 ++--
.../core/data/readers/ThriftRecordReaderTest.java | 6 +-
.../recordtransformer/RecordTransformerTest.java | 10 +--
.../recordtransformer/TimeTransformerTest.java | 10 +--
.../generator/SegmentGeneratorConfigTest.java | 6 +-
.../MutableSegmentImplAggregateMetricsTest.java | 10 +--
.../MutableSegmentImplNullValueVectorTest.java | 6 +-
.../mutable/MutableSegmentImplTest.java | 8 +--
.../mutable/MutableSegmentImplTestUtils.java | 2 +-
.../minion/MergeRollupSegmentConverterTest.java | 14 ++--
.../pinot/core/minion/SegmentConverterTest.java | 14 ++--
.../pinot/core/minion/SegmentPurgerTest.java | 8 +--
.../function/BaseTransformFunctionTest.java | 10 +--
.../function/DateTruncTransformFunctionTest.java | 8 +--
...adataAndDictionaryAggregationPlanMakerTest.java | 6 +-
.../pinot/core/predicate/InPredicateTest.java | 2 +-
.../NoDictionaryEqualsPredicateEvaluatorsTest.java | 4 +-
.../NoDictionaryInPredicateEvaluatorTest.java | 4 +-
.../NoDictionaryRangePredicateEvaluatorTest.java | 5 +-
.../predicate/PredicateEvaluatorTestUtils.java | 1 -
.../impl/dictionary/MutableDictionaryTest.java | 4 +-
.../impl/fakestream/FakeStreamConfigUtils.java | 2 +-
.../impl/fakestream/FakeStreamConsumerFactory.java | 4 +-
.../impl/fakestream/FakeStreamLevelConsumer.java | 2 +-
.../impl/fakestream/FakeStreamMessageDecoder.java | 4 +-
.../stream/AvroRecordToPinotRowGeneratorTest.java | 8 +--
.../core/segment/index/ColumnMetadataTest.java | 4 +-
.../segment/index/creator/RawIndexCreatorTest.java | 10 +--
.../SegmentGenerationWithBytesTypeTest.java | 16 ++---
.../SegmentGenerationWithNullValueVectorTest.java | 10 +--
.../SegmentGenerationWithTimeColumnTest.java | 10 +--
.../index/creator/SegmentPartitionTest.java | 8 +--
.../core/segment/index/loader/LoaderTest.java | 10 +--
.../index/loader/SegmentPreProcessorTest.java | 4 +-
.../index/readers/ImmutableDictionaryTest.java | 8 +--
.../ImmutableDictionaryTypeConversionTest.java | 8 +--
.../core/startree/OffHeapStarTreeBuilderTest.java | 12 ++--
.../startree/StarTreeIndexTestSegmentHelper.java | 10 +--
.../OffHeapStarTreeBuilderWithHllFieldTest.java | 14 ++--
.../hll/SegmentWithHllIndexCreateHelper.java | 6 +-
.../pinot/core/startree/v2/AvgStarTreeV2Test.java | 2 +-
.../pinot/core/startree/v2/BaseStarTreeV2Test.java | 6 +-
.../core/startree/v2/CountStarTreeV2Test.java | 2 +-
.../v2/DistinctCountHLLStarTreeV2Test.java | 2 +-
.../pinot/core/startree/v2/MaxStarTreeV2Test.java | 2 +-
.../startree/v2/MinMaxRangeStarTreeV2Test.java | 2 +-
.../pinot/core/startree/v2/MinStarTreeV2Test.java | 2 +-
.../startree/v2/PercentileEstStarTreeV2Test.java | 2 +-
.../v2/PercentileTDigestStarTreeV2Test.java | 2 +-
.../v2/PreAggregatedAvgStarTreeV2Test.java | 2 +-
...reAggregatedDistinctCountHLLStarTreeV2Test.java | 2 +-
.../v2/PreAggregatedMinMaxRangeStarTreeV2Test.java | 2 +-
.../PreAggregatedPercentileEstStarTreeV2Test.java | 2 +-
...eAggregatedPercentileTDigestStarTreeV2Test.java | 2 +-
.../pinot/core/startree/v2/SumStarTreeV2Test.java | 2 +-
.../pinot/core/util/trace/TraceContextTest.java | 2 +-
...ByteSingleColumnMultiValueReaderWriterTest.java | 2 +-
.../pinot/queries/BaseMultiValueQueriesTest.java | 4 +-
.../pinot/queries/BaseSingleValueQueriesTest.java | 4 +-
.../apache/pinot/queries/DistinctQueriesTest.java | 8 +--
.../apache/pinot/queries/FastHllQueriesTest.java | 4 +-
...terSegmentAggregationMultiValueQueriesTest.java | 2 +-
...erSegmentAggregationSingleValueQueriesTest.java | 2 +-
.../queries/PercentileTDigestMVQueriesTest.java | 12 ++--
.../queries/PercentileTDigestQueriesTest.java | 12 ++--
.../pinot/queries/SerializedBytesQueriesTest.java | 8 +--
.../apache/pinot/queries/TransformQueriesTest.java | 8 +--
.../DefaultAggregationExecutorTest.java | 8 +--
.../DictionaryBasedGroupKeyGeneratorTest.java | 8 +--
.../groupby/NoDictionaryGroupKeyGeneratorTest.java | 11 ++-
.../query/pruner/ColumnValueSegmentPrunerTest.java | 2 +-
.../converter/RealtimeSegmentConverterTest.java | 10 +--
.../v1/creator/BitmapInvertedIndexCreatorTest.java | 4 +-
.../segments/v1/creator/DictionariesTest.java | 10 +--
.../v1/creator/OnHeapDictionariesTest.java | 8 +--
.../segments/v1/creator/SegmentTestUtils.java | 21 +++---
pinot-hadoop | 1 -
.../pinot/hadoop/io/JsonPinotOutputFormat.java | 6 +-
.../apache/pinot/hadoop/io/PinotOutputFormat.java | 2 +-
.../org/apache/pinot/hadoop/io/PinotRecord.java | 4 +-
.../pinot/hadoop/io/PinotRecordSerialization.java | 2 +-
.../pinot/hadoop/job/HadoopSegmentCreationJob.java | 1 +
.../hadoop/job/HadoopSegmentPreprocessingJob.java | 18 ++++-
.../hadoop/job/mappers/SegmentCreationMapper.java | 8 +--
.../pinot/hadoop/io/PinotOutputFormatTest.java | 6 +-
.../pinot/ingestion/common/ControllerRestApi.java | 2 +-
.../ingestion/common/DefaultControllerRestApi.java | 4 +-
.../pinot/ingestion/jobs/BaseSegmentJob.java | 4 +-
.../pinot/ingestion/jobs/SegmentCreationJob.java | 2 +-
.../ingestion/jobs/SegmentPreprocessingJob.java | 4 +-
.../spark/jobs/SparkSegmentCreationFunction.java | 8 +--
.../tests/BaseClusterIntegrationTest.java | 2 +-
.../tests/BaseClusterIntegrationTestSet.java | 2 +-
.../tests/ClusterIntegrationTestUtils.java | 4 +-
.../pinot/integration/tests/ClusterTest.java | 6 +-
.../ControllerPeriodicTasksIntegrationTests.java | 2 +-
.../DefaultCommitterRealtimeIntegrationTest.java | 8 +--
...lakyConsumerRealtimeClusterIntegrationTest.java | 4 +-
...mentBuildPushOfflineClusterIntegrationTest.java | 2 +-
.../tests/HybridClusterIntegrationTest.java | 4 +-
...ridClusterIntegrationTestCommandLineRunner.java | 4 +-
.../tests/MapTypeClusterIntegrationTest.java | 4 +-
...onaryAggregationPlanClusterIntegrationTest.java | 2 +-
.../tests/OfflineClusterIntegrationTest.java | 2 +-
.../tests/PinotURIUploadIntegrationTest.java | 2 +-
.../pinot/integration/tests/QueryGenerator.java | 2 +-
.../tests/StarTreeClusterIntegrationTest.java | 2 +-
.../minion/executor/PurgeTaskExecutorTest.java | 8 +--
.../pinot/orc/data/readers/ORCRecordReader.java | 25 +++----
.../orc/data/readers/ORCRecordReaderTest.java | 35 ++++-----
.../parquet/data/readers/ParquetRecordReader.java | 18 ++---
.../data/readers/ParquetRecordReaderTest.java | 7 +-
.../pinot/perf/BenchmarkDictionaryCreation.java | 4 +-
.../org/apache/pinot/perf/RawIndexBenchmark.java | 8 +--
.../pinot/perf/StringDictionaryPerfTest.java | 8 +--
pinot-server/pom.xml | 4 ++
.../api/resources/DefaultExceptionMapper.java | 2 +-
.../server/request/ScheduledRequestHandler.java | 2 +-
.../starter/helix/HelixInstanceDataManager.java | 3 +-
.../pinot/server/api/TablesResourceTest.java | 2 +-
.../apache/pinot/server/util/SegmentTestUtils.java | 2 +-
{pinot-api => pinot-spi}/pom.xml | 63 ++++++++--------
.../apache/pinot/spi}/data/DateTimeFieldSpec.java | 4 +-
.../pinot/spi}/data/DateTimeFormatPatternSpec.java | 6 +-
.../apache/pinot/spi}/data/DateTimeFormatSpec.java | 6 +-
.../pinot/spi}/data/DateTimeFormatUnitSpec.java | 4 +-
.../pinot/spi}/data/DateTimeGranularitySpec.java | 4 +-
.../apache/pinot/spi}/data/DimensionFieldSpec.java | 2 +-
.../java/org/apache/pinot/spi}/data/FieldSpec.java | 66 ++---------------
.../apache/pinot/spi}/data/MetricFieldSpec.java | 6 +-
.../java/org/apache/pinot/spi}/data/Schema.java | 16 ++---
.../org/apache/pinot/spi}/data/TimeFieldSpec.java | 6 +-
.../pinot/spi}/data/TimeGranularitySpec.java | 8 +--
.../apache/pinot/spi/data/readers}/GenericRow.java | 13 ++--
.../pinot/spi}/data/readers/RecordReader.java | 20 +++---
.../spi}/data/readers/RecordReaderConfig.java | 2 +-
.../org/apache/pinot/spi/utils}/ByteArray.java | 3 +-
.../org/apache/pinot/spi}/utils/BytesUtils.java | 3 +-
.../org/apache/pinot/spi}/utils/EqualityUtils.java | 2 +-
.../org/apache/pinot/spi}/utils/JsonUtils.java | 7 +-
.../src/test/resources/log4j2.xml | 0
.../org/apache/pinot/tools/HybridQuickstart.java | 2 +-
.../pinot/tools/PinotDataAndQueryAnonymizer.java | 14 ++--
.../org/apache/pinot/tools/PinotZKChanger.java | 2 +-
.../org/apache/pinot/tools/SegmentDumpTool.java | 2 +-
.../apache/pinot/tools/StarTreeIndexViewer.java | 2 +-
.../tools/admin/command/AddSchemaCommand.java | 2 +-
.../pinot/tools/admin/command/AddTableCommand.java | 2 +-
.../admin/command/AvroSchemaToPinotSchema.java | 4 +-
.../command/BackfillDateTimeColumnCommand.java | 6 +-
.../tools/admin/command/CreateSegmentCommand.java | 12 ++--
.../tools/admin/command/GenerateDataCommand.java | 14 ++--
.../tools/admin/command/MoveReplicaGroup.java | 2 +-
.../tools/admin/command/PostQueryCommand.java | 2 +-
.../tools/admin/command/QuickstartRunner.java | 2 +-
.../tools/admin/command/RebalanceTableCommand.java | 2 +-
.../tools/admin/command/ValidateConfigCommand.java | 2 +-
.../pinot/tools/backfill/BackfillSegmentUtils.java | 2 +-
.../tools/config/validator/SchemaValidator.java | 2 +-
.../pinot/tools/data/generator/AvroWriter.java | 9 +--
.../pinot/tools/data/generator/DataGenerator.java | 14 ++--
.../tools/data/generator/DataGeneratorSpec.java | 4 +-
.../tools/data/generator/GeneratorFactory.java | 2 +-
.../tools/data/generator/NumberGenerator.java | 2 +-
.../pinot/tools/perf/PerfBenchmarkDriver.java | 2 +-
.../tools/query/comparison/QueryComparison.java | 2 +-
.../query/comparison/SegmentInfoProvider.java | 4 +-
.../query/comparison/StarQueryComparison.java | 2 +-
.../realtime/provisioning/MemoryEstimator.java | 4 +-
.../pinot/tools/scan/query/QueryResponse.java | 2 +-
.../tools/scan/query/ScanBasedQueryProcessor.java | 2 +-
.../converter/DictionaryToRawIndexConverter.java | 2 +-
.../converter/PinotSegmentToAvroConverter.java | 2 +-
.../converter/PinotSegmentToCsvConverter.java | 2 +-
.../converter/PinotSegmentToJsonConverter.java | 4 +-
.../segment/converter/SegmentMergeCommand.java | 2 +-
.../pinot/tools/streams/AirlineDataStream.java | 8 +--
.../pinot/tools/streams/MeetupRsvpStream.java | 4 +-
.../pinot/tools/tuner/IndexTunerCommand.java | 4 +-
.../meta/manager/JsonFileMetaManagerImpl.java | 2 +-
.../collector/SegmentMetadataCollector.java | 2 +-
pom.xml | 6 ++
457 files changed, 1252 insertions(+), 1177 deletions(-)
create mode 100644
pinot-common/src/main/java/org/apache/pinot/common/utils/AvroSchemaUtil.java
delete mode 120000 pinot-hadoop
copy {pinot-api => pinot-spi}/pom.xml (68%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/DateTimeFieldSpec.java (98%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/DateTimeFormatPatternSpec.java
(95%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/DateTimeFormatSpec.java (98%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/DateTimeFormatUnitSpec.java
(97%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/DateTimeGranularitySpec.java
(98%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/DimensionFieldSpec.java (98%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/FieldSpec.java (87%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/MetricFieldSpec.java (96%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/Schema.java (97%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/TimeFieldSpec.java (98%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/TimeGranularitySpec.java
(98%)
rename {pinot-core/src/main/java/org/apache/pinot/core/data =>
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers}/GenericRow.java (90%)
rename {pinot-core/src/main/java/org/apache/pinot/core =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/readers/RecordReader.java
(74%)
rename {pinot-core/src/main/java/org/apache/pinot/core =>
pinot-spi/src/main/java/org/apache/pinot/spi}/data/readers/RecordReaderConfig.java
(95%)
rename {pinot-common/src/main/java/org/apache/pinot/common/utils/primitive =>
pinot-spi/src/main/java/org/apache/pinot/spi/utils}/ByteArray.java (96%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/utils/BytesUtils.java (95%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/utils/EqualityUtils.java (99%)
rename {pinot-common/src/main/java/org/apache/pinot/common =>
pinot-spi/src/main/java/org/apache/pinot/spi}/utils/JsonUtils.java (96%)
copy {pinot-minion => pinot-spi}/src/test/resources/log4j2.xml (100%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
