This is an automated email from the ASF dual-hosted git repository.
ajantha pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/carbondata.git.
from 52b5a2a [maven-release-plugin] prepare for next development iteration
add bfc9533 [HOTFIX] Avoid calling SecondaryIndexUtil.readFileFooter()
for every splits identified during SI creation
add d850c95 [CARBONDATA-3905] NPE due to null length while querying in
Presto
add e4337eb [CARBONDATA-4053] Fix alter table rename column failed when
column name is "a"
add d2df6d1 [CARBONDATA-4029] Fix oldTimeStamp issue in alter table add
segment query.
add 32c2306 [CARBONDATA-4057] Support Complex DataType when Save
DataFrame with MODE.OVERWRITE
add e7aec95 [CARBONDATA-4056] Added global sort for data files merge
operation in SI segments.
add e9b38cd [HOTFIX] Fix a random CI failure in HiveCarbonTest:
'metastore_db' already exists
add 09be330 [CARBONDATA-4066] data mismatch observed with SI and without
SI when SI global sort and SI segment merge is true
add 86080cd [CARBONDATA-4064] Fix tpcds query failure with SI
add d88ea4b [CARBONDATA-3978] Support .Trash folder
add 2add61e [CARBONDATA-4052] Handled insert overwrite scenario for SI
add 7ca3775 [CARBONDATA-4020] Fixed drop index when multiple index exists
add e91a645 [CARBONDATA-4022] Fix invalid path issue for segment added
through alter add segment query
add b379ebe [CARBONDATA-4046] Handled multiple partition columns for
partition cache
add c5f464f [CARBONDATA-4050]Avoid redundant RPC calls to get file status
when CarbonFile is instantiated with fileStatus constructor
add e1ce3fa [CARBONDATA-4054] Support data size control for minor
compaction
add d8799ff [CARBONDATA-4029] [CARBONDATA-3908] Issue while adding
segments through alter add segment command
add 7e58bbb [CARBONDATA-4067]: Removing force option in clean files
command and changing behaviour when MFD, Compacted and stale Inprogress
segments can be deleted
add 74e967e [CARBONDATA-4062] Refactor clean files feature
add afb7626 [HOTFIX] Refact Carbon Util
add 6b03967 [CARBONDATA-4069] [CARBONDATA-4068] handled set streaming and
long string for SI table or table having SI
add 7aafb6b [CARBONDATA-4081] Fix multiple issues with clean files command
add b8c3702 [CARBONDATA-4087] Handled issue with huge data (exceeding 32K
records) after enabling local dictionary in Presto
add 29ecd5f [CARBONDATA-4077] Refactor and Fix Insert into partition
issue with FileMergeSortComparator
add 35c73e6 [CARBONDATA-4080] Wrong results for select count on invalid
segments
add 8fc8225 [CARBONDATA-4078] Add external segment and query with index
server fails
add e347ca9 [CARBONDATA-4072] Clean files command is not deleting
.segment files for the segments added through alter table add segment query.
add eeee018 [CARBONDATA-4071] Fix wrong values of date or timestamp child
columns on reading through SDK
add 88566e0 [CARBONDATA-4076] Fix MV having Subquery alias used in query
projection #4038
add 297b18f [CARBONDATA-4084] Fixed data corruption issue after fallback
of Local dictionary
add ecebee5 [CARBONDATA-4092] Fix concurrent issues in delete segment
API's and MV flow
add aae93c1 [CARBONDATA-4093] Added logs for MV and method to verify if
mv is in Sync during query
add 1dfcdec [CARBONDATA-4094]: Fix fallback count(*) issue on partition
table with index server
add c8cec12 [CARBONDATA-4089] Create table with location, if the location
doesn't have scheme, the default will be local file system, which is not the
file system defined by fs.defaultFS
add 11ae435 [CARBONDATA-4095] Fix Select Query with SI filter fails, when
columnDrift is Set
add 385d9ab [CARBONDATA-4088] Drop metacache didn't clear some cache
information which leads to memory leak
add 316939b [CARBONDATA-4099] Fixed select query on main table with a SI
table in case of concurrent load, compact and clean files operation
add 19f9027 [CARBONDATA-4100] Fix SI segments are in inconsistent state
with maintable after concurrent Load & Compaction operation
add 8831af4 [CARBONDATA-4073] Added FT for missing scenarios and removed
dead code in Presto integration
add 44db434 [CARBONDATA-3987] Handled filter and IUD operation for
pagination reader in SDK
add 4d8a01f [CARBONDATA-4070] [CARBONDATA-4059] Fixed SI issues and
improved FT
add e019806 [CARBONDATA-4065] Support MERGE INTO SQL Command
add 2129466 [DOC] Running the Thrift JDBC/ODBC server with
CarbonExtensions
add aa2121e [CARBONDATA-4055]Fix creation of empty segment directory and
meta entry when there is no update/insert data
add 7585656 [CARBONDATA-4096] SDK read fails from cluster and sdk read
filter query on sort column giving wrong result with IndexServer
add 5971417 [CARBONDATA-4051] Geo spatial index algorithm improvement and
UDFs enhancement
add f5e35cd [CARBONDATA-4097] ColumnVectors should not be initialized as
ColumnVectorWrapperDirect for alter tables
add 54f8697 [CARBONDATA-4104] Vector filling for complex decimal type
needs to be handled
add 46a46a0 [CARBONDATA-4109] Improve carbondata coverage for
presto-integration code
add 5a2edc3 [CARBONDATA-4112] Data mismatch issue in SI global sort merge
flow
add 440ab03 [CARBONDATA-4113] Partition prune and cache fix when
carbon.read.partition.hive.direct is disabled
add aa7efda [CARBONDATA-4082] Fix alter table add segment query on adding
a segment having delete delta files
add 9b04540 [CARBONDATA-4107] Added related MV tables Map to fact table
and added lock while touchMDTFile
add afbf531 [CARBONDATA-4111] Filter query having invalid results after
add segment to table having SI with Indexserver
add ec1c0ca [CARBONDATA-4102] Added UT and FT to improve coverage of SI
module.
add 115182d [CARBONDATA-4122] Use CarbonFile API instead of java File API
for Flink CarbonLocalWriter
add 791857b [CARBONDATA-4125] SI compatability issue fix
add 91f1b69 [CARBONDATA-4124] Fix Refresh MV which does not exist error
message
add 3f1db97 [CARBONDATA-4117][CARBONDATA-4123] cg index and bloom index
query issue with Index server
add 1cab165 [CARBONDATA-3962] Fixed concurrent load failure with flat
folder structure.
add 5ec3536 [CARBONDATA-4126] Concurrent compaction failed with load on
table
add 59ad77a [CARBONDATA-4121] Prepriming is not working in Index Server
add 0112268 [CARBONDATA-4115] Successful load and insert will return
segment ID
add 8f2ee7f [CARBONDATA-4137] Refactor CarbonDataSourceScan without the
soruces.Filter of Spark 3
add 35c4b33 [CARBONDATA-4133] Concurrent Insert Overwrite with static
partition on Index server fails
add 25c5687 [CARBONDATA-4141] Index Server is not caching indexes for
external tables with sdk segments
add d5b3b8c [CARBONDATA-4075] Using withEvents instead of fireEvent
add d9f69ae [CARBONDATA-4110] Support clean files dry run operation and
show statistics after clean files operation
No new revisions were added by this update.
Summary of changes:
.../core/constants/CarbonCommonConstants.java | 57 ++
.../impl/LocalDictDimensionDataChunkStore.java | 6 +
.../filesystem/AbstractDFSCarbonFile.java | 11 +-
.../page/DecoderBasedFallbackEncoder.java | 20 +-
.../carbondata/core/index/IndexInputFormat.java | 12 +-
.../apache/carbondata/core/index/IndexUtil.java | 37 +-
.../org/apache/carbondata/core/index/Segment.java | 14 +
.../apache/carbondata/core/index/TableIndex.java | 2 +-
.../core/indexstore/BlockletIndexStore.java | 52 +-
.../core/indexstore/ExtendedBlocklet.java | 18 +-
.../core/indexstore/SegmentWrapperContainer.java | 3 +
.../blockletindex/BlockletIndexFactory.java | 8 +-
.../generator/ColumnLocalDictionaryGenerator.java | 11 +
.../generator/LocalDictionaryGenerator.java | 6 +
.../carbondata/core/metadata/CarbonMetadata.java | 13 +-
.../carbondata/core/metadata/SegmentFileStore.java | 41 +-
.../metadata/datatype/DecimalConverterFactory.java | 125 +--
.../core/metadata/schema/table/CarbonTable.java | 14 +
.../carbondata/core/mutate/CarbonUpdateUtil.java | 236 +----
.../scan/executor/impl/AbstractQueryExecutor.java | 4 +-
.../core/scan/executor/util/RestructureUtil.java | 7 +-
.../result/vector/impl/CarbonColumnVectorImpl.java | 8 +
.../core/statusmanager/LoadMetadataDetails.java | 9 +-
.../core/statusmanager/SegmentStatusManager.java | 309 +++---
.../carbondata/core/util/CarbonProperties.java | 94 ++
.../carbondata/core/util/CarbonTestUtil.java | 12 +
.../carbondata/core/util/CleanFilesUtil.java | 221 ++++
.../apache/carbondata/core/util/CustomIndex.java | 8 +
.../carbondata/core/util/DeleteLoadFolders.java | 96 +-
.../apache/carbondata/core/util/SessionParams.java | 1 +
.../org/apache/carbondata/core/util/TrashUtil.java | 271 +++++
.../carbondata/core/util/path/CarbonTablePath.java | 15 +-
.../org/apache/carbondata/core/view/MVManager.java | 87 +-
.../apache/carbondata/core/view/MVProvider.java | 80 +-
.../scan/executor/util/RestructureUtilTest.java | 8 +-
dev/findbugs-exclude.xml | 22 +-
docs/clean-files.md | 104 ++
docs/configuration-parameters.md | 3 +
docs/ddl-of-carbondata.md | 4 +-
docs/dml-of-carbondata.md | 6 +
docs/flink-integration-guide.md | 3 +-
docs/images/spatial-index-polygonlist.png | Bin 0 -> 43631 bytes
docs/images/spatial-index-polylinelist.png | Bin 0 -> 20858 bytes
docs/images/spatial-index-rangelist.png | Bin 0 -> 64776 bytes
docs/mv-guide.md | 6 +
docs/quick-start-guide.md | 19 +-
docs/scd-and-cdc-guide.md | 38 +-
docs/spatial-index-guide.md | 81 +-
.../benchmark/ConcurrentQueryBenchmark.scala | 2 +-
.../carbondata/examples/DataMergeIntoExample.scala | 164 +++
.../apache/carbondata/examples/S3CsvExample.scala | 2 +-
.../org/apache/carbondata/geo/GeoConstants.java | 30 +
.../org/apache/carbondata/geo/GeoHashIndex.java | 248 +----
.../org/apache/carbondata/geo/GeoHashUtils.java | 409 ++++++++
.../{GeoConstants.java => GeoOperationType.java} | 28 +-
.../org/apache/carbondata/geo/QuadTreeCls.java | 80 +-
.../geo/scan/expression/PolygonExpression.java | 45 +-
.../geo/scan/expression/PolygonListExpression.java | 81 ++
.../expression/PolygonRangeListExpression.java | 124 +++
.../scan/expression/PolylineListExpression.java | 105 ++
.../filter/executor/PolygonFilterExecutorImpl.java | 9 +-
.../apache/carbondata/geo/GeoHashUtilsTest.java | 266 +++++
.../org/apache/carbondata/geo/QuadTreeClsTest.java | 47 +-
.../hadoop/api/CarbonFileInputFormat.java | 28 +-
.../carbondata/hadoop/api/CarbonInputFormat.java | 20 +-
.../hadoop/api/CarbonOutputCommitter.java | 6 +-
.../hadoop/api/CarbonTableInputFormat.java | 35 +-
.../hadoop/api/CarbonTableOutputFormat.java | 7 +-
index/secondary-index/pom.xml | 5 +
.../CarbonDataFileMergeTestCaseOnSI.scala | 213 ++--
.../testsuite/secondaryindex/DropTableTest.scala | 37 +-
.../TestCarbonInternalMetastore.scala | 153 +++
.../TestCreateIndexForCleanAndDeleteSegment.scala | 34 +-
.../secondaryindex/TestCreateIndexTable.scala | 174 +++-
.../TestCreateIndexWithLoadAndCompaction.scala | 198 +++-
.../TestIndexModelWithAggQueries.scala | 24 +-
.../secondaryindex/TestIndexModelWithIUD.scala | 27 +-
.../testsuite/secondaryindex/TestIndexRepair.scala | 41 +-
.../secondaryindex/TestQueryWithSkipSI.scala | 42 +
.../TestRegisterIndexCarbonTable.scala | 21 +-
.../secondaryindex/TestSIWithAddSegment.scala | 17 +-
.../TestSIWithComplexArrayType.scala | 119 +++
.../secondaryindex/TestSIWithInsertOverwrite.scala | 84 ++
.../secondaryindex/TestSIWithRangeColumn.scala | 88 ++
.../secondaryindex/TestSIWithSecondaryIndex.scala | 235 ++++-
.../secondaryindex/TestSecondaryIndexUtils.scala | 99 ++
.../apache/spark/util/TestCarbonSegmentUtil.scala | 4 +-
.../org/apache/carbon/flink/CarbonLocalWriter.java | 29 +-
.../java/org/apache/carbon/flink/CarbonWriter.java | 30 +-
.../hive/MapredCarbonOutputCommitter.java | 12 +-
.../org/apache/carbondata/hive/HiveTestUtils.java | 5 +
.../presto/CarbonColumnVectorWrapper.java | 365 -------
.../carbondata/presto/CarbonVectorBatch.java | 2 +-
.../presto/PrestoCarbonVectorizedRecordReader.java | 7 +-
.../presto/impl/CarbonLocalInputSplit.java | 19 -
.../presto/impl/CarbonLocalMultiBlockSplit.java | 7 -
.../presto/CarbondataColumnConstraint.java | 93 --
.../presto/readers/SliceStreamReader.java | 2 +
.../presto/CarbondataColumnConstraint.java | 93 --
.../apache/carbondata/presto/PrestoFilterUtil.java | 98 --
.../carbondata/presto/impl/CarbonTableReader.java | 23 -
.../presto/readers/ObjectStreamReader.java | 77 --
.../presto/readers/SliceStreamReader.java | 2 +
.../carbondata/presto/server/PrestoTestUtil.scala | 93 ++
.../carbondata/presto/server/PrestoTestUtil.scala | 93 ++
.../PrestoTestNonTransactionalTableFiles.scala | 36 +
.../PrestoTestUsingSparkStore.scala | 423 ++++++++
integration/spark/pom.xml | 21 +
.../org/apache/spark/sql/parser/CarbonSqlBase.g4 | 646 ++++++++++++
.../apache/spark/sql/CarbonAntlrSqlVisitor.java | 323 ++++++
.../spark/sql/CarbonMergeIntoSQLCommand.scala | 119 +++
.../sql/merge/model/CarbonJoinExpression.java | 60 +-
.../sql/merge/model/CarbonMergeIntoModel.java | 80 ++
.../apache/spark/sql/merge/model/ColumnModel.java | 35 +-
.../apache/spark/sql/merge/model/TableModel.java} | 43 +-
.../load/CarbonInternalLoaderUtil.java | 17 +-
.../query/SecondaryIndexQueryResultProcessor.java | 2 -
.../org/apache/carbondata/api/CarbonStore.scala | 132 +--
.../carbondata/events/CleanFilesEvents.scala | 16 +-
.../org/apache/carbondata/events/package.scala | 18 +-
.../org/apache/carbondata/geo/GeoUdfRegister.scala | 49 +
.../org/apache/carbondata/geo/GeoUtilUDFs.scala | 66 ++
.../org/apache/carbondata/geo/InPolygonUDF.scala | 31 +-
.../org/apache/carbondata/index/TextMatchUDF.scala | 12 -
.../apache/carbondata/indexserver/IndexJobs.scala | 5 +
.../carbondata/indexserver/IndexServer.scala | 7 +-
.../scala/org/apache/carbondata/spark/KeyVal.scala | 4 +-
.../spark/rdd/CarbonDataRDDFactory.scala | 115 +--
.../spark/rdd/CarbonTableCompactor.scala | 76 +-
.../carbondata/spark/rdd/CompactionFactory.scala | 4 -
.../apache/carbondata/spark/rdd/Compactor.scala | 27 +-
.../carbondata/spark/rdd/StreamHandoffRDD.scala | 2 +-
.../carbondata/spark/util/CarbonSparkUtil.scala | 10 +
.../apache/carbondata/spark/util/CommonUtil.scala | 71 +-
.../apache/carbondata/trash/DataTrashManager.scala | 315 ++++++
.../apache/carbondata/view/MVCatalogInSpark.scala | 4 +
.../org/apache/carbondata/view/MVHelper.scala | 97 +-
.../apache/carbondata/view/MVManagerInSpark.scala | 16 +-
.../org/apache/carbondata/view/MVRefresher.scala | 44 +-
.../apache/spark/sql/CarbonBoundReference.scala | 17 -
.../apache/spark/sql/CarbonCatalystOperators.scala | 4 +-
.../org/apache/spark/sql/CarbonCountStar.scala | 2 +-
.../apache/spark/sql/CarbonDataFrameWriter.scala | 3 +
.../spark/sql/CarbonDatasourceHadoopRelation.scala | 86 +-
.../scala/org/apache/spark/sql/CarbonEnv.scala | 21 +-
.../org/apache/spark/sql/CarbonExtensions.scala | 14 +-
.../{CarbonUtils.scala => CarbonThreadUtil.scala} | 10 +-
.../spark/sql/CustomDeterministicExpression.scala | 2 +-
.../sql/execution/CastExpressionOptimization.scala | 208 ++--
.../command/carbonTableSchemaCommon.scala | 3 +-
.../execution/command/index/DropIndexCommand.scala | 31 +-
.../command/index/IndexRepairCommand.scala | 42 +-
.../command/management/CarbonAddLoadCommand.scala | 149 ++-
.../CarbonAlterTableCompactionCommand.scala | 56 +-
.../management/CarbonCleanFilesCommand.scala | 173 ++--
.../management/CarbonDeleteLoadByIdCommand.scala | 28 +-
.../CarbonDeleteLoadByLoadDateCommand.scala | 27 +-
.../management/CarbonInsertIntoCommand.scala | 23 +-
.../management/CarbonInsertIntoWithDf.scala | 6 -
.../command/management/CarbonLoadDataCommand.scala | 31 +-
.../command/management/CommonLoadUtils.scala | 27 +-
.../management/RefreshCarbonTableCommand.scala | 21 +-
.../command/mutation/CarbonTruncateCommand.scala | 36 +-
.../command/mutation/DeleteExecution.scala | 4 +-
.../mutation/merge/CarbonMergeDataSetCommand.scala | 7 +-
.../command/mutation/merge/MergeProjection.scala | 11 +-
.../command/mutation/merge/interfaces.scala | 7 +-
.../CarbonAlterTableAddHivePartitionCommand.scala | 19 +-
.../CarbonAlterTableDropHivePartitionCommand.scala | 50 +-
.../schema/CarbonAlterTableDropColumnCommand.scala | 4 +
.../table/CarbonCreateTableLikeCommand.scala | 4 +-
.../table/CarbonDescribeFormattedCommand.scala | 4 +
.../command/table/CarbonDropTableCommand.scala | 63 +-
.../command/view/CarbonCreateMVCommand.scala | 57 +-
.../command/view/CarbonDropMVCommand.scala | 24 +-
.../command/view/CarbonRefreshMVCommand.scala | 32 +-
.../execution/strategy/CarbonDataSourceScan.scala | 180 ++++
.../strategy/CarbonLateDecodeStrategy.scala | 1070 --------------------
.../sql/execution/strategy/CarbonPlanHelper.scala | 101 +-
.../execution/strategy/CarbonSourceStrategy.scala | 432 ++++++++
.../spark/sql/execution/strategy/DDLStrategy.scala | 20 +-
.../spark/sql/execution/strategy/DMLStrategy.scala | 255 +++++
.../execution/strategy/MixedFormatHandler.scala | 30 +-
.../sql/execution/strategy/PushDownHelper.scala | 2 +-
.../strategy/StreamingTableStrategy.scala | 13 +-
.../spark/sql/hive/CarbonAnalysisRules.scala | 14 +-
.../spark/sql/hive/CarbonFileMetastore.scala | 16 +-
.../apache/spark/sql/index/CarbonIndexUtil.scala | 342 ++++---
.../spark/sql/listeners/PrePrimingListener.scala | 4 +-
.../apache/spark/sql/optimizer/CarbonFilters.scala | 786 +++++++-------
.../org/apache/spark/sql/optimizer/MVMatcher.scala | 114 ++-
.../org/apache/spark/sql/optimizer/MVRewrite.scala | 167 +--
.../apache/spark/sql/optimizer/MVRewriteRule.scala | 47 +-
.../spark/sql/parser/CarbonAntlrParser.scala | 41 +
.../sql/parser/CarbonExtensionSqlParser.scala | 45 +-
.../spark/sql/parser/CarbonSpark2SqlParser.scala | 10 +-
.../spark/sql/parser/CarbonSparkSqlParser.scala | 4 +-
.../command/RegisterIndexTableCommand.scala | 14 +-
.../secondaryindex/command/SICreationCommand.scala | 143 ++-
.../command/SIRebuildSegmentRunner.scala | 86 +-
.../AlterTableCompactionPostEventListener.scala | 52 +-
.../AlterTableMergeIndexSIEventListener.scala | 3 -
.../events/CleanFilesPostEventListener.scala | 73 +-
.../events/SIDropEventListener.scala | 2 -
.../events/SILoadEventListener.scala | 8 +-
.../SILoadEventListenerForFailedSegments.scala | 12 +-
.../hive/CarbonInternalMetastore.scala | 15 +-
.../joins/BroadCastSIFilterPushJoin.scala | 22 +-
.../optimizer/CarbonSITransformationRule.scala | 19 +-
.../optimizer/CarbonSecondaryIndexOptimizer.scala | 10 +-
.../secondaryindex/rdd/CarbonSIRebuildRDD.scala | 47 +-
.../rdd/CarbonSecondaryIndexRDD.scala | 48 +-
.../secondaryindex/rdd/SecondaryIndexCreator.scala | 89 +-
.../secondaryindex/util/SecondaryIndexUtil.scala | 190 +++-
.../org/apache/spark/sql/test/util/QueryTest.scala | 10 +
.../org/apache/spark/util/AlterTableUtil.scala | 52 +-
.../scala/org/apache/spark/util/CleanFiles.scala | 45 +-
.../apache/spark/util/PartitionCacheManager.scala | 32 +-
.../scala/org/apache/spark/util/SparkUtil.scala | 22 +-
.../apache/spark/sql/CarbonToSparkAdapter.scala | 28 +-
.../execution/strategy/CarbonDataSourceScan.scala | 58 --
.../spark/sql/hive/CarbonSessionStateBuilder.scala | 8 +-
.../apache/spark/sql/CarbonToSparkAdapter.scala | 30 +-
.../execution/strategy/CarbonDataSourceScan.scala | 60 --
.../spark/sql/hive/CarbonSessionStateBuilder.scala | 10 +-
integration/spark/src/test/resources/geodata2.csv | 31 +
.../resources/geodataWithCorrectSpatialIndex.csv | 17 +
.../resources/geodataWithErrorSpatialIndex.csv | 17 +
.../spark/src/test/resources/secindex/array.csv | 8 +-
.../scala/org/apache/carbondata/geo/GeoTest.scala | 514 +++++++++-
.../bloom/BloomCoarseGrainIndexFunctionSuite.scala | 28 +-
.../testsuite/binary/TestBinaryDataType.scala | 6 +-
.../complexType/TestAllComplexDataType.scala | 58 +-
.../complexType/TestArrayContainsPushDown.scala | 36 +-
.../dataload/SparkStoreCreatorForPresto.scala | 489 +++++++++
.../testsuite/dataload/TestLoadDataGeneral.scala | 5 +-
.../dataload/TestLoadDataWithAutoLoadMerge.scala | 5 +-
.../dataload/TestLoadDataWithCompression.scala | 49 +-
.../testsuite/addsegment/AddSegmentTestCase.scala | 201 +++-
.../allqueries/InsertIntoCarbonTableTestCase.scala | 13 +-
...ryWithColumnMetCacheAndCacheLevelProperty.scala | 18 +-
.../cleanfiles/TestCleanFileCommand.scala | 597 +++++++++++
.../TestCleanFilesCommandPartitionTable.scala | 360 +++++++
.../testsuite/cloud/AllDataSourceTestCase.scala | 2 +-
.../createTable/TestCreateTableAsSelect.scala | 10 +
...lumnComment.scala => TestCreateTablePath.scala} | 43 +-
.../createTable/TestRenameTableWithIndex.scala | 7 +-
.../CompactionSupportGlobalSortFunctionTest.scala | 16 +-
.../MajorCompactionIgnoreInMinorTest.scala | 141 +++
.../TableLevelCompactionOptionTest.scala | 12 +-
.../dataload/TestGlobalSortDataLoad.scala | 5 +-
.../filterexpr/AllDataTypesTestCaseFilter.scala | 8 +-
.../filterexpr/TestImplicitFilterExpression.scala | 5 +-
.../FlatFolderTableLoadingTestCase.scala | 25 +-
.../spark/testsuite/index/CGIndexTestCase.scala | 15 +-
.../testsuite/iud/DeleteCarbonTableTestCase.scala | 8 +-
.../iud/MergeIntoCarbonTableTestCase.scala | 314 ++++++
.../testsuite/iud/UpdateCarbonTableTestCase.scala | 8 +-
.../spark/testsuite/merge/MergeTestCase.scala | 2 +-
.../testsuite/segment/ShowSegmentTestCase.scala | 22 +-
.../segmentreading/TestSegmentReading.scala | 104 ++
.../TestSegmentReadingForMultiThreading.scala | 20 +-
.../StandardPartitionGlobalSortTestCase.scala | 12 +-
.../StandardPartitionTableCleanTestCase.scala | 7 +-
.../StandardPartitionTableCompactionTestCase.scala | 7 +-
.../StandardPartitionTableLoadingTestCase.scala | 47 +-
.../StandardPartitionTableQueryTestCase.scala | 5 +-
.../carbondata/view/rewrite/MVCreateTestCase.scala | 34 +
.../view/rewrite/TestAllOperationsOnMV.scala | 6 +
.../view/rewrite/TestPartitionWithMV.scala | 7 +-
.../carbondata/query/TestFilterReordering.scala | 75 --
.../vectorreader/AddColumnTestCases.scala | 23 +
.../AlterTableColumnRenameTestCase.scala | 13 +-
.../vectorreader/VectorReaderTestCase.scala | 2 +-
.../apache/spark/sql/CarbonExtensionSuite.scala | 2 +-
.../org/apache/spark/util/CarbonCommandSuite.scala | 15 +-
pom.xml | 7 +-
.../loading/TableProcessingOperations.java | 114 ---
.../impl/SpatialIndexFieldConverterImpl.java | 4 +-
.../loading/parser/impl/RowParserImpl.java | 4 +-
.../unsafe/holder/UnsafeFinalMergePageHolder.java | 6 +-
.../sort/unsafe/holder/UnsafeInmemoryHolder.java | 6 +-
.../holder/UnsafeSortTempFileChunkHolder.java | 6 +-
.../InputProcessorStepWithNoConverterImpl.java | 8 -
.../processing/merger/CarbonDataMergerUtil.java | 56 +-
.../merger/CompactionResultSortProcessor.java | 11 +-
.../sort/sortdata/FileMergeSortComparator.java | 100 +-
.../processing/sort/sortdata/SortParameters.java | 73 +-
.../sort/sortdata/SortTempFileChunkHolder.java | 12 +-
.../processing/sort/sortdata/TableFieldStat.java | 27 +-
.../processing/util/CarbonDataProcessorUtil.java | 147 ++-
.../processing/util/CarbonLoaderUtil.java | 112 +-
.../sort/sortdata/FileMergeSortComparatorTest.java | 27 +-
.../org/apache/carbondata/sdk/file/CarbonIUD.java | 16 +
.../apache/carbondata/sdk/file/CarbonReader.java | 85 +-
.../carbondata/sdk/file/CarbonReaderBuilder.java | 65 +-
.../carbondata/sdk/file/CarbonWriterBuilder.java | 4 +-
.../sdk/file/PaginationCarbonReader.java | 27 +-
.../carbondata/sdk/file/CarbonReaderTest.java | 267 +++++
.../sdk/file/PaginationCarbonReaderTest.java | 138 +++
300 files changed, 14549 insertions(+), 5963 deletions(-)
create mode 100644
core/src/main/java/org/apache/carbondata/core/util/CleanFilesUtil.java
create mode 100644
core/src/main/java/org/apache/carbondata/core/util/TrashUtil.java
create mode 100644 docs/clean-files.md
create mode 100644 docs/images/spatial-index-polygonlist.png
create mode 100644 docs/images/spatial-index-polylinelist.png
create mode 100644 docs/images/spatial-index-rangelist.png
create mode 100644
examples/spark/src/main/scala/org/apache/carbondata/examples/DataMergeIntoExample.scala
create mode 100644
geo/src/main/java/org/apache/carbondata/geo/GeoHashUtils.java
copy geo/src/main/java/org/apache/carbondata/geo/{GeoConstants.java =>
GeoOperationType.java} (65%)
create mode 100644
geo/src/main/java/org/apache/carbondata/geo/scan/expression/PolygonListExpression.java
create mode 100644
geo/src/main/java/org/apache/carbondata/geo/scan/expression/PolygonRangeListExpression.java
create mode 100644
geo/src/main/java/org/apache/carbondata/geo/scan/expression/PolylineListExpression.java
create mode 100644
geo/src/test/java/org/apache/carbondata/geo/GeoHashUtilsTest.java
create mode 100644
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestCarbonInternalMetastore.scala
create mode 100644
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestQueryWithSkipSI.scala
create mode 100644
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestSIWithInsertOverwrite.scala
create mode 100644
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestSIWithRangeColumn.scala
delete mode 100644
integration/presto/src/main/java/org/apache/carbondata/presto/CarbonColumnVectorWrapper.java
delete mode 100755
integration/presto/src/main/prestodb/org/apache/carbondata/presto/CarbondataColumnConstraint.java
delete mode 100755
integration/presto/src/main/prestosql/org/apache/carbondata/presto/CarbondataColumnConstraint.java
delete mode 100644
integration/presto/src/main/prestosql/org/apache/carbondata/presto/readers/ObjectStreamReader.java
create mode 100644
integration/presto/src/test/scala/org/apache/carbondata/presto/integrationtest/PrestoTestUsingSparkStore.scala
create mode 100644
integration/spark/src/main/antlr4/org/apache/spark/sql/parser/CarbonSqlBase.g4
create mode 100644
integration/spark/src/main/java/org/apache/spark/sql/CarbonAntlrSqlVisitor.java
create mode 100644
integration/spark/src/main/java/org/apache/spark/sql/CarbonMergeIntoSQLCommand.scala
copy
core/src/main/java/org/apache/carbondata/core/datastore/columnar/DummyBlockIndexerStorage.java
=>
integration/spark/src/main/java/org/apache/spark/sql/merge/model/CarbonJoinExpression.java
(50%)
create mode 100644
integration/spark/src/main/java/org/apache/spark/sql/merge/model/CarbonMergeIntoModel.java
copy
core/src/main/java/org/apache/carbondata/core/metadata/datatype/TimestampType.java
=>
integration/spark/src/main/java/org/apache/spark/sql/merge/model/ColumnModel.java
(63%)
copy
integration/{hive/src/main/java/org/apache/carbondata/hive/CarbonHiveRow.java
=> spark/src/main/java/org/apache/spark/sql/merge/model/TableModel.java} (57%)
copy
core/src/main/java/org/apache/carbondata/core/exception/CarbonFileException.java
=> integration/spark/src/main/scala/org/apache/carbondata/events/package.scala
(65%)
create mode 100644
integration/spark/src/main/scala/org/apache/carbondata/geo/GeoUdfRegister.scala
create mode 100644
integration/spark/src/main/scala/org/apache/carbondata/geo/GeoUtilUDFs.scala
create mode 100644
integration/spark/src/main/scala/org/apache/carbondata/trash/DataTrashManager.scala
rename
integration/spark/src/main/scala/org/apache/spark/sql/{CarbonUtils.scala =>
CarbonThreadUtil.scala} (89%)
create mode 100644
integration/spark/src/main/scala/org/apache/spark/sql/execution/strategy/CarbonDataSourceScan.scala
delete mode 100644
integration/spark/src/main/scala/org/apache/spark/sql/execution/strategy/CarbonLateDecodeStrategy.scala
create mode 100644
integration/spark/src/main/scala/org/apache/spark/sql/execution/strategy/CarbonSourceStrategy.scala
create mode 100644
integration/spark/src/main/scala/org/apache/spark/sql/execution/strategy/DMLStrategy.scala
create mode 100644
integration/spark/src/main/scala/org/apache/spark/sql/parser/CarbonAntlrParser.scala
delete mode 100644
integration/spark/src/main/spark2.3/org/apache/spark/sql/execution/strategy/CarbonDataSourceScan.scala
delete mode 100644
integration/spark/src/main/spark2.4/org/apache/spark/sql/execution/strategy/CarbonDataSourceScan.scala
create mode 100644 integration/spark/src/test/resources/geodata2.csv
create mode 100644
integration/spark/src/test/resources/geodataWithCorrectSpatialIndex.csv
create mode 100644
integration/spark/src/test/resources/geodataWithErrorSpatialIndex.csv
create mode 100644
integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataload/SparkStoreCreatorForPresto.scala
create mode 100644
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/cleanfiles/TestCleanFileCommand.scala
create mode 100644
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/cleanfiles/TestCleanFilesCommandPartitionTable.scala
copy
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/{TestCreateTableWithColumnComment.scala
=> TestCreateTablePath.scala} (50%)
create mode 100644
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/MergeIntoCarbonTableTestCase.scala
delete mode 100644
integration/spark/src/test/scala/org/apache/spark/carbondata/query/TestFilterReordering.scala
