carbondata git commit: [CARBONDATA-2727][BloomDataMap] Support create bloom datamap on newly added column

Repository: carbondata Updated Branches: refs/heads/master aec47e06f -> 81038f55e [CARBONDATA-2727][BloomDataMap] Support create bloom datamap on newly added column Add a result collector with rowId infomation for datamap rebuild if table schema is changed; Use keygenerator to retrieve

carbondata git commit: [CARBONDATA-2724][DataMap]Unsupported create datamap on table with V1 or V2 format data

Repository: carbondata Updated Branches: refs/heads/master 81038f55e -> a16289786 [CARBONDATA-2724][DataMap]Unsupported create datamap on table with V1 or V2 format data block creating datamap on carbon table with V1 or V2 format Currently the version info is read from carbon data file This

[2/2] carbondata git commit: Revert "[CARBONDATA-2532][Integration] Carbon to support spark 2.3 version, ColumnVector Interface"

Revert "[CARBONDATA-2532][Integration] Carbon to support spark 2.3 version, ColumnVector Interface" This reverts commit 2b8ae2628d50efcd095696b5bf614eab2fcdb8d2. Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit:

[1/2] carbondata git commit: Revert "[CARBONDATA-2532][Integration] Carbon to support spark 2.3 version, ColumnVector Interface"

Repository: carbondata Updated Branches: refs/heads/carbonstore 0aab4e7c6 -> 96fe233a2 http://git-wip-us.apache.org/repos/asf/carbondata/blob/96fe233a/streaming/src/main/java/org/apache/carbondata/streaming/CarbonStreamRecordReader.java

[43/50] [abbrv] carbondata git commit: [CARBONDATA-2613] Support csv based carbon table

[CARBONDATA-2613] Support csv based carbon table 1. create csv based carbon table using CREATE TABLE fact_table (col1 bigint, col2 string, ..., col100 string) STORED BY 'CarbonData' TBLPROPERTIES( 'foramt'='csv', 'csv.delimiter'=',', 'csv.header'='col1,col2,col100') 2. Load data to this

[03/50] [abbrv] carbondata git commit: [CARBONDATA-2720] Remove dead code

http://git-wip-us.apache.org/repos/asf/carbondata/blob/f9114036/processing/src/main/java/org/apache/carbondata/processing/loading/steps/DataWriterProcessorStepImpl.java -- diff --git

[48/50] [abbrv] carbondata git commit: [CARBONDATA-2705][CarbonStore] CarbonStore Java API and Implementation

[CARBONDATA-2705][CarbonStore] CarbonStore Java API and Implementation Support two implementations: 1.LocalCarbonStore for usage in local mode 2.DistributedCarbonStore leveraging multiple server (Master and Workers) via RPC This closes #2473 Project:

[27/50] [abbrv] carbondata git commit: [CARBONDATA-2482] Pass uuid while writing segment file if possible

[CARBONDATA-2482] Pass uuid while writing segment file if possible Pass the uuid (segmentFileName) to the writeSegmentFile method file if possible. Problem: When the supporting tables depends on the segmentFileName of the main table, query is failing, as it is expected to be the same name as

[46/50] [abbrv] carbondata git commit: [CARBONDATA-2705][CarbonStore] CarbonStore Java API and Implementation

http://git-wip-us.apache.org/repos/asf/carbondata/blob/85cdc404/store/core/src/main/java/org/apache/carbondata/store/impl/rpc/model/QueryResponse.java -- diff --git

[11/50] [abbrv] carbondata git commit: [CARBONDATA-2655][BloomDataMap] BloomFilter datamap support in operator

[CARBONDATA-2655][BloomDataMap] BloomFilter datamap support in operator Now queries with in expression on bloom index column can leverage the BloomFilter datamap. This closes #2445 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit:

[05/50] [abbrv] carbondata git commit: [CARBONDATA-2720] Remove dead code

http://git-wip-us.apache.org/repos/asf/carbondata/blob/f9114036/datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneCoarseGrainDataMap.java -- diff --git

[02/50] [abbrv] carbondata git commit: [CARBONDATA-2712] Added fix for Local Dictionary Exclude for multi level complex columns

[CARBONDATA-2712] Added fix for Local Dictionary Exclude for multi level complex columns What was the problem? When Local Dictionary Exclude was defined for multi level complex columns, the columns were still considered for Local Dictionary Include What has been changed? The index value was

[18/50] [abbrv] carbondata git commit: [CARBONDATA-2723][DataMap] Fix bugs in recreate datamap on table

[CARBONDATA-2723][DataMap] Fix bugs in recreate datamap on table While we drop datamap/table, the executor side cache for datamap is stale. So if we recreate the datamap with different index columns, when we are doing data loading, the cache should be cleaned, otherwise the DataMapWriterListener

[49/50] [abbrv] carbondata git commit: [CARBONDATA-2688][CarbonStore] Support SQL in REST API

[CARBONDATA-2688][CarbonStore] Support SQL in REST API Support SQL interface in Horizon service. Support REST client for SQL This closes #2481 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/d4a28a25 Tree:

[08/50] [abbrv] carbondata git commit: [CARBONDATA-2720] Remove dead code

http://git-wip-us.apache.org/repos/asf/carbondata/blob/f9114036/core/src/main/java/org/apache/carbondata/core/scan/executor/util/QueryUtil.java -- diff --git

[21/50] [abbrv] carbondata git commit: [HOTFIX] Removed BatchedDataSourceScanExec class and extended directly from FileSourceScanExec

[HOTFIX] Removed BatchedDataSourceScanExec class and extended directly from FileSourceScanExec Problem: Since some of the code of BatchedDataSourceScanExec is copied from spark, it is difficult to maintain from version upgrades of spark. Currently we face issues during spark 2.3 upgrade so

[09/50] [abbrv] carbondata git commit: [CARBONDATA-2720] Remove dead code

http://git-wip-us.apache.org/repos/asf/carbondata/blob/f9114036/core/src/main/java/org/apache/carbondata/core/datastore/block/SegmentTaskIndexWrapper.java -- diff --git

[24/50] [abbrv] carbondata git commit: [CARBONDATA-2606][Complex DataType Enhancements]Fix Null result if First two Projection column have same parent and third column has different Parent Struct

[CARBONDATA-2606][Complex DataType Enhancements]Fix Null result if First two Projection column have same parent and third column has different Parent Struct Problem: When multiple columns are there,then the first child elements is only going to make parent Object Array. For all other cases it

[07/50] [abbrv] carbondata git commit: [CARBONDATA-2720] Remove dead code

http://git-wip-us.apache.org/repos/asf/carbondata/blob/f9114036/core/src/test/java/org/apache/carbondata/core/datastore/SegmentTaskIndexStoreTest.java -- diff --git

[47/50] [abbrv] carbondata git commit: [CARBONDATA-2705][CarbonStore] CarbonStore Java API and Implementation

http://git-wip-us.apache.org/repos/asf/carbondata/blob/85cdc404/store/core/src/main/java/org/apache/carbondata/store/conf/StoreConf.java -- diff --git a/store/core/src/main/java/org/apache/carbondata/store/conf/StoreConf.java

[37/50] [abbrv] carbondata git commit: [CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework

[CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework This closes #2372 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/d9b40bf9 Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/d9b40bf9

[30/50] [abbrv] carbondata git commit: [CARBONDATA-2649] Fixed arrayIndexOutOfBoundException while loading Blocklet DataMap after alter add column operation

[CARBONDATA-2649] Fixed arrayIndexOutOfBoundException while loading Blocklet DataMap after alter add column operation Things done as part of this PR Fixed arrayIndexOutOfBoundException while loading Blocklet DataMap after alter add column operation Problem: Array Index out of bound exception

[04/50] [abbrv] carbondata git commit: [CARBONDATA-2720] Remove dead code

http://git-wip-us.apache.org/repos/asf/carbondata/blob/f9114036/integration/spark-common/src/main/scala/org/apache/carbondata/events/AlterTableEvents.scala -- diff --git

[01/50] [abbrv] carbondata git commit: [CARBONDATA-2712] Added fix for Local Dictionary Exclude for multi level complex columns [Forced Update!]

Repository: carbondata Updated Branches: refs/heads/carbonstore 4b96ed8ca -> 239a6cadb (forced update)

[45/50] [abbrv] carbondata git commit: [CARBONDATA-2705][CarbonStore] CarbonStore Java API and Implementation

http://git-wip-us.apache.org/repos/asf/carbondata/blob/85cdc404/store/core/src/main/java/org/apache/carbondata/store/rpc/model/BaseResponse.java -- diff --git

carbondata git commit: [CARBONDATA-2746][BloomDataMap] Fix bug for getting datamap file when table has multiple datamaps

Repository: carbondata Updated Branches: refs/heads/master a16289786 -> 4612e0031 [CARBONDATA-2746][BloomDataMap] Fix bug for getting datamap file when table has multiple datamaps Currently, if table has multiple bloom datamap and carbon is set to use distributed datamap, query will throw

[12/50] [abbrv] carbondata git commit: [CARBONDATA-2637][DataMap] Fix bugs in rebuild datamap

[CARBONDATA-2637][DataMap] Fix bugs in rebuild datamap In cluster mode, readCommitScope is null while rebuilding datamap for segments, this will cause NPE. Here we use the origin segment object whose readCommitScope is not null and will work fine. This closes #2493 Project:

[33/50] [abbrv] carbondata git commit: [CARBONDATA-2727][BloomDataMap] Support create bloom datamap on newly added column

[CARBONDATA-2727][BloomDataMap] Support create bloom datamap on newly added column Add a result collector with rowId infomation for datamap rebuild if table schema is changed; Use keygenerator to retrieve surrogate value of dictIndexColumn from query result; This closes #2490 Project:

[06/50] [abbrv] carbondata git commit: [CARBONDATA-2720] Remove dead code

http://git-wip-us.apache.org/repos/asf/carbondata/blob/f9114036/core/src/test/java/org/apache/carbondata/core/scan/expression/conditional/GreaterThanExpressionUnitTest.java -- diff --git

[23/50] [abbrv] carbondata git commit: [CARBONDATA-2528][MV] Fixed order by in mv and aggregation functions inside projection expressions are fixed

[CARBONDATA-2528][MV] Fixed order by in mv and aggregation functions inside projection expressions are fixed Problem: Order by queries and the queries with functions like sum(a)+sum(b) are not working in MV. Please check jira for more details. Solution: The queries which have projection

[13/50] [abbrv] carbondata git commit: [HOTFIX][CARBONDATA-2716][DataMap] fix bug for loading datamap

[HOTFIX][CARBONDATA-2716][DataMap] fix bug for loading datamap In some scenarios, input parameter of getCarbonFactDataHandlerModel called carbonTable may be different from the one in loadmodel. This close #2497 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit:

[16/50] [abbrv] carbondata git commit: [CARBONDATA-2708][BloomDataMap] clear index file in case of data load failure

[CARBONDATA-2708][BloomDataMap] clear index file in case of data load failure When data loading failed, clean the index DataMap files that generated This closes #2463 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit:

[31/50] [abbrv] carbondata git commit: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2732][BloomDataMap] block some operations of bloomfilter datamap

[CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2732][BloomDataMap] block some operations of bloomfilter datamap 1.Block create bloomfilter datamap index on column which its datatype is complex type; 2.Block change datatype for bloomfilter index datamap; 3.Block dropping index columns for

[40/50] [abbrv] carbondata git commit: [CARBONDATA-2690][CarbonStore] implement RESTful API: create table, load data and select

http://git-wip-us.apache.org/repos/asf/carbondata/blob/4437920a/store/core/src/main/java/org/apache/carbondata/store/rpc/impl/StoreServiceImpl.java -- diff --git

[32/50] [abbrv] carbondata git commit: [HotFix] Getting carbon table identifier to datamap events

[HotFix] Getting carbon table identifier to datamap events Passing the table identifier to keep track of table in event in case preload and postload of datamap event. This closes #2448 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit:

[39/50] [abbrv] carbondata git commit: [CARBONDATA-2690][CarbonStore] implement RESTful API: create table, load data and select

http://git-wip-us.apache.org/repos/asf/carbondata/blob/4437920a/store/horizon/src/main/java/org/apache/carbondata/horizon/antlr/gen/ExpressionLexer.java -- diff --git

[10/50] [abbrv] carbondata git commit: [CARBONDATA-2720] Remove dead code

[CARBONDATA-2720] Remove dead code For acturate coverage results and easy maintainance This closes #2354 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/f9114036 Tree:

[19/50] [abbrv] carbondata git commit: [CARBONDATA-2717] fixed table id empty problem while taking drop lock

[CARBONDATA-2717] fixed table id empty problem while taking drop lock This closes #2472 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/637a9746 Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/637a9746

[38/50] [abbrv] carbondata git commit: [CARBONDATA-2690][CarbonStore] implement RESTful API: create table, load data and select

http://git-wip-us.apache.org/repos/asf/carbondata/blob/4437920a/store/horizon/src/main/java/org/apache/carbondata/horizon/rest/model/view/LoadRequest.java -- diff --git

[41/50] [abbrv] carbondata git commit: [CARBONDATA-2690][CarbonStore] implement RESTful API: create table, load data and select

[CARBONDATA-2690][CarbonStore] implement RESTful API: create table, load data and select This PR adds: 1.basic framework rewrite the carbon store's Master, Worker and Scheduler code in Java 2.RESTful API support create a table by using file meta store support load data to a table in single work

[14/50] [abbrv] carbondata git commit: [CARBONDATA-2719] Block update and delete on table having datamaps

[CARBONDATA-2719] Block update and delete on table having datamaps Table update/delete is needed to block on table which has datamaps. This close #2483 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/56e7dad7

[22/50] [abbrv] carbondata git commit: [CARBONDATA-2729][file-format] Schema Compatibility problem between version 1.3.0 and 1.4.0

[CARBONDATA-2729][file-format] Schema Compatibility problem between version 1.3.0 and 1.4.0 Problem: In TableSchema the field Name schemaEvaluation is changed to schemaEvoluation and in DataMapSchema field name className is changed to providerName. Due to this current Carbon version & Version

[42/50] [abbrv] carbondata git commit: [CARBONDATA-2613] Support csv based carbon table

http://git-wip-us.apache.org/repos/asf/carbondata/blob/2009009a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonAddSegmentCommand.scala -- diff --git

[15/50] [abbrv] carbondata git commit: [CARBONDATA-2734] Fix struct of date issue in create table

[CARBONDATA-2734] Fix struct of date issue in create table problem: Struct of date is not supported currently in create table flow as date datatype check is missing during parsing. Hence child date column was not appended with parent name, leading to StringOutOfIndex exception. solution: Handle

[28/50] [abbrv] carbondata git commit: [CARBONDATA-2714] Support merge index files for the segment

http://git-wip-us.apache.org/repos/asf/carbondata/blob/73419071/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableWithRowParser.scala -- diff --git

[50/50] [abbrv] carbondata git commit: [REBASE] Rebasing with master branch and Fixing rebase conflict

[REBASE] Rebasing with master branch and Fixing rebase conflict Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/239a6cad Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/239a6cad Diff:

[17/50] [abbrv] carbondata git commit: [CARBONDATA-2648] Fixed NPE issue with legacy store when CACHE_LEVEL is Blocklet

[CARBONDATA-2648] Fixed NPE issue with legacy store when CACHE_LEVEL is Blocklet Things done as part of this PR: Fixed Null pointer exception when store is of <= 1.1 version and DataMap is of type BlockletDataMap. Added clearing of SegmentProperties cache holder from executor Problem 1: Null

[34/50] [abbrv] carbondata git commit: [CARBONDATA-2724][DataMap]Unsupported create datamap on table with V1 or V2 format data

[CARBONDATA-2724][DataMap]Unsupported create datamap on table with V1 or V2 format data block creating datamap on carbon table with V1 or V2 format Currently the version info is read from carbon data file This closes #2488 Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo

[29/50] [abbrv] carbondata git commit: [CARBONDATA-2714] Support merge index files for the segment

[CARBONDATA-2714] Support merge index files for the segment Problem : The first-time query of carbon becomes very slow. It is because of reading many small carbonindex files and cache to the driver at the first time. Many carbonindex files are created in below case Loading data in large cluster

carbondata git commit: [CARBONDATA-2736][CARBONSTORE] Kafka integration with Carbon StreamSQL

Repository: carbondata Updated Branches: refs/heads/carbonstore 96fe233a2 -> 4b96ed8ca [CARBONDATA-2736][CARBONSTORE] Kafka integration with Carbon StreamSQL Modification in this PR: 1.Pass source table properties to streamReader.load() 2.Do not pass schema when sparkSession.readStream

[44/50] [abbrv] carbondata git commit: [CARBONDATA-2705][CarbonStore] CarbonStore Java API and Implementation

http://git-wip-us.apache.org/repos/asf/carbondata/blob/85cdc404/store/horizon/src/main/java/org/apache/carbondata/horizon/rest/model/descriptor/SelectDescriptor.java -- diff --git

[36/50] [abbrv] carbondata git commit: [CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework

http://git-wip-us.apache.org/repos/asf/carbondata/blob/d9b40bf9/store/core/src/main/scala/org/apache/carbondata/store/Master.scala -- diff --git a/store/core/src/main/scala/org/apache/carbondata/store/Master.scala

[20/50] [abbrv] carbondata git commit: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue fixes

[CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue fixes [CARBONDATA-2722][SDK] [JsonWriter] NPE when schema and data are not of same length or Data is null. problem: Null data is not handled in the json object to carbon row conversion. solution: add a null check when object is fetched from

[35/50] [abbrv] carbondata git commit: [CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework

http://git-wip-us.apache.org/repos/asf/carbondata/blob/d9b40bf9/store/search/src/main/scala/org/apache/spark/rpc/Scheduler.scala -- diff --git a/store/search/src/main/scala/org/apache/spark/rpc/Scheduler.scala

[26/50] [abbrv] carbondata git commit: [CARBONDATA-2704] Index file size in describe formatted command is not updated correctly with the segment file

[CARBONDATA-2704] Index file size in describe formatted command is not updated correctly with the segment file Problem: Describe formatted command is not showing correct index files size after index files merge. Solution: Segment file should be updated with the actual index files size of that

carbondata git commit: [CARBONDATA-2714][Merge Index] Fixed block dataMap cache refresh issue after creation of merge index file

Repository: carbondata Updated Branches: refs/heads/master 4612e0031 -> a26be1b18 [CARBONDATA-2714][Merge Index] Fixed block dataMap cache refresh issue after creation of merge index file Things handled as part of this PR Fixed block dataMap cache refresh issue after creation of merge index

carbondata git commit: [CARBONDATA-2738]Block Preaggregate, Compaction, Dictionary Exclude/Include for child columns for Complex datatype

Repository: carbondata Updated Branches: refs/heads/master a26be1b18 -> 0c9a60e01 [CARBONDATA-2738]Block Preaggregate, Compaction, Dictionary Exclude/Include for child columns for Complex datatype Block Preaggregate, Compaction, Dictionary Exclude/Include for child column s and Update Complex

carbondata git commit: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2732][BloomDataMap] block some operations of bloomfilter datamap

Repository: carbondata Updated Branches: refs/heads/master 8e7895715 -> 1c4358e89 [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2732][BloomDataMap] block some operations of bloomfilter datamap 1.Block create bloomfilter datamap index on column which its datatype is complex type; 2.Block

Jenkins build became unstable: carbondata-master-spark-2.2 » Apache CarbonData :: Spark Common Test #673

See

Jenkins build became unstable: carbondata-master-spark-2.2 #673

See

Jenkins build is back to normal : carbondata-master-spark-2.2 #671

See

carbondata git commit: [HotFix] Getting carbon table identifier to datamap events

Repository: carbondata Updated Branches: refs/heads/master 1c4358e89 -> aec47e06f [HotFix] Getting carbon table identifier to datamap events Passing the table identifier to keep track of table in event in case preload and postload of datamap event. This closes #2448 Project:

Jenkins build is back to normal : carbondata-master-spark-2.1 #2631

See