GitHub user ravipesala reopened a pull request:
https://github.com/apache/carbondata/pull/2043
[WIP] PR-2030 sdv fix
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance
test report.
- Any additional information to help reviewers in testing this
change.
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata pr-2030-new
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2043.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2043
----
commit 7dbac45223555b30de42aa5304397e535fd074fa
Author: SangeetaGulia <sangeeta.gulia@...>
Date: 2017-09-21T09:26:26Z
[CARBONDATA-1827] S3 Carbon Implementation
1.Provide support for s3 in carbondata.
2.Added S3Example to create carbon table on s3.
3.Added S3CSVExample to load carbon table using csv from s3.
This closes #1805
commit fd5d3f9a208934ab555bf4d180ac899f7437b40f
Author: sounakr <sounakr@...>
Date: 2017-09-28T10:51:05Z
[CARBONDATA-1480]Min Max Index Example for DataMap
Datamap Example. Implementation of Min Max Index through Datamap. And Using
the Index while prunning.
This closes #1359
commit ea31f6c0639752ca8fe298a01e439f6b36a9f471
Author: ravipesala <ravi.pesala@...>
Date: 2017-11-15T14:18:40Z
[CARBONDATA-1544][Datamap] Datamap FineGrain implementation
Implemented interfaces for FG datamap and integrated to filterscanner to
use the pruned bitset from FG datamap.
FG Query flow as follows.
1.The user can add FG datamap to any table and implement there interfaces.
2. Any filter query which hits the table with datamap will call prune
method of FGdatamap.
3. The prune method of FGDatamap return list FineGrainBlocklet , these
blocklets contain the information of block, blocklet, page and rowids
information as well.
4. The pruned blocklets are internally wriitten to file and returns only
the block , blocklet and filepath information as part of Splits.
5. Based on the splits scanrdd schedule the tasks.
6. In filterscanner we check the datamapwriterpath from split and reNoteads
the bitset if exists. And pass this bitset as input to it.
This closes #1471
commit b752b6dde963dd5dc0b12cbac1f02cf513168f67
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-02T15:46:14Z
[CARBONDATA-1968] Add external table support
This PR adds support for creating external table with existing carbondata
files, using Hive syntax.
CREATE EXTERNAL TABLE tableName STORED BY 'carbondata' LOCATION 'path'
This closes #1749
commit 5bedd77b0128e69f0853ce779de40e0105f7c801
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-06T12:28:44Z
[CARBONDATA-1992] Remove partitionId in CarbonTablePath
In CarbonTablePath, there is a deprecated partition id which is always 0,
it should be removed to avoid confusion.
This closes #1765
commit d0e3df7d9cfee4976b0f2acc45ddaaf825ea78a7
Author: Jatin <jatin.demla@...>
Date: 2018-01-25T11:23:00Z
[CARBONDATA-2080] [S3-Implementation] Propagated hadoopConf from driver to
executor for s3 implementation in cluster mode.
Problem : hadoopconf was not getting propagated from driver to the executor
that's why load was failing to the distributed environment.
Solution: Setting the Hadoop conf in base class CarbonRDD
How to verify this PR :
Execute the load in the cluster mode It should be a success using location
s3.
This closes #1860
commit bceb12175dfb5d62b9f0a1178e0daa3f40f75577
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-30T13:24:04Z
[CARBONDATA-2099] Refactor query scan process to improve readability
Unified concepts in scan process flow:
1.QueryModel contains all parameter for scan, it is created by API in
CarbonTable. (In future, CarbonTable will be the entry point for various table
operations)
2.Use term ColumnChunk to represent one column in one blocklet, and use
ChunkIndex in reader to read specified column chunk
3.Use term ColumnPage to represent one page in one ColumnChunk
4.QueryColumn => ProjectionColumn, indicating it is for projection
This closes #1874
commit 40872e618c268d16b6a62ab046b36f9b0c937b6d
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-31T08:14:27Z
[CARBONDATA-2025] Unify all path construction through CarbonTablePath
static method
Refactory CarbonTablePath:
1.Remove CarbonStorePath and use CarbonTablePath only.
2.Make CarbonTablePath an utility without object creation, it can avoid
creating object before using it, thus code is cleaner and GC is less.
This closes #1768
commit 47872e8fa10ec37c104c8eaacf6c889d5563f23a
Author: Jacky Li <jacky.likun@...>
Date: 2018-01-31T16:25:31Z
[REBASE] Solve conflict after rebasing master
commit 7791ed69c8e3bd03ee6ee7b3831f7e02c5254cb4
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-08T06:35:14Z
[CARBONDATA-2018][DataLoad] Optimization in reading/writing for sort temp
row
Pick up the no-sort fields in the row and pack them as bytes array and skip
parsing them during merge sort to reduce CPU consumption
This closes #1792
commit 51592c17aaa82a58a91e79085c153e1ac9abacb0
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-08T06:42:39Z
[CARBONDATA-2023][DataLoad] Add size base block allocation in data loading
Carbondata assign blocks to nodes at the beginning of data loading.
Previous block allocation strategy is block number based and it will
suffer skewed data problem if the size of input files differs a lot.
We introduced a size based block allocation strategy to optimize data
loading performance in skewed data scenario.
This closes #1808
commit 0fc046508e50f87a37c932abac64a522d5acb143
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-08T07:39:45Z
[HotFix][CheckStyle] Fix import related checkstyle
This closes #1952
commit a40ccda04629908b486c906fca93fc3be34fb1e8
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-08T17:39:20Z
[REBASE] Solve conflict after rebasing master
commit 21fda7795d55e4aa05ffde84598f0c8f0b030bea
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-10T02:34:59Z
Revert "[CARBONDATA-2023][DataLoad] Add size base block allocation in data
loading"
This reverts commit 6dd8b038fc898dbf48ad30adfc870c19eb38e3d0.
commit d5f9532a8923d56caf594d04b46f8279ddc551f5
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-10T11:44:23Z
[CARBONDATA-1997] Add CarbonWriter SDK API
Added a new module called store-sdk, and added a CarbonWriter API, it can
be used to write Carbondata files to a specified folder, without Spark and
Hadoop dependency. User can use this API in any environment.
This closes #1967
commit 3832d5e8aab151325c3227cfc9361823080ffdb9
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-10T12:11:25Z
Revert "[CARBONDATA-2018][DataLoad] Optimization in reading/writing for
sort temp row"
This reverts commit de92ea9a123b17d903f2d1d4662299315c792954.
commit 1d7b4e09606c86cb1d66cbd1949f7e746dbb39e1
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-11T02:12:10Z
[CARBONDATA-2156] Add interface annotation
InterfaceAudience and InterfaceStability annotation should be added for
user and developer
1.InetfaceAudience can be User and Developer
2.InterfaceStability can be Stable, Evolving, Unstable
This closes #1968
commit 9db10ca3ee404b10550eb8240f5f79415b01f192
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-27T00:51:25Z
[REBASE] resolve conflict after rebasing to master
commit f09af11d815f54690082d3defffcbb797606881f
Author: Ravindra Pesala <ravi.pesala@...>
Date: 2017-11-21T10:19:11Z
[CARBONDATA-1543] Supported DataMap chooser and expression for supporting
multiple datamaps in single query
This PR supports 3 features.
1.Load datamaps from the DataMapSchema which are created through DDL.
2.DataMap Chooser: It chooses the datamap out of available datamaps based
on simple logic. Like if there is filter condition on column1 then for
supposing 2 datamaps(1. column1 2. column1+column2) are supporting this column
then we choose the datamap which has fewer columns that is the first datamap.
3.Expression support: Based on the filter expressions we convert them to
the possible DataMap expressions and do apply expression on it.
For example, there are 2 datamaps available on table1
Datamap1 : column1
Datamap2 : column2
Query: select * from table1 where column1 ='a' and column2 =b
For the above query, we create datamap expression as
AndDataMapExpression(Datamap1, DataMap2). So for the above query both the
datamaps are included and the output of them will be applied AND condition to
improve the performance
This closes #1510
commit 07d0e690b9b1d70cf539775b902ca2baf6c75be0
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-08T06:35:14Z
[CARBONDATA-2018][DataLoad] Optimization in reading/writing for sort temp
row
Pick up the no-sort fields in the row and pack them as bytes array and skip
parsing them during merge sort to reduce CPU consumption
This closes #1792
commit 6dcb97f60e7abdf9980232592634b393f05b328e
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-08T06:42:39Z
[CARBONDATA-2023][DataLoad] Add size base block allocation in data loading
Carbondata assign blocks to nodes at the beginning of data loading.
Previous block allocation strategy is block number based and it will
suffer skewed data problem if the size of input files differs a lot.
We introduced a size based block allocation strategy to optimize data
loading performance in skewed data scenario.
This closes #1808
commit b11ceaff0caed97d65fa67eb3df2e2d54230095d
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-11T13:37:04Z
[CARBONDATA-2159] Remove carbon-spark dependency in store-sdk module
To make assembling JAR of store-sdk module, it should not depend on
carbon-spark module
This closes #1970
commit c0f463fe273111ef037cadcfcf71e26e298e3848
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-13T01:12:09Z
Support generating assembling JAR for store-sdk module
Support generating assembling JAR for store-sdk module and remove junit
dependency
This closes #1976
commit babf28ecff64174ae5ab9636f0a461f3e0998a77
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-13T02:58:06Z
[CARBONDATA-2091][DataLoad] Support specifying sort column bounds in data
loading
Enhance data loading performance by specifying sort column bounds
1. Add row range number during convert-process-step
2. Dispatch rows to each sorter by range number
3. Sort/Write process step can be done concurrently in each range
4. Since all sorttemp files will be written in one folders, we add range
number to the file name to distingush them
Tests added and docs updated
This closes #1953
commit 248302c2ec8244e03b4e035bed11fa5a38992f25
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-20T03:16:53Z
[CARBONDATA-2186] Add InterfaceAudience.Internal to annotate internal
interface
This closes #1986
commit 50e55987665821b568b3c495285c023063127246
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-22T12:59:59Z
[CARBONDATA-2189] Add DataMapProvider developer interface
Add developer interface for 2 types of DataMap:
1.IndexDataMap: DataMap that leveraging index to accelerate filter query
2.MVDataMap: DataMap that leveraging Materialized View to accelerate olap
style query, like SPJG query (select, predicate, join, groupby)
This PR adds support for following logic when creating and dropping the
DataMap
This closes #1987
commit 7f4b84802ddb11f6508b4ab47d6ab3c55cf79107
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-02-24T13:18:17Z
[CARBONDATA-1114][Tests] Fix bugs in tests in windows env
Fix bugs in tests that will cause failure under windows env
This closes #1994
commit 18ee863a9ca3cf1d65d69ad6216ecec88c182f5b
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-26T02:04:51Z
[HOTFIX] Add dava doc for datamap interface
1. Rename some of the datamap interface
2. Add more java doc for all public class of datamap interface
This closes #1998
commit 3e0d893d36ad11b9e61ab0eb6271546ca7c63346
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-26T08:30:38Z
[CARBONDATA-2206] support lucene index datamap
This PR is an initial effort to integrate lucene as an index datamap into
carbondata.
A new module called carbondata-lucene is added to support lucene datamap:
1.Add LuceneFineGrainDataMap, implement FineGrainDataMap interface.
2.Add LuceneCoarseGrainDataMap, implement CoarseGrainDataMap interface.
3.Support writing lucene index via LuceneDataMapWriter.
4.Implement LuceneDataMapFactory
5.A UDF called TEXT_MATCH is added, use it to do filtering on string column
by lucene
This closes #2003
commit 8c0d96c43ebb3a0553d3283201d6d3a226f6ebba
Author: Jacky Li <jacky.likun@...>
Date: 2018-02-27T03:26:30Z
[REBASE] Solve conflict after merging master
----
---
