[GitHub] carbondata pull request #1988: [CARBONDATA-2193] Support register analyzer a...
Github user jackylk closed the pull request at: https://github.com/apache/carbondata/pull/1988 ---
[GitHub] carbondata pull request #1988: [CARBONDATA-2193] Support register analyzer a...
Github user jackylk closed the pull request at: https://github.com/apache/carbondata/pull/1988 ---
[GitHub] carbondata pull request #1988: [CARBONDATA-2193] Support register analyzer a...
GitHub user jackylk reopened a pull request:
https://github.com/apache/carbondata/pull/1988
[CARBONDATA-2193] Support register analyzer and optimizer rules for
MVDataMap
This PR is based on #1987
User can register analyzer and optimizer rules when creating CarbonSession,
for example:
```
val spark = SparkSession
.builder()
.config(conf)
.master(TestQueryExecutor.masterUrl)
.appName("Spark2TestQueryExecutor")
.enableHiveSupport()
.enableMVDataMap(new PreaggregateMVDataMapRules)
.getOrCreateCarbonSession()
```
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance
test report.
- Any additional information to help reviewers in testing this
change.
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata
datamap-query-olapdatamap
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1988.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1988
commit fa67d1b72c9ceb3f7a6707f8f31335c1937d77a1
Author: Jacky Li
Date: 2018-02-20T17:10:57Z
support datamap rules register
fix compile
commit de5e073a4acb18f131574464dbc522e752e2ed6e
Author: Jacky Li
Date: 2018-02-27T03:40:38Z
fix test
commit 8a291f7f9ab22ea7b6a283cc3b33b78266abfd5f
Author: Jacky Li
Date: 2018-02-27T16:49:38Z
fix comment
commit 520608fa032e6a574931b115ae1f517d24b2ed52
Author: Jacky Li
Date: 2018-02-28T14:50:06Z
fix comment
---
[GitHub] carbondata pull request #1988: [CARBONDATA-2193] Support register analyzer a...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1988#discussion_r170990002
--- Diff:
integration/spark2/src/main/java/org/apache/carbondata/datamap/preaggregate/PreaggregateMVDataMapRules.java
---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.preaggregate;
+
+import org.apache.carbondata.datamap.MVDataMapRules;
+
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.apache.spark.sql.catalyst.rules.Rule;
+import org.apache.spark.sql.hive.CarbonPreAggregateDataLoadingRules;
+import org.apache.spark.sql.hive.CarbonPreAggregateQueryRules;
+
+public class PreaggregateMVDataMapRules implements MVDataMapRules {
--- End diff --
fixed
---
[GitHub] carbondata pull request #1988: [CARBONDATA-2193] Support register analyzer a...
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1988#discussion_r170894492
--- Diff:
integration/spark2/src/main/java/org/apache/carbondata/datamap/preaggregate/PreaggregateMVDataMapRules.java
---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.preaggregate;
+
+import org.apache.carbondata.datamap.MVDataMapRules;
+
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.apache.spark.sql.catalyst.rules.Rule;
+import org.apache.spark.sql.hive.CarbonPreAggregateDataLoadingRules;
+import org.apache.spark.sql.hive.CarbonPreAggregateQueryRules;
+
+public class PreaggregateMVDataMapRules implements MVDataMapRules {
--- End diff --
add interface annotation
---
[GitHub] carbondata pull request #1988: [CARBONDATA-2193] Support register analyzer a...
GitHub user jackylk opened a pull request:
https://github.com/apache/carbondata/pull/1988
[CARBONDATA-2193] Support register analyzer and optimizer rules for
MVDatamMap
User can register analyzer and optimizer rules when creating CarbonSession,
for example:
```
val spark = SparkSession
.builder()
.config(conf)
.master(TestQueryExecutor.masterUrl)
.appName("Spark2TestQueryExecutor")
.enableHiveSupport()
.enableMVDataMap(new PreaggregateMVDataMapRules)
.getOrCreateCarbonSession()
```
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance
test report.
- Any additional information to help reviewers in testing this
change.
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata
datamap-query-olapdatamap
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1988.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1988
commit 952665a8c1c52f28951463fef989333ae0e6d83e
Author: Jacky Li
Date: 2018-01-06T12:28:44Z
[CARBONDATA-1992] Remove partitionId in CarbonTablePath
In CarbonTablePath, there is a deprecated partition id which is always 0,
it should be removed to avoid confusion.
This closes #1765
commit 111c3821557820241d1114d87eae2f7cd017e610
Author: Jacky Li
Date: 2018-01-02T15:46:14Z
[CARBONDATA-1968] Add external table support
This PR adds support for creating external table with existing carbondata
files, using Hive syntax.
CREATE EXTERNAL TABLE tableName STORED BY 'carbondata' LOCATION 'path'
This closes #1749
commit 80b42ac662ebd2bc243ca91c86b035717223daf4
Author: SangeetaGulia
Date: 2017-09-21T09:26:26Z
[CARBONDATA-1827] S3 Carbon Implementation
1.Provide support for s3 in carbondata.
2.Added S3Example to create carbon table on s3.
3.Added S3CSVExample to load carbon table using csv from s3.
This closes #1805
commit 71c2d8ca4a3212cff1eedbe78ee03e521f57fbbc
Author: Jacky Li
Date: 2018-01-31T16:25:31Z
[REBASE] Solve conflict after rebasing master
commit 15b4e192ee904a2e7c845ac67e0fcf1ba151a683
Author: Jacky Li
Date: 2018-01-30T13:24:04Z
[CARBONDATA-2099] Refactor query scan process to improve readability
Unified concepts in scan process flow:
1.QueryModel contains all parameter for scan, it is created by API in
CarbonTable. (In future, CarbonTable will be the entry point for various table
operations)
2.Use term ColumnChunk to represent one column in one blocklet, and use
ChunkIndex in reader to read specified column chunk
3.Use term ColumnPage to represent one page in one ColumnChunk
4.QueryColumn => ProjectionColumn, indicating it is for projection
This closes #1874
commit c3e99681bcd397ed33bc90e8d73b1fd33e0e60f7
Author: Jacky Li
Date: 2018-01-31T08:14:27Z
[CARBONDATA-2025] Unify all path construction through CarbonTablePath
static method
Refactory CarbonTablePath:
1.Remove CarbonStorePath and use CarbonTablePath only.
2.Make CarbonTablePath an utility without object creation, it can avoid
creating object before using it, thus code is cleaner and GC is less.
This closes #1768
commit e502c59a2d0b95d80db3aff04c749654254eadbe
Author: Jatin
Date: 2018-01-25T11:23:00Z
[CARBONDATA-2080] [S3-Implementation] Propagated hadoopConf from driver to
executor for s3 implementation in cluster mode.
Problem : hadoopconf was not getting propagated from driver to the executor
that's why load was failing to the distributed environment.
Solution: Setting the Hadoop conf in base class CarbonRDD
How to verify this PR :
Execute the load in the cluster mode It should be a success using location
s3.
This closes #1860
commit cae74a8cecea74e8899a87dcb7d12e0dec1b8069
Author: sounakr
Date: 2017-09-28T10:51:05Z
[CARBONDATA-1480]Min Max Index Example for DataMap
Datamap Example. Implementation of Min Max Index through Datamap. And Using
the Index while prunning.
This closes #1359
commit
