[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2617 retest this please ---
[GitHub] carbondata pull request #2968: [CARBONDATA-3141] Removed Carbon Table Detail...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2968 [CARBONDATA-3141] Removed Carbon Table Detail Command Test case ### What has been changed? Removed Carbon Table Detail Command Test case, as this is not used. But every time there is a change in metadata or anything else, the test case has to be modified repeatedly. - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. Test case removed - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata tabledetail Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2968.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2968 commit 2f003fdebe83149440c665b13537c01b567424a3 Author: praveenmeenakshi56 Date: 2018-11-30T12:35:08Z Removed Carbon Table Detail Test case ---
[GitHub] carbondata pull request #2948: [CARBONDATA-3124] Updated log message in Unsa...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2948#discussion_r237020151
--- Diff: docs/faq.md ---
@@ -216,20 +216,18 @@
TimeZone.setDefault(TimeZone.getTimeZone("Asia/Shanghai"))
## How to check LRU cache memory footprint?
To observe the LRU cache memory footprint in the logs, configure the below
properties in log4j.properties file.
```
-log4j.logger.org.apache.carbondata.core.memory.UnsafeMemoryManager = DEBUG
log4j.logger.org.apache.carbondata.core.cache.CarbonLRUCache = DEBUG
```
-These properties will enable the DEBUG log for the CarbonLRUCache and
UnsafeMemoryManager which will print the information of memory consumed using
which the LRU cache size can be decided. **Note:** Enabling the DEBUG log will
degrade the query performance.
+This properties will enable the DEBUG log for the CarbonLRUCache and
UnsafeMemoryManager which will print the information of memory consumed using
which the LRU cache size can be decided. **Note:** Enabling the DEBUG log will
degrade the query performance. Ensure carbon.max.driver.lru.cache.size is
configured to observe the current cache size.
--- End diff --
+1
---
[GitHub] carbondata pull request #2948: [CARBONDATA-3124] Updated log message in Unsa...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2948#discussion_r236992267
--- Diff: docs/faq.md ---
@@ -216,20 +216,18 @@
TimeZone.setDefault(TimeZone.getTimeZone("Asia/Shanghai"))
## How to check LRU cache memory footprint?
To observe the LRU cache memory footprint in the logs, configure the below
properties in log4j.properties file.
```
-log4j.logger.org.apache.carbondata.core.memory.UnsafeMemoryManager = DEBUG
log4j.logger.org.apache.carbondata.core.cache.CarbonLRUCache = DEBUG
```
-These properties will enable the DEBUG log for the CarbonLRUCache and
UnsafeMemoryManager which will print the information of memory consumed using
which the LRU cache size can be decided. **Note:** Enabling the DEBUG log will
degrade the query performance.
+This properties will enable the DEBUG log for the CarbonLRUCache and
UnsafeMemoryManager which will print the information of memory consumed using
which the LRU cache size can be decided. **Note:** Enabling the DEBUG log will
degrade the query performance. Ensure carbon.max.driver.lru.cache.size is
configured to observe the current cache size.
--- End diff --
These is the correct usage. Revert back to the same
---
[GitHub] carbondata pull request #2766: [WIP] Added documentation for fallback condit...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2766 [WIP] Added documentation for fallback condition for complex columns in local Dictionary ### What has been changed? 1. Added documentation for fallback condition for complex columns in local Dictionary 2. Added documentation for system level property "carbon.local.dictionary.decoder.fallback" - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? Document Updated - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_dict_doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2766.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2766 commit f2e8692c0c47a3048c73411aac90172a153d521f Author: praveenmeenakshi56 Date: 2018-09-26T07:10:37Z Added fallback condition for complex columns in local Dictionary ---
[GitHub] carbondata pull request #2652: [WIP] Supported Sort Scope for SDK
Github user praveenmeenakshi56 closed the pull request at: https://github.com/apache/carbondata/pull/2652 ---
[GitHub] carbondata pull request #2652: [CARBONDATA-2879] Supported Sort Scope for SD...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2652 [CARBONDATA-2879] Supported Sort Scope for SDK ### What was the problem? For SDK, only Local sort was supported. ### What has been Changed? Hard coding of Local_Sort has been removed. Batch_Sort and No_Sort have been supported. In case of no options specified, default option changed to Local_ Sort - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? Yes, will be updated later. - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. Testing done and UT's added. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata sdk_sort Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2652.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2652 commit 0553bf8f912511c8a806106b1d76fd7f17fee984 Author: praveenmeenakshi56 Date: 2018-08-23T10:28:35Z Supported Sort Scope for SDK ---
[GitHub] carbondata pull request #2632: [CARBONDATA-2206] Enhanced document on Lucene...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2632 [CARBONDATA-2206] Enhanced document on Lucene datamap Support Enhanced documentation of Lucene DataMap - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? Document Updated - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata lucene_doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2632.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2632 commit 15f2929f0dafbf7b7d7f8a62c5ae9d2f66955528 Author: praveenmeenakshi56 Date: 2018-08-13T07:15:02Z Updated document on Lucene datamap Support ---
[GitHub] carbondata pull request #2625: [CARBONDATA-2750] Updated Documentation for L...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2625 [CARBONDATA-2750] Updated Documentation for Local Dictionary Support ### What has been added? Described about increase in memory footprint in local dictionary. Added to reduce block size when local dictionary is enabled, - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? Document updated. - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2625.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2625 commit 2710ffe6a55c7252e0ff962fbaf11f1719e570c1 Author: praveenmeenakshi56 Date: 2018-08-09T16:21:54Z Updated Documentation for Local Dictionary Support ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2617 retest this please ---
[GitHub] carbondata pull request #2621: [CARBONDATA-2840] Added SDV testcases for Com...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2621#discussion_r208528095
--- Diff:
integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/ComplexDataTypeTestCase.scala
---
@@ -0,0 +1,407 @@
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.cluster.sdv.generated
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream,
DataInputStream, File, InputStream}
+import java.sql.Timestamp
+
+import scala.collection.mutable
+
+import org.apache.avro.file.DataFileWriter
+import org.apache.avro.generic.{GenericDatumReader, GenericDatumWriter,
GenericRecord}
+import org.apache.avro.io.{DecoderFactory, Encoder}
+import org.apache.commons.io.FileUtils
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.common.util.QueryTest
+import org.apache.spark.sql.test.TestQueryExecutor
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.sdk.file.CarbonWriter
+
+/**
+ * Test Class for ComplexDataTypeTestCase to verify all scenerios
+ */
+
+class ComplexDataTypeTestCase extends QueryTest with BeforeAndAfterAll {
+
+ val filePath = TestQueryExecutor.integrationPath +
"/spark-common-test/src/test/resources"
+ val writerPath =
+s"${ resourcesPath }" + "/SparkCarbonFileFormat/WriterOutputComplex/"
+
+ override def beforeAll(): Unit = {
+FileUtils.deleteDirectory(new File(writerPath))
+sql("DROP TABLE IF EXISTS complexcarbontable")
+sql("DROP TABLE IF EXISTS test")
+sql("DROP TABLE IF EXISTS sdkOutputTable")
+ }
+
+ override def afterAll(): Unit = {
+FileUtils.deleteDirectory(new File(writerPath))
+sql("DROP TABLE IF EXISTS complexcarbontable")
+sql("DROP TABLE IF EXISTS test")
+sql("DROP TABLE IF EXISTS sdkOutputTable")
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+ .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT)
+ }
+
+ // check create table with complex data type
+ test("test Complex_DataType-001") {
+sql("DROP TABLE IF EXISTS test")
+sql(
+ "create table test(person
struct>) stored by 'carbondata'")
+sql("DROP TABLE IF EXISTS test")
+sql(
+ "create table test(p1 array,p2 array,p3
array,p4 array,p5 " +
+ "array,p6 array) stored by 'carbondata'")
+ }
--- End diff --
remove this test case. It is not making use of the cluster. If it is in UT,
it is enough
---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2617 retest this please ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2617 retest SDV please ---
[GitHub] carbondata pull request #2617: [CARBONDATA-2838] Added SDV test cases for Lo...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2617 [CARBONDATA-2838] Added SDV test cases for Local Dictionary Support Added SDV test cases for Local Dictionary Support - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. Test cases are added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_dict_sdv Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2617.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2617 commit d05ba074b239e3d52672e1638eece39367fddd40 Author: praveenmeenakshi56 Date: 2018-08-08T06:15:49Z Added SDV test cases for Local Dictionary Support ---
[GitHub] carbondata issue #2590: [CARBONDATA-2750] Updated documentation on Local Dic...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2590 retest this please ---
[GitHub] carbondata pull request #2552: [CARBONDATA-2781] Added fix for Null Pointer ...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2552#discussion_r206521693
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonCreateDataMapCommand.scala
---
@@ -181,7 +181,18 @@ case class CarbonCreateDataMapCommand(
override def undoMetadata(sparkSession: SparkSession, exception:
Exception): Seq[Row] = {
if (dataMapProvider != null) {
- dataMapProvider.cleanMeta()
+ if (dataMapSchema.getProviderName
--- End diff --
We need this check because we want the implementation of cleanMeta() method
in PreAggregregateDataMapProvider. For others, It will take from
IndexDataMapProvider
---
[GitHub] carbondata pull request #2590: [CARBONDATA-2750] Updated documentation on Lo...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2590 [CARBONDATA-2750] Updated documentation on Local Dictionary Supoort Updated Documentation on Local Dictionary Support. Changed default scenario for Local dictionary to false. - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? Document Updated - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2590.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2590 commit 540066df96d96f8a4f980441e9bb3636aa31 Author: praveenmeenakshi56 Date: 2018-07-31T10:19:10Z Updated documentation on Local Dictionary Supoort ---
[GitHub] carbondata issue #2552: [CARBONDATA-2781] Added fix for Null Pointer Excpeti...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2552 retest this please ---
[GitHub] carbondata issue #2552: [CARBONDATA-2781] Added fix for Null Pointer Excpeti...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2552 retest sdv please ---
[GitHub] carbondata issue #2552: [CARBONDATA-2781] Added fix for Null Pointer Excpeti...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2552 retest sdv please ---
[GitHub] carbondata issue #2552: [CARBONDATA-2781] Added fix for Null Pointer Excpeti...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2552 retest sdv please ---
[GitHub] carbondata pull request #2570: [wip]disable local dictionary by default
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2570#discussion_r205687209
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/localdictionary/LocalDictionarySupportCreateTableTest.scala
---
@@ -1575,7 +1571,7 @@ class LocalDictionarySupportCreateTableTest extends
QueryTest with BeforeAndAfte
val descLoc = sql("describe formatted local1").collect
descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled"))
match {
- case Some(row) => assert(row.get(1).toString.contains("true"))
+ case Some(row) => assert(row.get(1).toString.contains("false"))
--- End diff --
If local dictionary is disabled here, it will be the same as previous test
case. Can you make it true and add other properties like
local_dictionary_threshold or local_dictionary_include, etc.
---
[GitHub] carbondata pull request #2570: [wip]disable local dictionary by default
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2570#discussion_r205686883
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/localdictionary/LocalDictionarySupportCreateTableTest.scala
---
@@ -1554,7 +1550,7 @@ class LocalDictionarySupportCreateTableTest extends
QueryTest with BeforeAndAfte
val descLoc = sql("describe formatted local1").collect
descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled"))
match {
- case Some(row) => assert(row.get(1).toString.contains("true"))
+ case Some(row) => assert(row.get(1).toString.contains("false"))
--- End diff --
If local dictionary is disabled here, it will be the same as previous test
case. Can you make it true and add other properties like
local_dictionary_threshold or local_dictionary_include, etc.
---
[GitHub] carbondata pull request #2570: [wip]disable local dictionary by default
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2570#discussion_r205684804
--- Diff:
integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/CreateTableWithLocalDictionaryTestCase.scala
---
@@ -28,7 +28,7 @@ class CreateTableWithLocalDictionaryTestCase extends
QueryTest with BeforeAndAft
sql("DROP TABLE IF EXISTS LOCAL1")
}
- test("test local dictionary default configuration") {
+ test("test local dictionary default configuration") {
--- End diff --
remove the extra space.
---
[GitHub] carbondata issue #2552: [CARBONDATA-2781] Added fix for Null Pointer Excpeti...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2552 retest this please ---
[GitHub] carbondata issue #2552: [CARBONDATA-2781] Added fix for Null Pointer Excpeti...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2552 retest this please ---
[GitHub] carbondata pull request #2520: [CARBONDATA-2750] Added Documentation for Loc...
GitHub user praveenmeenakshi56 reopened a pull request: https://github.com/apache/carbondata/pull/2520 [CARBONDATA-2750] Added Documentation for Local Dictionary Support ### What has been added? Documentation for Local Dictionary Support has been added. - [x] Any interfaces changed? NA - [x] Any backward compatibility impacted? NA - [x] Document update required? Document has been added in this PR. - [x] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_dict_doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2520.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2520 commit 0e45c06137eac49508de1844bfc31321ba29acf2 Author: praveenmeenakshi56 Date: 2018-07-18T06:07:29Z Added Documentation for Local Dictionary Support Conflicts: docs/data-management-on-carbondata.md commit 9093c09463758aafca590ee4fd476a679902fe94 Author: praveenmeenakshi56 Date: 2018-07-25T15:08:05Z Added Documentation for Local Dictionary Support ---
[GitHub] carbondata pull request #2520: [CARBONDATA-2750] Added Documentation for Loc...
Github user praveenmeenakshi56 closed the pull request at: https://github.com/apache/carbondata/pull/2520 ---
[GitHub] carbondata pull request #2520: [CARBONDATA-2750] Added Documentation for Loc...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2520#discussion_r205108171
--- Diff: docs/data-management-on-carbondata.md ---
@@ -124,6 +124,41 @@ This tutorial is going to introduce all commands and
data operations on CarbonDa
TBLPROPERTIES ('streaming'='true')
```
+ - **Local Dictionary Configuration**
+
+ Local Dictionary is generated only for no-dictionary string/varchar
datatype columns. It helps in:
+ 1. Getting more compression on dimension columns with less cardinality.
+ 2. Filter queries and full scan queries on No-dictionary columns with
local dictionary will be faster as filter will be done on encoded data.
+ 3. Reducing the store size and memory footprint as only unique values
will be stored as part of local dictionary and corresponding data will be
stored as encoded data.
+
+ By default, Local Dictionary will be enabled and generated for all
no-dictionary string/varchar datatype columns.
+
+ Users will be able to pass following properties in create table
command:
+
+ | Properties | Default value | Description |
+ | -- | - | --- |
+ | LOCAL_DICTIONARY_ENABLE | true | By default, local dictionary
will be enabled for the table |
+ | LOCAL_DICTIONARY_THRESHOLD | 1 | The maximum cardinality for
local dictionary generation (range- 1000 to 10) |
--- End diff --
It is Segment/Task Level. Please refer to JIRA 2584.
---
[GitHub] carbondata pull request #2520: [CARBONDATA-2750] Added Documentation for Loc...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2520#discussion_r205102030
--- Diff: docs/data-management-on-carbondata.md ---
@@ -124,6 +124,41 @@ This tutorial is going to introduce all commands and
data operations on CarbonDa
TBLPROPERTIES ('streaming'='true')
```
+ - **Local Dictionary Configuration**
+
+ Local Dictionary is generated only for no-dictionary string/varchar
datatype columns. It helps in:
+ 1. Getting more compression on dimension columns with less cardinality.
+ 2. Filter queries and full scan queries on No-dictionary columns with
local dictionary will be faster as filter will be done on encoded data.
+ 3. Reducing the store size and memory footprint as only unique values
will be stored as part of local dictionary and corresponding data will be
stored as encoded data.
+
+ By default, Local Dictionary will be enabled and generated for all
no-dictionary string/varchar datatype columns.
+
+ Users will be able to pass following properties in create table
command:
+
+ | Properties | Default value | Description |
+ | -- | - | --- |
+ | LOCAL_DICTIONARY_ENABLE | true | By default, local dictionary
will be enabled for the table |
+ | LOCAL_DICTIONARY_THRESHOLD | 1 | The maximum cardinality for
local dictionary generation (range- 1000 to 10) |
+ | LOCAL_DICTIONARY_INCLUDE | all no-dictionary string/varchar
columns | Columns for which Local Dictionary is generated. |
+ | LOCAL_DICTIONARY_EXCLUDE | none | Columns for which Local
Dictionary is not generated |
+
--- End diff --
All the aforesaid are supported with Local Dictionary. The additional
information is already present in the Design Document in the JIRA. Please refer
the same.
---
[GitHub] carbondata pull request #2520: [CARBONDATA-2750] Added Documentation for Loc...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2520#discussion_r205094917
--- Diff: docs/data-management-on-carbondata.md ---
@@ -122,6 +122,45 @@ This tutorial is going to introduce all commands and
data operations on CarbonDa
TBLPROPERTIES ('streaming'='true')
```
+ - **Local Dictionary Configuration**
+
+ Local Dictionary is generated only for no-dictionary string/varchar
datatype columns. It helps in:
+ 1. Getting more compression on dimension columns with less cardinality.
+ 2. Filter queries and full scan queries on No-dictionary columns with
local dictionary will be faster as filter will be done on encoded data.
+ 3. Reducing the store size and memory footprint as only unique values
will be stored as part of local dictionary and corresponding data will be
stored as encoded data.
+
+ By default, Local Dictionary will be enabled and generated for all
no-dictionary string/varchar datatype columns.
--- End diff --
Data Loading Performance is affected only by 8%. Test with 3.5 billion
records (103 columns)
---
[GitHub] carbondata pull request #2552: [CARBONDATA-2781] Added fix for Null Pointer ...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2552 [CARBONDATA-2781] Added fix for Null Pointer Excpetion when create datamap killed from UI What was the issue? In undo meta, datamap was not being dropped. What is the solution? Datamap gets dropped when create command is killed from UI during undo meta. - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. Tested in Local - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata preagg_defct Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2552.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2552 commit 0039b4e42df96c41da0321325554a6894d4b613f Author: praveenmeenakshi56 Date: 2018-07-25T06:36:56Z Added fix for Null Pointer Excpetion when create datamap killed from UI ---
[GitHub] carbondata issue #2520: [CARBONDATA-2750] Added Documentation for Local Dict...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2520 retest this please ---
[GitHub] carbondata issue #2520: [CARBONDATA-2750] Added Documentation for Local Dict...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2520 retest this please ---
[GitHub] carbondata pull request #2520: [CARBONDATA-2750] Added Documentation for Loc...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2520 [CARBONDATA-2750] Added Documentation for Local Dictionary Support ### What has been added? Documentation for Local Dictionary Support has been added. - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? Document has been added in this PR. - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_dict_doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2520.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2520 commit d625284b0c8a371b92f41d722fc299e69412f426 Author: praveenmeenakshi56 Date: 2018-07-18T06:07:29Z Added Documentation for Local Dictionary Support ---
[GitHub] carbondata issue #2469: [CARBONDATA-2712] Added fix for Local Dictionary Exc...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2469 retest this please ---
[GitHub] carbondata issue #2469: [CARBONDATA-2712] Added fix for Local Dictionary Exc...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2469 retest this please ---
[GitHub] carbondata pull request #2469: [CARBONDATA-2712] Added fix for Local Diction...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2469#discussion_r201267613
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/localdictionary/LocalDictionarySupportAlterTableTest.scala
---
@@ -1161,6 +1159,126 @@ class LocalDictionarySupportAlterTableTest extends
QueryTest with BeforeAndAfter
}
}
+ test("test alter for local dictionary for complex columns when local
dictionary exclude is defined _001") {
+sql("drop table if exists local1")
+sql(
+ """
+| CREATE TABLE local1(id int, name string,city string, st
array>)
+| STORED BY 'org.apache.carbondata.format'
+|
tblproperties('long_string_columns'='name','local_dictionary_enable'='true')
+ """.stripMargin)
+sql("alter table local1 set
tblproperties('local_dictionary_exclude'='st,name')")
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold"))
match {
+ case Some(row) => assert(row.get(1).toString.contains("1"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled"))
match {
+ case Some(row) => assert(row.get(1).toString.contains("true"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Exclude"))
match {
+ case Some(row) =>
assert(row.get(1).toString.contains("st.val.sd,name"))
--- End diff --
si is int and Local Dictionary Include/Exclude will display only
no-dictionary string/varchar columns
---
[GitHub] carbondata pull request #2469: [CARBONDATA-2712] Added fix for Local Diction...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2469#discussion_r201267432
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/localdictionary/LocalDictionarySupportAlterTableTest.scala
---
@@ -1161,6 +1159,126 @@ class LocalDictionarySupportAlterTableTest extends
QueryTest with BeforeAndAfter
}
}
+ test("test alter for local dictionary for complex columns when local
dictionary exclude is defined _001") {
+sql("drop table if exists local1")
+sql(
+ """
+| CREATE TABLE local1(id int, name string,city string, st
array>)
+| STORED BY 'org.apache.carbondata.format'
+|
tblproperties('long_string_columns'='name','local_dictionary_enable'='true')
+ """.stripMargin)
+sql("alter table local1 set
tblproperties('local_dictionary_exclude'='st,name')")
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold"))
match {
+ case Some(row) => assert(row.get(1).toString.contains("1"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled"))
match {
+ case Some(row) => assert(row.get(1).toString.contains("true"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Exclude"))
match {
+ case Some(row) =>
assert(row.get(1).toString.contains("st.val.sd,name"))
+}
+ }
+
+ test("test alter for local dictionary for complex columns when local
dictionary exclude is defined _002") {
+sql("drop table if exists local1")
+sql(
+ """
+| CREATE TABLE local1(id int, name string,city string, st
array>,f string,g int,h string)
+| STORED BY 'org.apache.carbondata.format'
+|
tblproperties('long_string_columns'='name','local_dictionary_enable'='true','local_dictionary_include'='st')
+ """.stripMargin)
+sql("alter table local1 unset
tblproperties('local_dictionary_include')")
+sql("alter table local1 set
tblproperties('local_dictionary_exclude'='st,name,h')")
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold"))
match {
+ case Some(row) => assert(row.get(1).toString.contains("1"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled"))
match {
+ case Some(row) => assert(row.get(1).toString.contains("true"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Exclude"))
match {
+ case Some(row) =>
assert(row.get(1).toString.contains("h,st.val.sd,name"))
--- End diff --
Local Dictionary Exclude will display only no-dictionary string/varchar
columns. Those are the only child columns which will be displayed.
---
[GitHub] carbondata pull request #2469: Added fix for Local Dictionary Exclude for mu...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2469 Added fix for Local Dictionary Exclude for multi level complex columns ### What was the problem? When Local Dictionary Exclude was defined for multi level complex columns, the columns were still considered for Local Dictionary Include ### What has been changed? The index value was not getting updated on return from the recursive method needed for traversal. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_dict_complex_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2469.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2469 commit 7dab21ce5df001235d7956e0b4eb32bedf3bee22 Author: praveenmeenakshi56 Date: 2018-07-09T15:06:36Z Added fix for Local Dictionary Exclude for multi level complex columns ---
[GitHub] carbondata pull request #2422: [CARBONDATA-2585][CARBONDATA-2586][Local Dict...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2422#discussion_r200237535
--- Diff:
integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala
---
@@ -391,7 +495,7 @@ object TableNewProcessor {
if (dataType == DataTypes.DATE) {
encoders.add(Encoding.DIRECT_DICTIONARY)
}
-if (dataType == DataTypes.TIMESTAMP && !
highCardinalityDims.contains(colName)) {
--- End diff --
these are all handled in #2450
---
[GitHub] carbondata pull request #2422: [CARBONDATA-2585][CARBONDATA-2586][Local Dict...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2422#discussion_r200237575
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/localdictionary/LocalDictionarySupportAlterTableTest.scala
---
@@ -0,0 +1,1183 @@
+package org.apache.carbondata.spark.testsuite.localdictionary
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+import
org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
+
+class LocalDictionarySupportAlterTableTest extends QueryTest with
BeforeAndAfterAll{
+
+ override protected def beforeAll(): Unit = {
+sql("DROP TABLE IF EXISTS LOCAL1")
+ }
+
+ test("test alter table add column") {
+sql("drop table if exists local1")
+sql(
+ """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'org.apache.carbondata.format'
tblproperties('local_dictionary_enable'='true',
+|
'local_dictionary_threshold'='2','local_dictionary_include'='city','no_inverted_index'='name')
+ """.stripMargin)
+sql("alter table local1 add columns (alt string)
tblproperties('local_dictionary_include'='alt')")
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold"))
match {
+ case Some(row) => assert(row.get(1).toString.contains("2"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled"))
match {
+ case Some(row) => assert(row.get(1).toString.contains("true"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Include"))
match {
+ case Some(row) => assert(row.get(1).toString.contains("city,alt"))
+}
+ }
+
+ test("test alter table add column default configs for local dictionary")
{
+sql("drop table if exists local1")
+sql(
+ """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'org.apache.carbondata.format'
tblproperties('local_dictionary_enable'='true',
+| 'local_dictionary_threshold'='2','no_inverted_index'='name')
+ """.stripMargin)
+sql("alter table local1 add columns (alt string)")
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold"))
match {
+ case Some(row) => assert(row.get(1).toString.contains("2"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled"))
match {
+ case Some(row) => assert(row.get(1).toString.contains("true"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Include"))
match {
+ case Some(row) =>
assert(row.get(1).toString.contains("name,city,alt"))
+}
+ }
+
+ test("test alter table add column where same column is in dictionary
include and local dictionary include") {
+sql("drop table if exists local1")
+sql(
+ """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'org.apache.carbondata.format'
tblproperties('local_dictionary_enable'='true',
+|
'local_dictionary_threshold'='2','local_dictionary_include'='city','no_inverted_index'='name')
+ """.stripMargin)
+val exception = intercept[MalformedCarbonCommandException] {
+ sql(
+"alter table local1 add columns (alt string)
tblproperties('local_dictionary_include'='alt','dictionary_include'='alt')")
+}
+assert(exception.getMessage
+ .contains(
+"LOCAL_DICTIONARY_INCLUDE/LOCAL_DICTIONARY_EXCLUDE column: alt
specified in Dictionary " +
+"include. Local Dictionary will not be generated for Dictionary
include columns. " +
+"Please check the DDL."))
+ }
+
+ test("test alter table add column where duplicate columns present in
local dictionary include") {
+sql("drop table if exists local1")
+sql(
+ """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'org.apache.carbondata.format'
tblproperties('local_dictionary_enable'='true',
+|
'local_dictionary_threshold'='2','local_dictionary_
[GitHub] carbondata pull request #2450: [CARBONDATA-2689] Added validations for compl...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2450 [CARBONDATA-2689] Added validations for complex columns in alter set statements ### What was the issue Alter set statements were not validating complex dataType columns correctly. ### What is the solution Added a recursive method to validate string and varchar child columns of complex dataType columns. Added UTs for the same Dependent on #2422 - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. UT done, Test cases added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_dict_varchar Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2450.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2450 commit bb89f1139ddd1f35bddf541a3d701c0c1cc33c0b Author: brijoobopanna Date: 2018-06-27T09:22:46Z Added Validations for Complex datatype columns for local dictionary include and exclude Added test cases for local dictionary support for alter table,set, unset and preaggregate commit 7884a436f20d0660d61b6c8bb0e1313ad240cb68 Author: praveenmeenakshi56 Date: 2018-07-04T18:22:46Z Added check for Complex Type columns and added Test cases for alter statements ---
[GitHub] carbondata issue #2422: [CARBONDATA-2585][CARBONDATA-2586][Local Dictionary]...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2422 retest this please ---
[GitHub] carbondata issue #2390: [CARBONDATA-2624] Added validations for complex data...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2390 retest sdv please ---
[GitHub] carbondata issue #2390: [CARBONDATA-2624] Added validations for complex data...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2390 retest this please ---
[GitHub] carbondata pull request #2390: [CARBONDATA-2624] Added validations for compl...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2390 [CARBONDATA-2624] Added validations for complex dataType columns in create table command for Local Dictionary Support Added Validations for Complex DataType command in create table command. Added Unit Test cases for the same Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? will be updated in another PR - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. Unit Test cases tested and added in this PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_dict1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2390.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2390 commit ee3eeaf008ed14d32f10ab69f84ade4494bf522b Author: praveenmeenakshi56 Date: 2018-06-20T22:03:11Z Added validations for create table command with complex dataType columns for Local Dictionary Support ---
[GitHub] carbondata pull request #2377: [CARBONDATA-2611] Added Test Cases for Local ...
Github user praveenmeenakshi56 closed the pull request at: https://github.com/apache/carbondata/pull/2377 ---
[GitHub] carbondata issue #2377: [CARBONDATA-2611] Added Test Cases for Local Diction...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2377 retest this please ---
[GitHub] carbondata issue #2377: [CARBONDATA-2611] Added Test Cases for Local Diction...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2377 retest this please ---
[GitHub] carbondata pull request #2377: [CARBONDATA-2611] Added Test Cases for Local ...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2377 [CARBONDATA-2611] Added Test Cases for Local Dictionary Support for Create Table comand Dependent on #2375 - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. UT test cases added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_dict Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2377.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2377 commit 565a3e319cf12d5981b95097314efcae07cc79b6 Author: akashrn5 Date: 2018-06-06T15:03:39Z Support adding local dictionary configuration in create table statement and show the configs in describe formatted table commit c554c63757a0f00731d7d2b0f029ab6cdcefcff3 Author: praveenmeenakshi56 Date: 2018-06-14T18:06:59Z Added Test Case for Local Dictionary Support for Create table statement ---
[GitHub] carbondata issue #2175: [CARBONDATA-2352] Added SDV test cases for Partition...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2175 retest SDV please ---
[GitHub] carbondata issue #2175: [CARBONDATA-2352] Added SDV test cases for Partition...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2175 retest SDV please ---
[GitHub] carbondata issue #2175: [CARBONDATA-2352] Added SDV test cases for Partition...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2175 retest SDV please ---
[GitHub] carbondata issue #2175: [CARBONDATA-2352] Added SDV test cases for Partition...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2175 retest this please ---
[GitHub] carbondata issue #2259: [CARBONDATA-2426] Added fix for query execution fail...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2259 retest this please ---
[GitHub] carbondata pull request #2259: [CARBONDATA-2426] Added fix for data mismatch...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2259 [CARBONDATA-2426] Added fix for data mismatch after compaction on Partition with Pre-Aggregate tables **Problem:** Partition directory is getting deleted when one on the segment is marked for delete and another segment is loaded in the same partition. In case of aggregate table we only have partition specs for the specified segments therefore when the deleted segment is scanned against locationMap for stale partitions one of the valid partition is considered as stale as we dont get partitionSpecs for all the segments. **Solution:** Delete the index file instead of the partition directory and then delete the directory if it is empty. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata fix_ioex Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2259.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2259 commit 46b4e7d5dc3efc17cc19fa576207d13537499970 Author: praveenmeenakshi56 <praveenmeenakshi56@...> Date: 2018-05-02T10:09:28Z Added fix for data mismatch after compaction on Partition with Pre-Aggregate tables ---
[GitHub] carbondata pull request #2258: [CARBONDATA-2424] Added documentation for pro...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2258 [CARBONDATA-2424] Added documentation for properties of Pre-aggregate tables 'path' and 'partitioning' under DMPROPERTIES have been updated in the document - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? Document updated - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata doc_preagg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2258.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2258 commit 1c931c597fa17766f7f29ff0a0dbaf6ecc617613 Author: praveenmeenakshi56 <praveenmeenakshi56@...> Date: 2018-05-02T08:17:25Z Added documentation for properties of Pre-aggregate tables ---
[GitHub] carbondata issue #2175: [CARBONDATA-2352] Added SDV test cases for Partition...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2175 retest SDV please ---
[GitHub] carbondata issue #2175: [CARBONDATA-2352] Added SDV test cases for Partition...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2175 Added in SDVSuites .. ---
[GitHub] carbondata issue #2175: [CARBONDATA-2352] Added SDV test cases for Partition...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2175 retest SDV please ---
[GitHub] carbondata issue #2175: [CARBONDATA-2352] Added SDV test cases for Partition...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2175 retest SDV please ---
[GitHub] carbondata issue #2157: [CARBONDATA-2334] Added Property enabling user to bl...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2157 retest this please ---
[GitHub] carbondata issue #2175: [CARBONDATA-2352] Added SDV test cases for Partition...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2175 retest SDV please ---
[GitHub] carbondata issue #2166: [CARBONDATA-2341] Added Clean up of files for Pre-Ag...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2166 retest SDV please ---
[GitHub] carbondata issue #2170: [CARBONDATA-2346] Added fix for NULL error while dro...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2170 retest this please ---
[GitHub] carbondata issue #2157: [CARBONDATA-2334] Added Property enabling user to bl...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2157 retest this please ---
[GitHub] carbondata issue #2166: [CARBONDATA-2341] Added Clean up of files for Pre-Ag...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2166 retest SDV please ---
[GitHub] carbondata pull request #2170: [CARBONDATA-2346] Added fix for NULL error wh...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2170 [CARBONDATA-2346] Added fix for NULL error while dropping partition with multiple Pre-Aggregate tables Fixed null value issue for childcolumn - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata defect_part Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2170 commit dd3d3d1181847a1930048144740bfa053c878dd8 Author: praveenmeenakshi56 <praveenmeenakshi56@...> Date: 2018-04-13T10:31:35Z Added fix for error while dropping partition with multiple Pre-Aggregate tables ---
[GitHub] carbondata issue #2157: [CARBONDATA-2334] Added Property enabling user to bl...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2157 retest this please ---
[GitHub] carbondata pull request #2166: [CARBONDATA-2341] Added Clean up of files for...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2166 [CARBONDATA-2341] Added Clean up of files for Pre-Aggregate table Clean Files command on the main table will handle Cleanup for Pre-Aggregate tables - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata cleanup_preagg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2166.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2166 commit 1664fded2cc5c293f68865e3ebce4ae29354cebd Author: praveenmeenakshi56 <praveenmeenakshi56@...> Date: 2018-04-12T15:48:24Z Added Clean up of files for Pre-Aggregate table ---
[GitHub] carbondata pull request #2157: [CARBONDATA-2334] Added Property enabling use...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2157 [CARBONDATA-2334] Added Property enabling user to block partitioning of Pre-Aggregate t⦠While creating PreAgg table, in DMPROPERTIES, 'partitioning'='false' will disable partitioning of the Pre-Aggregate table - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata part_block Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2157.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2157 commit ac8e4db6fd1371ac0f28d0a9dc0b964de6a4e1d9 Author: praveenmeenakshi56 <praveenmeenakshi56@...> Date: 2018-04-11T08:43:14Z Added Property enabling user to block partitioning of Pre-Aggregate table ---
[GitHub] carbondata pull request #2147: [CARBONDATA-2322] Added fix for data mismatch...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2147 [CARBONDATA-2322] Added fix for data mismatch after compaction on Pre-agg with partition Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata defect_compact Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2147.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2147 commit f010276bd3dea3f1b7db4b96cff4d8f69496dbf1 Author: praveenmeenakshi56 <praveenmeenakshi56@...> Date: 2018-04-09T09:07:09Z added fix for data mismatch after compaction on Pre-agg with partition ---
[GitHub] carbondata pull request #2137: [HOTFIX] Removed block for Streaming with Pre...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2137 [HOTFIX] Removed block for Streaming with Preaggregate table Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata Preagg_Str Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2137.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2137 commit e5481fa2045d9836bb00fa303384dda2a23fe924 Author: praveenmeenakshi56 <praveenmeenakshi56@...> Date: 2018-04-03T12:20:21Z Removed block for Streaming with Preaggregate table ---
[GitHub] carbondata issue #2042: [CARBONDATA-2236]added sdv test cases for standard p...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2042 retest SDV please ---
[GitHub] carbondata issue #2042: [CARBONDATA-2236]added sdv test cases for standard p...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2042 retest this please ---
[GitHub] carbondata pull request #2066: [CARBONDATA-2257] Added SDV test cases for Pa...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2066 [CARBONDATA-2257] Added SDV test cases for Partition with Global Sort Added SDV test cases for Partition with Global Sort Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done - Verified in the cluster Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata global_partition Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2066.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2066 commit 02e35537305077dbb817d87b5dae52cdb4c438e5 Author: praveenmeenakshi56 <praveenmeenakshi56@...> Date: 2018-03-14T14:59:11Z Added SDV cases for Partition with Global Sort ---
[GitHub] carbondata issue #2042: [CARBONDATA-2236]added sdv test cases for standard p...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2042 retest this please ---
[GitHub] carbondata pull request #2042: [CARBONDATA-2236]added sdv test cases for sta...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2042 [CARBONDATA-2236]added sdv test cases for standard partition added sdv test cases for standard partition Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata partition_test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2042.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2042 commit 718af59351368db4577be1cad4808b9d789abf72 Author: praveenmeenakshi56 <praveenmeenakshi56@...> Date: 2018-03-07T15:02:39Z added sdv test cases for standard partition ---
[GitHub] carbondata pull request #1727: [CARBONDATA-1702] [Segment-Reading] added doc...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/1727 [CARBONDATA-1702] [Segment-Reading] added doc for "query from specified segments" feature Documentation of segment reading Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [X] Any interfaces changed? No - [X] Any backward compatibility impacted? No - [X] Document update required? NA - [X] Testing done NA - [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata praveenmeenakshi56-patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1727.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1727 commit 645cf97b8c98ea3fab22e61fc80b2554a06eb4d9 Author: praveenmeenakshi56 <praveenmeenakshi56@...> Date: 2017-12-27T14:00:52Z Update data-management-on-carbondata.md Documentation of segment reading ---
