[GitHub] incubator-carbondata pull request #339: [CARBONDATA-429][WIP] Remove unneces...
Github user piaoyats commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/339#discussion_r92345835
--- Diff:
core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java
---
@@ -167,12 +167,9 @@ private Dictionary getDictionary(
DictionaryColumnUniqueIdentifier dictionaryColumnUniqueIdentifier)
throws CarbonUtilException {
Dictionary reverseDictionary = null;
-// create column dictionary info object only if dictionary and its
-// metadata file exists for a given column identifier
-if (!isFileExistsForGivenColumn(dictionaryColumnUniqueIdentifier)) {
- throw new CarbonUtilException(
- "Either dictionary or its metadata does not exist for column
identifier :: "
- + dictionaryColumnUniqueIdentifier.getColumnIdentifier());
+// create column dictionary info object only if it is primitive type.
+if (dictionaryColumnUniqueIdentifier.getDataType().isComplexType()) {
+ return null;
--- End diff --
So, here logic is right.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #339: [CARBONDATA-429][WIP] Remove unneces...
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/339#discussion_r92332659
--- Diff:
core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java
---
@@ -167,12 +167,9 @@ private Dictionary getDictionary(
DictionaryColumnUniqueIdentifier dictionaryColumnUniqueIdentifier)
throws CarbonUtilException {
Dictionary reverseDictionary = null;
-// create column dictionary info object only if dictionary and its
-// metadata file exists for a given column identifier
-if (!isFileExistsForGivenColumn(dictionaryColumnUniqueIdentifier)) {
- throw new CarbonUtilException(
- "Either dictionary or its metadata does not exist for column
identifier :: "
- + dictionaryColumnUniqueIdentifier.getColumnIdentifier());
+// create column dictionary info object only if it is primitive type.
+if (dictionaryColumnUniqueIdentifier.getDataType().isComplexType()) {
+ return null;
--- End diff --
We will not invoke getDictionary() method on complex type column directly.
For complex type column, we call getDictionary() method on primitive type
sub-column.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #339: [CARBONDATA-429][WIP] Remove unneces...
Github user piaoyats commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/339#discussion_r92331735
--- Diff:
core/src/main/java/org/apache/carbondata/core/cache/dictionary/ReverseDictionaryCache.java
---
@@ -167,12 +167,9 @@ private Dictionary getDictionary(
DictionaryColumnUniqueIdentifier dictionaryColumnUniqueIdentifier)
throws CarbonUtilException {
Dictionary reverseDictionary = null;
-// create column dictionary info object only if dictionary and its
-// metadata file exists for a given column identifier
-if (!isFileExistsForGivenColumn(dictionaryColumnUniqueIdentifier)) {
- throw new CarbonUtilException(
- "Either dictionary or its metadata does not exist for column
identifier :: "
- + dictionaryColumnUniqueIdentifier.getColumnIdentifier());
+// create column dictionary info object only if it is primitive type.
+if (dictionaryColumnUniqueIdentifier.getDataType().isComplexType()) {
+ return null;
--- End diff --
I am not sure this condition check is right or not? Maybe we still need to
do dictionary encoding for complex type?
@QiangCai pls review this logic
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #339: [CARBONDATA-429][WIP] Remove unneces...
GitHub user jackylk opened a pull request: https://github.com/apache/incubator-carbondata/pull/339 [CARBONDATA-429][WIP] Remove unnecessary file name check in dictionary cache In dictionary cache, there are currently necessary file name check for each column, which increase the number of HDFS interactions unnecessarily. In this PR, file name check is replaced by primitive type check, and returns null if it is a complex type. Dictionary cache support for complex type is WIP. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jackylk/incubator-carbondata dictionary Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/339.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #339 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
