[GitHub] carbondata pull request #2897: [CARBONDATA-3080] Supporting local dictionary...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2897 ---
[GitHub] carbondata pull request #2897: [CARBONDATA-3080] Supporting local dictionary...
Github user BJangir commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2897#discussion_r243812841 --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/util/CarbonVectorizedRecordReader.java --- @@ -68,6 +67,10 @@ private AbstractDetailQueryResultIterator iterator; private QueryModel queryModel; + //This holds mapping of fetch index with respect to project col index. + // it is used when same col is used in projection many times.So need to fetch only that col. + private List projectionMapping = new ArrayList<>(); --- End diff -- During initBatch only it is done(not frequent lookup/creation ). ---
[GitHub] carbondata pull request #2897: [CARBONDATA-3080] Supporting local dictionary...
Github user BJangir commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2897#discussion_r243812352
--- Diff:
core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java
---
@@ -173,6 +174,7 @@ public synchronized void freeMemoryAll(String taskId) {
"Freeing offheap working memory of size %d. Current available
memory is %d",
occuppiedMemory, totalMemory - memoryUsed));
}
+ThreadLocalTaskInfo.clearCarbonTaskInfo();
--- End diff --
OK..
---
[GitHub] carbondata pull request #2897: [CARBONDATA-3080] Supporting local dictionary...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2897#discussion_r243810091 --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/util/CarbonVectorizedRecordReader.java --- @@ -68,6 +67,10 @@ private AbstractDetailQueryResultIterator iterator; private QueryModel queryModel; + //This holds mapping of fetch index with respect to project col index. + // it is used when same col is used in projection many times.So need to fetch only that col. + private List projectionMapping = new ArrayList<>(); --- End diff -- Better use an array to have fast lookups instead of list ---
[GitHub] carbondata pull request #2897: [CARBONDATA-3080] Supporting local dictionary...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2897#discussion_r243810009
--- Diff:
core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java
---
@@ -173,6 +174,7 @@ public synchronized void freeMemoryAll(String taskId) {
"Freeing offheap working memory of size %d. Current available
memory is %d",
occuppiedMemory, totalMemory - memoryUsed));
}
+ThreadLocalTaskInfo.clearCarbonTaskInfo();
--- End diff --
Better call explicitly from methods where we call `freeMemoryAll, thread
clearing should not be part of it.
---
[GitHub] carbondata pull request #2897: [CARBONDATA-3080] Supporting local dictionary...
Github user BJangir commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2897#discussion_r242629291
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/LocalDictDimensionDataChunkStore.java
---
@@ -94,10 +93,9 @@ public void fillVector(int[] invertedIndex, int[]
invertedIndexReverse, byte[] d
}
@Override public void fillRow(int rowId, CarbonColumnVector vector, int
vectorRow) {
-if (!dictionary.isDictionaryUsed()) {
- vector.setDictionary(dictionary);
- dictionary.setDictionaryUsed();
-}
+// always set dictionary otherwise
+// empty dictionary will get set if same col is called again in
projection.
+vector.setDictionary(dictionary);
--- End diff --
1. For Session it is ok, issue happens only in SDK reader , Now hanlded in
org.apache.carbondata.hadoop.util.CarbonVectorizedRecordReader.
LocalDictDimensionDataChunkStore.java reverted back.
2. OK. Done
---
[GitHub] carbondata pull request #2897: [CARBONDATA-3080] Supporting local dictionary...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2897#discussion_r242423291
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/LocalDictDimensionDataChunkStore.java
---
@@ -94,10 +93,9 @@ public void fillVector(int[] invertedIndex, int[]
invertedIndexReverse, byte[] d
}
@Override public void fillRow(int rowId, CarbonColumnVector vector, int
vectorRow) {
-if (!dictionary.isDictionaryUsed()) {
- vector.setDictionary(dictionary);
- dictionary.setDictionaryUsed();
-}
+// always set dictionary otherwise
+// empty dictionary will get set if same col is called again in
projection.
+vector.setDictionary(dictionary);
--- End diff --
@BJangir
1. Please check and confirm if the same problem occurs with CarbonSession
also
2. Modify the PR description and specify the details for bug fixed in this
PR after completion of point 1
---
