Problem: Array Index of bound exception thrown from dictionary look up while writing sort index file
Analysis: Whenever we load dictionary data into memory, then in case of populating reverse dictionary object sometimes a chunk which has no value is also getting added to the dictionary chunk list. This is happening because the logic for dictionary chunk distribution in case of forward dictionary is not implemented for reverse dictionary and 0 size dictionary chunks are not getting removed while adding to the list of dictionary chunks. Solution: Add the same distribution logic we have in forward dictionary for populating reverse dictionary object Impact area: Sort index generation Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/ad14039e Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/ad14039e Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/ad14039e Branch: refs/heads/branch-0.1 Commit: ad14039ef4e298345e48c9c4985754d6d5ae2c50 Parents: 7e3d46c Author: manishgupta88 <tomanishgupt...@gmail.com> Authored: Thu Sep 8 13:08:56 2016 +0530 Committer: ravipesala <ravi.pes...@gmail.com> Committed: Thu Sep 22 09:35:09 2016 +0530 ---------------------------------------------------------------------- .../core/cache/dictionary/DictionaryCacheLoaderImpl.java | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/ad14039e/core/src/main/java/org/apache/carbondata/core/cache/dictionary/DictionaryCacheLoaderImpl.java ---------------------------------------------------------------------- diff --git a/core/src/main/java/org/apache/carbondata/core/cache/dictionary/DictionaryCacheLoaderImpl.java b/core/src/main/java/org/apache/carbondata/core/cache/dictionary/DictionaryCacheLoaderImpl.java index e798c01..ce3107b 100644 --- a/core/src/main/java/org/apache/carbondata/core/cache/dictionary/DictionaryCacheLoaderImpl.java +++ b/core/src/main/java/org/apache/carbondata/core/cache/dictionary/DictionaryCacheLoaderImpl.java @@ -95,7 +95,7 @@ public class DictionaryCacheLoaderImpl implements DictionaryCacheLoader { int dictionaryChunkSize = CarbonUtil.getDictionaryChunkSize(); int sizeOfLastDictionaryChunk = dictionaryInfo.getSizeOfLastDictionaryChunk(); int sizeOfOneDictionaryChunk = dictionaryChunkSize - sizeOfLastDictionaryChunk; - if (sizeOfOneDictionaryChunk == 0) { + if (sizeOfOneDictionaryChunk <= 0) { sizeOfOneDictionaryChunk = dictionaryChunkSize; } List<List<byte[]>> dictionaryChunks = @@ -111,7 +111,9 @@ public class DictionaryCacheLoaderImpl implements DictionaryCacheLoader { } } for (List<byte[]> dictionaryChunk : dictionaryChunks) { - dictionaryInfo.addDictionaryChunk(dictionaryChunk); + if (!dictionaryChunk.isEmpty()) { + dictionaryInfo.addDictionaryChunk(dictionaryChunk); + } } }
