[
https://issues.apache.org/jira/browse/CARBONDATA-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401972#comment-15401972
]
ASF GitHub Bot commented on CARBONDATA-80:
------------------------------------------
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/44#discussion_r72971441
--- Diff:
core/src/main/java/org/carbondata/core/cache/dictionary/AbstractColumnDictionaryInfo.java
---
@@ -241,22 +257,21 @@ private void decrementAccessCount() {
*/
protected byte[] getDictionaryBytesFromSurrogate(int surrogateKey) {
byte[] dictionaryValueInBytes = null;
- int totalSizeOfDictionaryChunksTraversed = 0;
- for (List<byte[]> oneDictionaryChunk : dictionaryChunks) {
- totalSizeOfDictionaryChunksTraversed =
- totalSizeOfDictionaryChunksTraversed + oneDictionaryChunk.size();
- // skip the dictionary chunk till surrogate key is lesser than size
of
- // dictionary chunks traversed
- if (totalSizeOfDictionaryChunksTraversed < surrogateKey) {
- continue;
+ if (surrogateKey > 0) {
+ // surrogate key starts from 1 and list index will start from 0, so
lets say if surrogate
+ // key is 10 then value will present at index 9 of the dictionary
chunk list
+ int actualSurrogateIndex = surrogateKey - 1;
+ // lets say dictionaryOneChunkSize = 10, surrogateKey = 10, so
bucket index will
+ // be 0 and dictionary chunk index will be 9 to get the value
+ int dictionaryBucketIndex = actualSurrogateIndex /
dictionaryOneChunkSize;
+ if (dictionaryChunks.size() > dictionaryBucketIndex) {
+ int indexInsideBucket = actualSurrogateIndex %
dictionaryOneChunkSize;
+ List<byte[]> dictionaryBucketContainingSurrogateValue =
+ dictionaryChunks.get(dictionaryBucketIndex);
+ if (dictionaryBucketContainingSurrogateValue.size() >
indexInsideBucket) {
--- End diff --
Same as above.Please check
> Dictionary values should be equally distributed in buckets while loading in
> memory
> ----------------------------------------------------------------------------------
>
> Key: CARBONDATA-80
> URL: https://issues.apache.org/jira/browse/CARBONDATA-80
> Project: CarbonData
> Issue Type: Improvement
> Reporter: Manish Gupta
> Assignee: Manish Gupta
> Priority: Minor
>
> Whenever a query is executed, dictionary for columns queried is loaded in
> memory. For incremental loads dictionary values are loaded incrementally and
> thus one list contains several sub lists with dictionary values.
> The dictionary values on incremental load may not be equally distributed in
> the sub buckets and this might increase the search time of a value if there
> are too many incremental loads.
> Therefore the dictionary values should be divided equally in the sub buckets.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
