[
https://issues.apache.org/jira/browse/CARBONDATA-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405356#comment-15405356
]
ASF GitHub Bot commented on CARBONDATA-80:
------------------------------------------
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/44#discussion_r73283633
--- Diff:
core/src/main/java/org/carbondata/core/cache/dictionary/ColumnDictionaryInfo.java
---
@@ -112,10 +113,50 @@ public ColumnDictionaryInfo(DataType dataType) {
/**
* This method will add a new dictionary chunk to existing list of
dictionary chunks
*
- * @param dictionaryChunk
+ * @param newDictionaryChunk
*/
- @Override public void addDictionaryChunk(List<byte[]> dictionaryChunk) {
- dictionaryChunks.add(dictionaryChunk);
+ @Override public void addDictionaryChunk(List<byte[]>
newDictionaryChunk) {
+ if (dictionaryChunks.size() > 0) {
+ // Ensure that each time a new dictionary chunk is getting added to
the
+ // dictionary chunks list, equal distribution of dictionary values
should
+ // be there in the sublists of dictionary chunk list
+ List<byte[]> lastDictionaryChunk =
dictionaryChunks.get(dictionaryChunks.size() - 1);
+ int dictionaryOneChunkSize = CarbonUtil.getDictionaryChunkSize();
+ int differenceInLastDictionaryAndOneChunkSize =
+ dictionaryOneChunkSize - lastDictionaryChunk.size();
+ if (differenceInLastDictionaryAndOneChunkSize > 0) {
+ // if difference is greater than new dictionary size then copy a
part of list
+ // else copy the complete new dictionary chunk list in the last
dictionary chunk list
+ if (differenceInLastDictionaryAndOneChunkSize >=
newDictionaryChunk.size()) {
+ lastDictionaryChunk.addAll(newDictionaryChunk);
+ } else {
+ List<byte[]> subListOfNewDictionaryChunk =
+ newDictionaryChunk.subList(0,
differenceInLastDictionaryAndOneChunkSize);
+ lastDictionaryChunk.addAll(subListOfNewDictionaryChunk);
--- End diff --
@kumarvishal09 ...sublist will create a new object but it will return a
view of the parent list itself and all operations will be performed on the
parent list. It will not create and fill a new array. So I think it should be
ok to use sublist
> Dictionary values should be equally distributed in buckets while loading in
> memory
> ----------------------------------------------------------------------------------
>
> Key: CARBONDATA-80
> URL: https://issues.apache.org/jira/browse/CARBONDATA-80
> Project: CarbonData
> Issue Type: Improvement
> Reporter: Manish Gupta
> Assignee: Manish Gupta
> Priority: Minor
>
> Whenever a query is executed, dictionary for columns queried is loaded in
> memory. For incremental loads dictionary values are loaded incrementally and
> thus one list contains several sub lists with dictionary values.
> The dictionary values on incremental load may not be equally distributed in
> the sub buckets and this might increase the search time of a value if there
> are too many incremental loads.
> Therefore the dictionary values should be divided equally in the sub buckets.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
