GitHub user sgururajshetty opened a pull request:
https://github.com/apache/carbondata/pull/2603
[Documentation] Editorial review comment fixed
Minor issues fixed (spelling, syntax, and missing info)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sgururajshetty/carbondata editorial_review1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2603.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2603
----
commit 529f80dda6db3ce34e0baf766b03a9a13190b286
Author: sgururajshetty <sgururajshetty@...>
Date: 2018-07-25T12:44:07Z
Documentation for support for COLUMN_META_CACHE in create table and alter
table properties
commit d816aaa7a89155b3579906f960ed6a0ba4d4a59f
Author: sgururajshetty <sgururajshetty@...>
Date: 2018-07-25T12:48:43Z
Documentation to support for CACHE_LEVEL in create table and alter table
properties
commit 8ac243f8e9cff8359b6064352deb823eda7b9835
Author: sgururajshetty <sgururajshetty@...>
Date: 2018-07-25T13:24:52Z
Review comment fixed
commit 98501d35cfd110bcb9e75eb02628f3bce0c0f4ab
Author: sgururajshetty <sgururajshetty@...>
Date: 2018-07-25T13:26:58Z
review comment fixed
commit 62caf822cbcde1e519501c1d5db3c5cfc05fbd63
Author: Indhumathi27 <indhumathim27@...>
Date: 2018-07-21T10:46:21Z
[CARBONDATA-2606]Fix Complex array Pushdown and block auto merge compaction
1.Check for if Complex Column contains ArrayType at n levels and add parent
to projection if contains array.
2.Block Auto merge compaction for table containing complex datatype columns.
3.Fix Decimal Datatype scale and precision with two level struct type
4.Fix Dictionary Include for ComplexDataType
- If other complex columns other than first complex column is given in
dictionary include, then its insertion fails.
5.Fix BadRecord and dateformat for Complex primitive type-DATE
This closes #2535
commit d287a102b5c96e54261ac00c77038a1a56161fe9
Author: kumarvishal09 <kumarvishal1802@...>
Date: 2018-07-24T14:40:54Z
[CARBONDATA-2779]Fixed filter query issue in case of V1/v2 format store
Problem:
Filter query is failing for V1/V2 carbondata store
Root Cause:
in V1 store measure min max was not added in blockminmaxindex in executor
when filter is applied min max pruning is failing with array index out of cound
exception
Solution:
Need to add min max for measure column same as already handled in driver
block pruning
This closes #2550
commit b08745f68624ff066e0b23a41ce12d4a99618ac5
Author: Manhua <kevinjmh@...>
Date: 2018-07-25T08:51:49Z
[CARBONDATA-2783][BloomDataMap][Doc] Update document for bloom filter
datamap
add example for enable/disable datamap
This closes #2554
commit 964d26866468df6be130e9d65d339439cb4cf3b0
Author: praveenmeenakshi56 <praveenmeenakshi56@...>
Date: 2018-07-25T15:31:37Z
[CARBONDATA-2750] Added Documentation for Local Dictionary Support
Added Documentation for Local Dictionary Support
This closes #2520
commit 1fa9f64d70123d0bc988427a34c0750283f5daae
Author: BJangir <babulaljangir111@...>
Date: 2018-07-23T16:44:12Z
[CARBONDATA-2772] Size based dictionary fallback is failing even threshold
is not reached.
Issue:- Size Based Fallback happened even threshold is not reached.
RootCause:- Current size calculation is wrong. it is calculated for each
data. instead of generated dictionary data .
Solution :- Current size should be calculated only for generated dictionary
data.
This closes #2542
commit eae5817e56a20aecb7694c8d387dbb05b96e1045
Author: kunal642 <kunalkapoor642@...>
Date: 2018-07-24T10:42:54Z
[CARBONDATA-2778]Fixed bug when select after delete and cleanup is showing
empty records
Problem: In case if delete operation when it is found that the data being
deleted is leading to a state where one complete block data is getting
deleted. In that case the status if that block is marked for delete and
during the next delete operation run the block is deleted along with its
carbonIndex file. The problem arises due to deletion of carbonIndex file
because for multiple blocks there can be one carbonIndex file as one
carbonIndex file represents one task.
Solution: Do not delete the carbondata and carbonIndex file. After
compaction it will automatically take care of deleting the stale data and stale
segments.
This closes #2548
commit 6d6874a11482a8aa79f2280f6572e84b5e3cbc93
Author: dhatchayani <dhatcha.official@...>
Date: 2018-07-25T09:11:58Z
[CARBONDATA-2753][Compatibility] Row count of page is calculated wrong for
old store(V2 store)
Row count of page is calculated wrong for V2 store.
commit b6f5af6af96140876ec10ff09c3313d9b35ceb36
Author: Sssan520 <liangap2008@...>
Date: 2018-07-25T11:36:00Z
[CARBONDATA-2782]delete dead code in class 'CarbonCleanFilesCommand'
The variables(dmsãindexDms) in function processMetadata are nerver used.
This closes #2557
commit cf6bb46d89a9630209ef7f7e797ca8e156369206
Author: ajantha-bhat <ajanthabhat@...>
Date: 2018-07-24T07:03:47Z
[CARBONDATA-2775] Adaptive encoding fails for Unsafe OnHeap. if, target
datatype is SHORT_INT
problem:
[CARBONDATA-2775] Adaptive encoding fails for Unsafe OnHeap if, target data
type is SHORT_INT
solution: If ENABLE_OFFHEAP_SORT = false, in carbon property.
UnsafeFixLengthColumnPage.java will use different compress logic. Not the raw
compression. In that case, for SHORT_INT data type , conversion need to handle.
This closes #2546
commit 1fd3889cfec902e30cdcb9af6ed7b7f0df0f20e4
Author: mohammadshahidkhan <mohdshahidkhan1987@...>
Date: 2018-07-25T06:44:38Z
[HOTFIX] Fixed random test failure
Fixed random test failure
This closes #2553
commit 0f1aabb87b81670554188b698fc9fb4c598d8942
Author: Jatin <jatin.demla@...>
Date: 2018-07-25T19:12:50Z
[HOTFIX] CreateDataMapPost Event was skipped in case of preaggregate datamap
CreateDataMapPost Event was skipped in case of preaggregate datamap
This closes #2562
commit 9b05edbace9ccca5c7def6d5e4cc157ed0eb0d66
Author: ajantha-bhat <ajanthabhat@...>
Date: 2018-07-25T19:05:36Z
[CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed:Forever blocking wait
with more than 21 batch of data
problem: [CARBONDATA-2784]
[SDK writer] Forever blocking wait with more than 21 batch of data, when
consumer is dead due to data loading exception (bad record / out of memory)
root cause:
When the consumer is dead due to data loading exception, writer will be
forcefully closed. but queue.clear() cleared only snapshot of entries (10
batches) and close is set to true after that. In between clear() and close =
true, If more than 10 batches of data is again put into queue. For 11th batch,
queue.put() goes for forever block as consumer is dead.
Solution:
set close = true, before clearing the queue. This will avoid adding more
batches to queue from write().
problem [CARBONDATA-2786] NPE when SDK writer tries to write a file
solution and cause:
#2387 , in CarbonProperties.java
After systemLocation = getStorePath(); Null validation missing for
systemLocation.
because this can be null in SDK case. As Store location is not applicable
for SDK.
All a null validation.
This closes #2561
commit 1c620f83e3f5b1adb28b73f38c40021b2f06e1d0
Author: Indhumathi27 <indhumathim27@...>
Date: 2018-07-27T06:52:25Z
[CARBONDATA-2791]Fix Encoding for Double if exceeds LONG.Max_value
If Factor(decimalcount) * absMaxValue exceeds LONG.MAX_VALUE, then go for
direct compression.
This closes #2569
commit 39773ccf01345111bc950d8411646b886111f897
Author: manishgupta88 <tomanishgupta18@...>
Date: 2018-07-27T08:41:11Z
[HOTFIX] Removed file existence check to improve dataMap loading performance
Problem
DataMap loading performance degraded after adding file existence check.
Analysis
When carbonIndex file is read and carbondata file path to its metadata Info
map is prepared, file physical existence is getting checked every time which in
case of HDFS file system is a namenode call. This degrades the dataMap loading
performance. This is done to avoid failures for
Handle IUD scenario where after delete operation carbondata file is deleted
but the entry still exists in index file.
Fix
Modified code to check for physical file existence only in case when any
IUD operation has happened on the table
This closes #2560
commit a55722c1f6dad54d58e7cd69b9f48bfb3693c2a0
Author: Indhumathi27 <indhumathim27@...>
Date: 2018-07-27T18:09:50Z
[CARBONDATA-2794]Distinct count fails on ArrayOfStruct
This PR fixes Code Generator Error thrown when Select filter contains more
than one count of distinct of ArrayofStruct with group by Clause
This closes #2573
commit 6c4a2056f25ddc61d0fa0b205cb65a95c4298549
Author: mohammadshahidkhan <mohdshahidkhan1987@...>
Date: 2018-07-17T11:29:35Z
[CARBONDATA-2749][dataload] In HDFS Empty tablestatus file is written
during datalaod, iud or compaction when disk is full.
Problem:
When a failure happens due to disk full during load, IUD or Compaction,
then while updating the tablestatus file, the tablestaus.tmp file during
atomic file operation remains empty, and in the finally block the empty
tablestaus.tmp file is getting renamed to the actual file.
This leads to empty tablestatus file. Once such problem happens the
tablestatus file can not be retrieved and the already loaded data can be
used.
Solution:
If the failures happens during write then the the schema rename in the
finally
block must be avoided.
This closes #2517
commit 0a5431c1b645af8820486a55dd33530583ea9ae1
Author: Jimmy Casey <casey.jimmy@...>
Date: 2018-07-29T21:35:27Z
Fixed Spelling
Fixed Spelling
This closes #2584
commit 6fb852ad86cc7ee7dd97af211a6382ec8f4eb750
Author: chenliang613 <chenliang613@...>
Date: 2018-07-27T02:51:45Z
[CARBONDATA-2789] Support Hadoop 2.8.3 eco-system integration
Add hadoop 2.8.3 profile and passed the compile
This closes #2566
commit 937f68cf91bf6708de5c51fe787ea7672b3ee126
Author: ravipesala <ravi.pesala@...>
Date: 2018-07-30T12:45:32Z
[CARBONDATA-2801]Added documentation for flat folder
[CARBONDATA-2801]Added documentation for flat folder
This closes #2582
commit e1abff71c7feba90be0ab338ec3b5aabf9cf9380
Author: ajantha-bhat <ajanthabhat@...>
Date: 2018-07-25T13:51:02Z
[CARBONDATA-2606][Complex DataType Enhancements]Fix Null result if
projection column have null primitive column and struct
Problem:
In case if the actual value of the primitive data type is null, by PR#2489,
we are moving all the null values to the end of the collected row without
considering the data type.
Solution:
Place null in the end of output iff the null value is of complex primitive
column.
This closes #2559
commit 917b230b57cff9c350524c81d939b06af6a35cde
Author: akashrn5 <akashnilugal@...>
Date: 2018-07-27T07:01:48Z
[CARBONDATA-2585]disable local dictionary by default
make local dictionary false by default
This closes #2570
commit 5d2d6cf053accdd76bf26943db0a0f09dfc492e1
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-07-26T15:22:58Z
[HotFix][CARBONDATA-2788][BloomDataMap] Fix bugs in incorrect query result
with bloom datamap
This PR solve two problems which will affect the correctness of the query
on bloom.
Revert PR2539
After review the code, we found that modification in PR2539 is not needed,
so we revert that PR.
Bugs in overflow for blocklet count
Carbondata stores blocklet count for each block in byte data type, when a
block contains more than 128 blocklets, it will overflow the byte
limits. Here we change the data type to short.
For cache_leve=block, after pruning by main BlockDataMap, the blockletNo in
Blocklet is -1, which indicate that the following percedure will scan
the whole block -- all the blocklets in the block. So, when doing
intersection with the pruned result from BloomDataMap, we
need to take care of these blocklets. In this implementation, we added the
result from BloomDataMap based on the blocklet's existence in BlockDataMap.
This closes #2565
commit ec73f91817062495b085a231279f5133e44b1af8
Author: Indhumathi27 <indhumathim27@...>
Date: 2018-07-30T08:48:44Z
[CARBONDATA-2798] Fix Dictionary_Include for ComplexDataType
Problem1:
Select Filter is throwing BufferUnderFlow Exception as cardinality is
filled for Non-Dictionary columns.
Solution:
Check if a complex column has Encoding => Dictionary and fill cardinality
for that column only.
Problem2:
Transactional Table is throwing NullPointerException if csv fileheader is
not proper.
Solution:
Throw CarbonDataLoadingException if csv fileheader is not proper.
This closes #2578
commit 6fc498055801ab6cf5181b24ee4db47c0510f172
Author: akashrn5 <akashnilugal@...>
Date: 2018-07-30T13:52:29Z
[CARBONDATA-2805] Fix the ordering mismatch of segment numbers during cutom
compaction
Problem:
when we have segments from 0 to 6 and i give 1, 2, 3 for custom compaction,
then it should create 1.1 as compacted segment, but sometimes
it will create 3.1 as compacted segment which is wrong. This is beacuse
custom Segment IDs were passing in hashset and finally inserted in
hashmap, while identifying segments to be merged. hashmap and hashset does
not guarantee the insertion order which may lead to missmatch of segment
numbers.
Solution:
Use LinkedHashSet and LinkedHashMap which always sure about the insertion
order.
This closes #2585
commit 2f828ce2f438de67dabf565be25b9bbad76b49fc
Author: rahul <rahul.kumar@...>
Date: 2018-07-04T14:01:51Z
[CARBONDATA-2625] While BlockletDataMap loading, avoid multiple times
listing of files
CarbonReader is very slow for many files as blockletDataMap lists files
of folder while loading each segment. This optimization lists once across
segment loads.
This closes #2441
commit 436c8642185c7d262582c55c6c0a077922125be5
Author: xuchuanyin <xuchuanyin@...>
Date: 2018-07-27T03:54:21Z
[CARBONDATA-2790][BloomDataMap]Optimize default parameter for bloomfilter
datamap
To provide better query performance for bloomfilter datamap by default,
we optimize bloom_size from 32000 to 640000 and optimize bloom_fpp from
0.01 to 0.00001.
This closes #2567
----
---
