[GitHub] carbondata issue #2524: [CARBONDATA-2532][Integration] Carbon to support spa...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2524 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6019/ ---
[GitHub] carbondata issue #2567: [CARBONDATA-2790][BloomDataMap]Optimize default para...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2567 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6300/ ---
[GitHub] carbondata issue #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix bugs in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2565 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6299/ ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user rahulforallp commented on the issue: https://github.com/apache/carbondata/pull/2441 retest this please ---
[GitHub] carbondata issue #2567: [CARBONDATA-2790][BloomDataMap]Optimize default para...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2567 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7547/ ---
[GitHub] carbondata issue #2552: [CARBONDATA-2781] Added fix for Null Pointer Excpeti...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2552 retest this please ---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Github user mohammadshahidkhan commented on the issue: https://github.com/apache/carbondata/pull/2517 retest sdv please ---
[GitHub] carbondata issue #2566: [CARBONDATA-2789] Support Hadoop 2.8.3 eco-system in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2566 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6298/ ---
[GitHub] carbondata issue #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix bugs in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2565 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7545/ ---
[GitHub] carbondata issue #2555: [CARBONDATA-2753][Compatibility] Row count of page i...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2555 retest sdv please ---
[GitHub] carbondata pull request #2560: [HOTFIX] Removed file existence check to impr...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2560#discussion_r205666480
--- Diff:
core/src/main/java/org/apache/carbondata/core/indexstore/BlockletDataMapIndexStore.java
---
@@ -81,6 +82,8 @@ public BlockletDataMapIndexWrapper
get(TableBlockIndexUniqueIdentifierWrapper id
List dataMaps = new ArrayList<>();
if (blockletDataMapIndexWrapper == null) {
try {
+boolean isCarbonTableUpdated =
--- End diff --
Already list files done to get the locations of files, so better check in
the map `carbonDataFileBlockMetaInfoMapping`
---
[GitHub] carbondata pull request #2533: [wip][CARBONDATA-2765]handle flat folder supp...
Github user akashrn5 closed the pull request at: https://github.com/apache/carbondata/pull/2533 ---
[GitHub] carbondata issue #2566: [CARBONDATA-2789] Support Hadoop 2.8.3 eco-system in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2566 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7544/ ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2561 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6296/ ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2561 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7542/ ---
[jira] [Updated] (CARBONDATA-2788) Fix bugs in incorrect query result with bloom datamap
[ https://issues.apache.org/jira/browse/CARBONDATA-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xuchuanyin updated CARBONDATA-2788: --- Issue Type: Sub-task (was: Bug) Parent: CARBONDATA-2632 > Fix bugs in incorrect query result with bloom datamap > - > > Key: CARBONDATA-2788 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2788 > Project: CarbonData > Issue Type: Sub-task >Reporter: xuchuanyin >Assignee: xuchuanyin >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > revert modification in PR2539 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2567: [CARBONDATA-2790][BloomDataMap]Optimize defau...
GitHub user xuchuanyin opened a pull request: https://github.com/apache/carbondata/pull/2567 [CARBONDATA-2790][BloomDataMap]Optimize default parameter for bloomfilter datamap To provide better query performance for bloomfilter datamap by default, we optimize bloom_size from 32000 to 64 and optimize bloom_fpp from 0.01 to 0.1. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xuchuanyin/carbondata 0727_bloom_default_para Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2567.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2567 commit 50780e8159ae46e6a7c9596ee9d1ad8dd09048a5 Author: xuchuanyin Date: 2018-07-27T03:54:21Z Optimize default parameter for bloomfilter datamap To provide better query performance for bloomfilter datamap by default, we optimize bloom_size from 32000 to 64 and optimize bloom_fpp from 0.01 to 0.1. ---
[jira] [Created] (CARBONDATA-2790) Optimize default parameter for bloomfilter datamap
xuchuanyin created CARBONDATA-2790: -- Summary: Optimize default parameter for bloomfilter datamap Key: CARBONDATA-2790 URL: https://issues.apache.org/jira/browse/CARBONDATA-2790 Project: CarbonData Issue Type: Sub-task Reporter: xuchuanyin Assignee: xuchuanyin Optimize default parameter for bloomfilter datamap to provide better query performance by default -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2441 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6018/ ---
[jira] [Updated] (CARBONDATA-2788) Fix bugs in incorrect query result with bloom datamap
[ https://issues.apache.org/jira/browse/CARBONDATA-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xuchuanyin updated CARBONDATA-2788: --- Summary: Fix bugs in incorrect query result with bloom datamap (was: Revert optimization in blockletId in rebuilding datamap) > Fix bugs in incorrect query result with bloom datamap > - > > Key: CARBONDATA-2788 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2788 > Project: CarbonData > Issue Type: Bug >Reporter: xuchuanyin >Assignee: xuchuanyin >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > revert modification in PR2539 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix bugs in ...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2565 retest this please ---
[GitHub] carbondata issue #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix bugs in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2565 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6297/ ---
[GitHub] carbondata issue #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix bugs in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2565 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7543/ ---
[GitHub] carbondata pull request #2566: [CARBONDATA-2789] Support Hadoop 2.8.3 eco-sy...
GitHub user chenliang613 opened a pull request: https://github.com/apache/carbondata/pull/2566 [CARBONDATA-2789] Support Hadoop 2.8.3 eco-system integration Add hadoop 2.8.3 profile and passed the compile You can merge this pull request into a Git repository by running: $ git pull https://github.com/chenliang613/carbondata hadoop2.8.3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2566.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2566 commit 3e59ab844ee186f91ad788f2a62e44c900f78f97 Author: chenliang613 Date: 2018-07-27T02:51:45Z [CARBONDATA-2789] Support Hadoop 2.8.3 eco-system integration ---
[GitHub] carbondata pull request #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix b...
Github user xuchuanyin commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2565#discussion_r205657632 --- Diff: integration/spark2/src/main/scala/org/apache/carbondata/datamap/IndexDataMapRebuildRDD.scala --- @@ -357,13 +357,20 @@ class IndexDataMapRebuildRDD[K, V]( // skip clear datamap and we will do this adter rebuild reader.setSkipClearDataMapAtClose(true) +// currently blockletId in rowWithPosition is wrong, we cannot use it --- End diff -- OK ---
[GitHub] carbondata pull request #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix b...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2565#discussion_r205657316
--- Diff:
datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java
---
@@ -103,7 +106,19 @@ public void init(DataMapModel dataMapModel) throws
IOException {
/**
* init field converters for index columns
*/
- public void initIndexColumnConverters(CarbonTable carbonTable,
List indexedColumn) {
+ public void initIndexColumnConverters(CarbonTable carbonTable, String
dataMapName,
+ List indexedColumn) {
+String cacheLevel = MapUtils.getString(
+carbonTable.getTableInfo().getFactTable().getTableProperties(),
+CarbonCommonConstants.CACHE_LEVEL,
CarbonCommonConstants.CACHE_LEVEL_DEFAULT_VALUE);
+this.isBlockletCacheLevel = cacheLevel.equalsIgnoreCase("blocklet");
+if (!this.isBlockletCacheLevel) {
+ LOGGER.warn(
+ String.format("BloomFilter datamap %s runs with
cache_level=block for table %s.%s,"
+ + " which may decrease its pruning performance",
--- End diff --
OK
---
[GitHub] carbondata issue #2546: [CARBONDATA-2775] Adaptive encoding fails for Unsafe...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2546 @kumarvishal09 : PR is ready. please Merge ---
[GitHub] carbondata issue #2559: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2559 @kumarvishal09 : PR is ready please review ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2561 retest this please ---
[GitHub] carbondata pull request #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix b...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2565#discussion_r205656230 --- Diff: integration/spark2/src/main/scala/org/apache/carbondata/datamap/IndexDataMapRebuildRDD.scala --- @@ -357,13 +357,20 @@ class IndexDataMapRebuildRDD[K, V]( // skip clear datamap and we will do this adter rebuild reader.setSkipClearDataMapAtClose(true) +// currently blockletId in rowWithPosition is wrong, we cannot use it --- End diff -- This is a bit confusing, can you rephrase it ---
[GitHub] carbondata pull request #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix b...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2565#discussion_r205655699
--- Diff:
datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java
---
@@ -103,7 +106,19 @@ public void init(DataMapModel dataMapModel) throws
IOException {
/**
* init field converters for index columns
*/
- public void initIndexColumnConverters(CarbonTable carbonTable,
List indexedColumn) {
+ public void initIndexColumnConverters(CarbonTable carbonTable, String
dataMapName,
+ List indexedColumn) {
+String cacheLevel = MapUtils.getString(
+carbonTable.getTableInfo().getFactTable().getTableProperties(),
+CarbonCommonConstants.CACHE_LEVEL,
CarbonCommonConstants.CACHE_LEVEL_DEFAULT_VALUE);
+this.isBlockletCacheLevel = cacheLevel.equalsIgnoreCase("blocklet");
+if (!this.isBlockletCacheLevel) {
+ LOGGER.warn(
+ String.format("BloomFilter datamap %s runs with
cache_level=block for table %s.%s,"
+ + " which may decrease its pruning performance",
--- End diff --
change to `which may decrease its pruning benefit, which lead to read more
data`
---
[jira] [Resolved] (CARBONDATA-2782) dead code in class 'CarbonCleanFilesCommand'
[
https://issues.apache.org/jira/browse/CARBONDATA-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacky Li resolved CARBONDATA-2782.
--
Resolution: Fixed
> dead code in class 'CarbonCleanFilesCommand'
>
>
> Key: CARBONDATA-2782
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2782
> Project: CarbonData
> Issue Type: Improvement
>Reporter: lianganping
>Assignee: lianganping
>Priority: Trivial
> Fix For: 1.4.1
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> class:CarbonCleanFilesCommand
> dead code:
> override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
> carbonTable = CarbonEnv.getCarbonTable(databaseNameOp,
> tableName.get)(sparkSession)
> {color:#ff}val dms =
> carbonTable.getTableInfo.getDataMapSchemaList.asScala.map(_.getDataMapName){color}
> {color:#ff} val indexDms =
> DataMapStoreManager.getInstance.getAllDataMap(carbonTable).asScala{color}
> {color:#ff} .filter(_.getDataMapSchema.isIndexDataMap){color}
> ...
> }
> the variables(dms、indexDms) are nerver used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata pull request #2557: [CARBONDATA-2782]delete dead code in class 'C...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2557 ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2561 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6017/ ---
[GitHub] carbondata issue #2544: [CARBONDATA-2776][CarbonStore] Support ingesting dat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2544 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6293/ ---
[GitHub] carbondata issue #2544: [CARBONDATA-2776][CarbonStore] Support ingesting dat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2544 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7539/ ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2441 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6016/ ---
[jira] [Created] (CARBONDATA-2789) Support Hadoop 2.8.3 eco-system integration
Liang Chen created CARBONDATA-2789: -- Summary: Support Hadoop 2.8.3 eco-system integration Key: CARBONDATA-2789 URL: https://issues.apache.org/jira/browse/CARBONDATA-2789 Project: CarbonData Issue Type: Task Components: hadoop-integration Reporter: Liang Chen Assignee: Liang Chen -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2441 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6015/ ---
[GitHub] carbondata issue #2524: [CARBONDATA-2532][Integration] Carbon to support spa...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2524 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6014/ ---
[GitHub] carbondata issue #2562: [HOTFIX] CreateDataMapPost Event was skipped in case...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2562 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6013/ ---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2517 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6012/ ---
[GitHub] carbondata issue #2553: [HOTFIX] Fixed random test failure
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2553 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6011/ ---
[GitHub] carbondata issue #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Revert optim...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2565 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6292/ ---
[GitHub] carbondata issue #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Revert optim...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2565 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7538/ ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2561 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6010/ ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2561 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6289/ ---
[GitHub] carbondata pull request #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Rever...
GitHub user xuchuanyin opened a pull request: https://github.com/apache/carbondata/pull/2565 [HotFix][CARBONDATA-2788][BloomDataMap] Revert optimization for blockletId in rebuilding datamap We found querying huge data with rebuilding bloom datamap will give incorrect result. The root cause is that the blockletId in ResultCollector is wrong. (This was introduced in PR2539) We will revert the previous modification for this. Now it is checked and works fine. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xuchuanyin/carbondata 0726_revert_rebuild_rdd_blockletno Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2565.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2565 commit 8889078ea9d1328366dc27d633b3f5ebf1906322 Author: xuchuanyin Date: 2018-07-26T15:22:58Z Revert optimize blockletId in rebuilding datamap We found querying huge data with rebuilding bloom datamap will give incorrect result. The root cause is that the blockletId in ResultCollector is wrong. (This was introduced in PR2539) We will revert the previous modification for this. Now it is checked and works fine. ---
[jira] [Created] (CARBONDATA-2788) Revert optimization in blockletId in rebuilding datamap
xuchuanyin created CARBONDATA-2788: -- Summary: Revert optimization in blockletId in rebuilding datamap Key: CARBONDATA-2788 URL: https://issues.apache.org/jira/browse/CARBONDATA-2788 Project: CarbonData Issue Type: Bug Reporter: xuchuanyin Assignee: xuchuanyin revert modification in PR2539 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2524: [CARBONDATA-2532][Integration] Carbon to support spa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2524 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7537/ ---
[GitHub] carbondata issue #2524: [CARBONDATA-2532][Integration] Carbon to support spa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2524 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6291/ ---
[GitHub] carbondata issue #2546: [CARBONDATA-2775] Adaptive encoding fails for Unsafe...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2546 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6009/ ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2441 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6290/ ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2441 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7536/ ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2561 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7535/ ---
[GitHub] carbondata issue #2553: [HOTFIX] Fixed random test failure
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2553 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6285/ ---
[GitHub] carbondata issue #2559: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2559 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6008/ ---
[GitHub] carbondata issue #2553: [HOTFIX] Fixed random test failure
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2553 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7531/ ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2441 retest sdv please ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2441 retest this please ---
[jira] [Created] (CARBONDATA-2787) Data load on table with single pass true having preaggregate datamaps without precreated dictionary throws incorrect error message
Chetan Bhat created CARBONDATA-2787:
---
Summary: Data load on table with single pass true having
preaggregate datamaps without precreated dictionary throws incorrect error
message
Key: CARBONDATA-2787
URL: https://issues.apache.org/jira/browse/CARBONDATA-2787
Project: CarbonData
Issue Type: Bug
Components: data-load
Affects Versions: 1.4.1
Environment: Spark 2.2
Reporter: Chetan Bhat
Steps :
Create table .
Create 5 datamaps.
load data in table with single pass true without precreated dictionary
configuration.
0: jdbc:hive2://10.18.98.136:23040/default> CREATE TABLE
uniqdata_decimal(CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string, DOB
timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
int) STORED BY 'org.apache.carbondata.format'
TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1',
'SORT_SCOPE'='GLOBAL_SORT');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (0.21 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> create datamap uniqdata_agg_decimal
on table uniqdata_decimal using 'preaggregate' as select cust_name,
avg(DECIMAL_COLUMN1) from uniqdata_decimal group by DECIMAL_COLUMN1, cust_name;
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (0.557 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> create datamap
uniqdata_agg_sum_decimal on table uniqdata_decimal using 'preaggregate' as
select cust_name, sum(DECIMAL_COLUMN1) from uniqdata_decimal group by
DECIMAL_COLUMN1, cust_name;
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (0.515 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> create datamap
uniqdata_agg_count_decimal on table uniqdata_decimal using 'preaggregate' as
select cust_name, count(DECIMAL_COLUMN1) from uniqdata_decimal group by
DECIMAL_COLUMN1, cust_name;
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (0.492 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> create datamap
uniqdata_agg_min_decimal on table uniqdata_decimal using 'preaggregate' as
select cust_name, min(DECIMAL_COLUMN1) from uniqdata_decimal group by
DECIMAL_COLUMN1, cust_name;
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (0.571 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> create datamap
uniqdata_agg_max_decimal on table uniqdata_decimal using 'preaggregate' as
select cust_name, max(DECIMAL_COLUMN1) from uniqdata_decimal group by
DECIMAL_COLUMN1, cust_name;
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (0.471 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> show datamap on table
uniqdata_decimal;
+-+---+---+-+--+
| DataMapName | ClassName | Associated Table | DataMap Properties |
+-+---+---+-+--+
| uniqdata_agg_decimal | preaggregate |
test.uniqdata_decimal_uniqdata_agg_decimal | |
| uniqdata_agg_sum_decimal | preaggregate |
test.uniqdata_decimal_uniqdata_agg_sum_decimal | |
| uniqdata_agg_count_decimal | preaggregate |
test.uniqdata_decimal_uniqdata_agg_count_decimal | |
| uniqdata_agg_min_decimal | preaggregate |
test.uniqdata_decimal_uniqdata_agg_min_decimal | |
| uniqdata_agg_max_decimal | preaggregate |
test.uniqdata_decimal_uniqdata_agg_max_decimal | |
+-+---+---+-+--+
5 rows selected (0.035 seconds)
0: jdbc:hive2://10.18.98.136:23040/default> LOAD DATA INPATH
'hdfs://hacluster/chetan/2000_UniqData.csv' into table uniqdata_decimal
OPTIONS('DELIMITER'=',',
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1',
*'SINGLE_PASS'='true')*;
Issue : Load fails with incorrect error message as shown below.
0: jdbc:hive2://10.18.98.136:23040/default> LOAD DATA INPATH
'hdfs://hacluster/chetan/2000_UniqData.csv' into table uniqdata_decimal
OPTIONS('DELIMITER'=',',
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1',
*'SINGLE_PASS'='true'*);
*Error: java.lang.Exception: DataLoad failure (state=,code=0)*
Expected : The error message should be
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2441 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6283/ ---
[GitHub] carbondata issue #2524: [CARBONDATA-2532][Integration] Carbon to support spa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2524 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7532/ ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2561 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7530/ ---
[GitHub] carbondata issue #2524: [CARBONDATA-2532][Integration] Carbon to support spa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2524 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6286/ ---
[GitHub] carbondata pull request #2560: [HOTFIX] Removed file existence check to impr...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2560#discussion_r205436561
--- Diff:
core/src/main/java/org/apache/carbondata/core/indexstore/BlockletDataMapIndexStore.java
---
@@ -81,6 +82,8 @@ public BlockletDataMapIndexWrapper
get(TableBlockIndexUniqueIdentifierWrapper id
List dataMaps = new ArrayList<>();
if (blockletDataMapIndexWrapper == null) {
try {
+boolean isCarbonTableUpdated =
--- End diff --
Listing files from metadata folder will be less costly as compared to
listing files from segment path. Metadata contains only few files but segment
path can contain lot of files which will do more RPC calls to namenode
Also listing from segment file path will be very costly in case of flat
folder structure OR when storage file system is S3
---
[GitHub] carbondata issue #2562: [HOTFIX] CreateDataMapPost Event was skipped in case...
Github user jatin9896 commented on the issue: https://github.com/apache/carbondata/pull/2562 retest sdv please ---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2517 retest sdv please ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2561 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6284/ ---
[GitHub] carbondata issue #2562: [HOTFIX] CreateDataMapPost Event was skipped in case...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2562 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6280/ ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2561 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7528/ ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2441 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7529/ ---
[GitHub] carbondata issue #2562: [HOTFIX] CreateDataMapPost Event was skipped in case...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2562 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7526/ ---
[GitHub] carbondata pull request #2560: [HOTFIX] Removed file existence check to impr...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2560#discussion_r205411295
--- Diff:
core/src/main/java/org/apache/carbondata/core/indexstore/BlockletDataMapIndexStore.java
---
@@ -81,6 +82,8 @@ public BlockletDataMapIndexWrapper
get(TableBlockIndexUniqueIdentifierWrapper id
List dataMaps = new ArrayList<>();
if (blockletDataMapIndexWrapper == null) {
try {
+boolean isCarbonTableUpdated =
--- End diff --
Why not list files inside segment path once to list al carbodata files and
check the existance in that list?
---
[GitHub] carbondata issue #2559: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2559 retest sdv please ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2561 retest sdv please ---
[GitHub] carbondata issue #2546: [CARBONDATA-2775] Adaptive encoding fails for Unsafe...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2546 retest sdv please ---
[GitHub] carbondata pull request #2546: [CARBONDATA-2775] Adaptive encoding fails for...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2546#discussion_r205410245
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestAdaptiveEncodingUnsafeHeapColumnPageForComplexDataType.scala
---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.integration.spark.testsuite.complexType
+
+import java.io.File
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+/**
+ * Test class of Adaptive Encoding UnSafe Column Page with Complex Data
type
+ *
+ */
+
+class TestAdaptiveEncodingUnsafeHeapColumnPageForComplexDataType
+ extends QueryTest with BeforeAndAfterAll with TestAdaptiveComplexType {
+
+ override def beforeAll(): Unit = {
+
+new
File(CarbonProperties.getInstance().getSystemFolderLocation).delete()
+sql("DROP TABLE IF EXISTS adaptive")
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.ENABLE_UNSAFE_COLUMN_PAGE,
+"true")
+
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.ENABLE_OFFHEAP_SORT,
+"false")
+ }
+
--- End diff --
@manishgupta88 :
1. This new class extends, **TestAdaptiveComplexType** . So all the test
case present in that will run with this carbon property set. we need both the
scenarios to test with and without property.
2. Actually in this case creating a new class by extending one class has
avoided the duplication of the test case. It is very efficient test case !
---
[GitHub] carbondata pull request #2546: [CARBONDATA-2775] Adaptive encoding fails for...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2546#discussion_r205409053
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestAdaptiveEncodingUnsafeHeapColumnPageForComplexDataType.scala
---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.integration.spark.testsuite.complexType
+
+import java.io.File
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+/**
+ * Test class of Adaptive Encoding UnSafe Column Page with Complex Data
type
+ *
+ */
+
+class TestAdaptiveEncodingUnsafeHeapColumnPageForComplexDataType
+ extends QueryTest with BeforeAndAfterAll with TestAdaptiveComplexType {
+
+ override def beforeAll(): Unit = {
+
+new
File(CarbonProperties.getInstance().getSystemFolderLocation).delete()
+sql("DROP TABLE IF EXISTS adaptive")
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.ENABLE_UNSAFE_COLUMN_PAGE,
+"true")
+
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.ENABLE_OFFHEAP_SORT,
+"false")
+ }
+
--- End diff --
1. only beforeAll and afterAll implementation is therewhere is the test
case?
2. Avoid creating a new class, try to add the test case in one of existing
Encoding test case related classes
---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2441 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6006/ ---
[GitHub] carbondata issue #2561: [CARBONDATA-2784][CARBONDATA-2786][SDK writer] Fixed...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2561 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6005/ ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2441 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6004/ ---
[GitHub] carbondata issue #2562: [HOTFIX] CreateDataMapPost Event was skipped in case...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2562 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6003/ ---
[jira] [Commented] (CARBONDATA-2786) NPE when SDK writer tries to write a file
[ https://issues.apache.org/jira/browse/CARBONDATA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558118#comment-16558118 ] Ajantha Bhat commented on CARBONDATA-2786: -- #2387 , in [CarbonProperties.java|https://github.com/apache/carbondata/pull/2387/files#diff-4888f978087a7a1843a22fe016ea6532] After systemLocation = getStorePath(); Null validation missing for systemLocation. because this can be null in SDK case. As Store location is not applicable for SDK. > NPE when SDK writer tries to write a file > - > > Key: CARBONDATA-2786 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2786 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > > #2387 , in > [CarbonProperties.java|https://github.com/apache/carbondata/pull/2387/files#diff-4888f978087a7a1843a22fe016ea6532] > After systemLocation = getStorePath(); Null validation missing for > systemLocation. > because this can be null in SDK case. As Store location is not applicable for > SDK. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CARBONDATA-2587) Support Local dictionary in data loading
[ https://issues.apache.org/jira/browse/CARBONDATA-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar vishal reassigned CARBONDATA-2587: Assignee: kumar vishal > Support Local dictionary in data loading > > > Key: CARBONDATA-2587 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2587 > Project: CarbonData > Issue Type: Sub-task >Reporter: kumar vishal >Assignee: kumar vishal >Priority: Major > Fix For: 1.4.1 > > Time Spent: 11h 10m > Remaining Estimate: 0h > > Support local dictionary in data loading for low cardinality no dictionary > string data type column -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2584) CarbonData Local Dictionary Support
[
https://issues.apache.org/jira/browse/CARBONDATA-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kumar vishal resolved CARBONDATA-2584.
--
Resolution: Fixed
Assignee: kumar vishal
> CarbonData Local Dictionary Support
> ---
>
> Key: CARBONDATA-2584
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2584
> Project: CarbonData
> Issue Type: New Feature
>Reporter: kumar vishal
>Assignee: kumar vishal
>Priority: Major
> Attachments: CarbonData Local Dictionary Support Design Doc.docx
>
>
> Currently CarbonData supports global dictionary or No-Dictionary (Plain-Text
> stored in LV format) for storing dimension column data.
> *Bottleneck with Global Dictionary*
> It’s difficult for user to determine whether the column should be dictionary
> or not if number of columns in table is high.
> Global dictionary generation generally slows down the load process.
> Multiple IO operations are made during load even though dictionary already
> exists.
> During query, multiple IO operations done for reading dictionary files and
> carbondata files.
> *Bottleneck with No-Dictionary*
> Storage size is high as we store the data in LV format
> Query on No-Dictionary column is slower as data read/processed is more
> Filtering is slower on No-Dictionary columns as number of comparison is high
> Memory footprint is high
> *The above bottlenecks can be solved by generating dictionary for low
> cardinality columns at each blocklet level, which will help to achieve below
> benefits:*
> Reduces the extra IO operations read/write on the dictionary files generated
> in case of global dictionary.
> It will eliminate the problem for user to identify the dictionary columns
> when the number of columns are more in a table.
> It helps in getting more compression on dimension columns with less
> cardinality.
> Filter queries and full scan queries on No-dictionary columns with local
> dictionary will be faster as filter will be done on encoded data.
> It will help in reducing the store size and memory footprint as only unique
> values will be stored {color:#00}as {color}part of local dictionary and
> corresponding data will be stored as encoded data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2588) Support Local dictionary in data loading with complex type columns
[ https://issues.apache.org/jira/browse/CARBONDATA-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar vishal resolved CARBONDATA-2588. -- Resolution: Fixed Assignee: kumar vishal > Support Local dictionary in data loading with complex type columns > -- > > Key: CARBONDATA-2588 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2588 > Project: CarbonData > Issue Type: Sub-task >Reporter: kumar vishal >Assignee: kumar vishal >Priority: Major > > Generate local dictionary for complex type primitive columns(no dictionary > low cardinality column) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2778) Empty result in query after IUD delete operation
[
https://issues.apache.org/jira/browse/CARBONDATA-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manish Gupta resolved CARBONDATA-2778.
--
Resolution: Fixed
Fix Version/s: 1.4.1
> Empty result in query after IUD delete operation
>
>
> Key: CARBONDATA-2778
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2778
> Project: CarbonData
> Issue Type: Bug
>Reporter: Kunal Kapoor
>Assignee: Kunal Kapoor
>Priority: Minor
> Fix For: 1.4.1
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> # drop table if exists t1
> # create table t1 (c1 int,c2 string) STORED BY
> 'org.apache.carbondata.format' TBLPROPERTIES('table_blocksize'='1',
> 'dictionary_exclude'='c2')
> # LOAD DATA LOCAL INPATH 'test.csv' INTO table t1
> options('fileheader'='c1,c2')
> # run delete command which should delete a whole block
> # Run clean file operation.
> # select from t1.
>
> *NOTE*: Disable mergeindex property
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2778) Empty result in query after IUD delete operation
[
https://issues.apache.org/jira/browse/CARBONDATA-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manish Gupta updated CARBONDATA-2778:
-
Priority: Minor (was: Major)
> Empty result in query after IUD delete operation
>
>
> Key: CARBONDATA-2778
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2778
> Project: CarbonData
> Issue Type: Bug
>Reporter: Kunal Kapoor
>Assignee: Kunal Kapoor
>Priority: Minor
> Fix For: 1.4.1
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> # drop table if exists t1
> # create table t1 (c1 int,c2 string) STORED BY
> 'org.apache.carbondata.format' TBLPROPERTIES('table_blocksize'='1',
> 'dictionary_exclude'='c2')
> # LOAD DATA LOCAL INPATH 'test.csv' INTO table t1
> options('fileheader'='c1,c2')
> # run delete command which should delete a whole block
> # Run clean file operation.
> # select from t1.
>
> *NOTE*: Disable mergeindex property
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata pull request #2548: [CARBONDATA-2778]Fixed bug when select after ...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2548 ---
[GitHub] carbondata pull request #2441: [CARBONDATA-2625] optimize CarbonReader perfo...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2441#discussion_r205388461
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/executor/impl/AbstractQueryExecutor.java
---
@@ -647,6 +648,11 @@ private int getKeySize(List
queryDimension,
// must stop all the running execution otherwise it will keep
running and will hit
// the query performance.
queryProperties.executorService.shutdownNow();
+ try {
+queryProperties.executorService.awaitTermination(1,
TimeUnit.HOURS);
+ } catch (InterruptedException e) {
+e.printStackTrace();
--- End diff --
Remove this and use logger
---
[jira] [Assigned] (CARBONDATA-2786) NPE when SDK writer tries to write a file
[ https://issues.apache.org/jira/browse/CARBONDATA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat reassigned CARBONDATA-2786: Assignee: Ajantha Bhat > NPE when SDK writer tries to write a file > - > > Key: CARBONDATA-2786 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2786 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > > #2387 , in > [CarbonProperties.java|https://github.com/apache/carbondata/pull/2387/files#diff-4888f978087a7a1843a22fe016ea6532] > After systemLocation = getStorePath(); Null validation missing for > systemLocation. > because this can be null in SDK case. As Store location is not applicable for > SDK. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2786) NPE when SDK writer tries to write a file
Ajantha Bhat created CARBONDATA-2786: Summary: NPE when SDK writer tries to write a file Key: CARBONDATA-2786 URL: https://issues.apache.org/jira/browse/CARBONDATA-2786 Project: CarbonData Issue Type: Bug Reporter: Ajantha Bhat #2387 , in [CarbonProperties.java|https://github.com/apache/carbondata/pull/2387/files#diff-4888f978087a7a1843a22fe016ea6532] After systemLocation = getStorePath(); Null validation missing for systemLocation. because this can be null in SDK case. As Store location is not applicable for SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6278/ ---
[GitHub] carbondata issue #2548: [CARBONDATA-2778]Fixed bug when select after delete ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2548 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6002/ ---
[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7524/ ---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2517 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6001/ ---
[GitHub] carbondata issue #2548: [CARBONDATA-2778]Fixed bug when select after delete ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2548 retest sdv please ---
[GitHub] carbondata issue #2563: WIP test
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2563 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7525/ ---
[GitHub] carbondata issue #2546: [CARBONDATA-2775] Adaptive encoding fails for Unsafe...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2546 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6000/ ---
