[GitHub] carbondata issue #2690: [WIP] support long string columns with spark FileFor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2690 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/131/ ---
[GitHub] carbondata pull request #2694: [CARBONDATA-2876]AVRO datatype support throug...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2694 ---
[GitHub] carbondata issue #2690: [WIP] support long string columns with spark FileFor...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2690 @ravipesala : As per the discussion, sdk and spark file format uses varchar instead of string now . so, code changed again. please review again. ---
[GitHub] carbondata issue #2694: [CARBONDATA-2876]AVRO datatype support through SDK
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2694 LGTM ---
[GitHub] carbondata issue #2678: [CARBONDATA-2909] Multi user support for SDK on S3
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2678 @ravipesala Please review ---
[jira] [Resolved] (CARBONDATA-2915) Updates to CarbonData documentation and structure
[ https://issues.apache.org/jira/browse/CARBONDATA-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-2915. Resolution: Fixed Assignee: Raghunandan S Fix Version/s: 1.5.0 > Updates to CarbonData documentation and structure > - > > Key: CARBONDATA-2915 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2915 > Project: CarbonData > Issue Type: Improvement >Reporter: Raghunandan >Assignee: Raghunandan S >Priority: Minor > Fix For: 1.5.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > > 1.Split Our carbondata command into DDL and DML > 2.Add Presto integration along with Spark into quick start > 3.Add a master reference manual which lists all the commands supported in > carbondata.This manual shall have links to DDL and DML supported > 4.Add a introduction to carbondata covering architecture,design and features > supported > 5.Merge FAQ and troubleshooting documents into single document > 6.Add a separate md file to explain user how to navigate across our > documentation > 7.Add the TOC (Table of Contents) to all the md files which has multiple > sections > 8.Add list of supported properties at the beginning of each DDL or DML so > that user knows all the properties that are supported > 9.Rewrite the configuration properties description to explain the property in > bit more detail and also highlight when to use the command and any caveats > 10.ReOrder our configuration properties table to group features wise > 11.Update our webpage(carbondata.apache.org) to have a better navigation for > documentation section > 12.Change the grammar and sentences -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2592: [CARBONDATA-2915] Updated & enhanced Document...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2592 ---
[GitHub] carbondata issue #2592: [CARBONDATA-2915] Updated & enhanced Documentation o...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/2592 LGTM ---
[GitHub] carbondata issue #2693: [CARBONDATA-2915] Reformat Documentation of CarbonDa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2693 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8368/ ---
[GitHub] carbondata issue #2693: [CARBONDATA-2915] Reformat Documentation of CarbonDa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2693 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/298/ ---
[GitHub] carbondata issue #2693: [CARBONDATA-2915] Reformat Documentation of CarbonDa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2693 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/130/ ---
[GitHub] carbondata pull request #2654: [CARBONDATA-2896] Adaptive Encoding for Primi...
Github user kevinjmh commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2654#discussion_r215829032
--- Diff:
datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java
---
@@ -331,8 +332,18 @@ private BloomQueryModel
buildQueryModelInternal(CarbonColumn carbonColumn,
// for dictionary/date columns, convert the surrogate key to bytes
internalFilterValue = CarbonUtil.getValueAsBytes(DataTypes.INT,
convertedValue);
} else {
- // for non dictionary dimensions, is already bytes,
- internalFilterValue = (byte[]) convertedValue;
+ // for non dictionary dimensions, numeric columns will be of
original data,
+ // so convert the data to bytes
+ if (DataTypeUtil.isPrimitiveColumn(carbonColumn.getDataType())) {
+if (convertedValue == null) {
+ convertedValue =
DataConvertUtil.getNullValueForMeasure(carbonColumn.getDataType(),
+ carbonColumn.getColumnSchema().getScale());
+}
+internalFilterValue =
+CarbonUtil.getValueAsBytes(carbonColumn.getDataType(),
convertedValue);
--- End diff --
The above problem is similar to your second commit for minmax
---
[GitHub] carbondata pull request #2654: [CARBONDATA-2896] Adaptive Encoding for Primi...
Github user kevinjmh commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2654#discussion_r215827023
--- Diff:
datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java
---
@@ -331,8 +332,18 @@ private BloomQueryModel
buildQueryModelInternal(CarbonColumn carbonColumn,
// for dictionary/date columns, convert the surrogate key to bytes
internalFilterValue = CarbonUtil.getValueAsBytes(DataTypes.INT,
convertedValue);
} else {
- // for non dictionary dimensions, is already bytes,
- internalFilterValue = (byte[]) convertedValue;
+ // for non dictionary dimensions, numeric columns will be of
original data,
+ // so convert the data to bytes
+ if (DataTypeUtil.isPrimitiveColumn(carbonColumn.getDataType())) {
+if (convertedValue == null) {
+ convertedValue =
DataConvertUtil.getNullValueForMeasure(carbonColumn.getDataType(),
+ carbonColumn.getColumnSchema().getScale());
+}
+internalFilterValue =
+CarbonUtil.getValueAsBytes(carbonColumn.getDataType(),
convertedValue);
--- End diff --
Result of `getValueAsBytes` conflicts with existing bloom index data which
will affect query result.
For measure in 'NoDict', original implementation used
`NonDictionaryFieldConverterImpl` to convert value, but now it uses
`MeasureFieldConverterImpl` to convert value
---
[GitHub] carbondata issue #2683: [CARBONDATA-2916] Add CarbonCli tool for data summar...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2683 @jackylk Better print local_dictionary enabled or not in schema. And also if possible please print the local dictionary size of each column in column details. ---
[GitHub] carbondata issue #2683: [CARBONDATA-2916] Add CarbonCli tool for data summar...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2683 ``` ## Summary 1 blocks, 1 shards, 1 blocklets, 9 pages, 259,304 rows, 9.95MB ## Column Statistics (column 'L_DISCOUNT') Shard #1 (72636812283890_batchno0-0-null-1536219825841) BLK BLKLT Meta Size Data Size Card Min/Max range (total width is 80 characters) 00 1.06KB 9.75MB 2,147,483,647 ``` 1. In the above, my actual file size is 10.4 MB but it shows only 9.95 MB. 2. In column statistics of L_DISCOUNT it shows 9.95 MB, it means it does not show only that column size? 3. What is `Card` here?is it cardinality? It does not make senese to print Integer max I guess ---
[GitHub] carbondata issue #2683: [CARBONDATA-2916] Add CarbonCli tool for data summar...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2683 @jackylk Better create another folder under tools ---
[GitHub] carbondata issue #2698: [HOTFIX] Fixed LRU cache bug to invalidate the cache...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2698 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8367/ ---
[GitHub] carbondata issue #2698: [HOTFIX] Fixed LRU cache bug to invalidate the cache...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2698 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/297/ ---
[GitHub] carbondata issue #2698: [HOTFIX] Fixed LRU cache bug to invalidate the cache...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2698 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/129/ ---
[GitHub] carbondata pull request #2698: [HOTFIX] Fixed LRU cache bug to invalidate th...
GitHub user manishgupta88 opened a pull request: https://github.com/apache/carbondata/pull/2698 [HOTFIX] Fixed LRU cache bug to invalidate the cacheable object to clean up the resources This PR contains the fix for LRU cache bug to invalidate the Cacheable object while removing it from LRU cache. This will help in clearing the unsafe memory for cacheable objects like BlockDataMaps Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/manishgupta88/carbondata lru_cache_bug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2698.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2698 commit ba4221cc18700b1c22866dbd86055065aefdb1d4 Author: manishgupta88 Date: 2018-09-06T17:26:49Z Fixed LRU cache of bug to invalidate the Cacheable object while removing it from LRU cache. This will help in clearing the unsafe memory for cacheable objects like BlockDataMaps ---
[GitHub] carbondata issue #2692: [CARBONDATA-2879] [CARBONDATA-2918] support sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2692 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8366/ ---
[GitHub] carbondata issue #2692: [CARBONDATA-2879] [CARBONDATA-2918] support sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2692 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/296/ ---
[GitHub] carbondata issue #2692: [CARBONDATA-2879] [CARBONDATA-2918] support sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2692 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/128/ ---
[GitHub] carbondata issue #2692: [CARBONDATA-2879] [CARBONDATA-2918] support sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2692 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8365/ ---
[GitHub] carbondata issue #2692: [CARBONDATA-2879] [CARBONDATA-2918] support sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2692 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/295/ ---
[GitHub] carbondata pull request #2592: [CARBONDATA-2915] Updated & enhanced Document...
Github user sraghunandan commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2592#discussion_r215672435 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -470,15 +447,6 @@ */ @CarbonProperty public static final String CARBON_DATE_FORMAT = "carbon.date.format"; - /** - * STORE_LOCATION_HDFS - */ - @CarbonProperty - public static final String STORE_LOCATION_HDFS = "carbon.storelocation.hdfs"; --- End diff -- it is not being used ---
[GitHub] carbondata pull request #2683: [CARBONDATA-2916] Add CarbonCli tool for data...
Github user chenliang613 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2683#discussion_r215669915 --- Diff: pom.xml --- @@ -706,6 +706,12 @@ datamap/mv/core + + tool --- End diff -- suggest using "tools" ---
[GitHub] carbondata pull request #2592: [CARBONDATA-2915] Updated & enhanced Document...
Github user chenliang613 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2592#discussion_r215659429 --- Diff: docs/configuration-parameters.md --- @@ -16,152 +16,135 @@ --> # Configuring CarbonData - This tutorial guides you through the advanced configurations of CarbonData : - + This guide explains the configurations that can be used to tune CarbonData to achieve better performance.Some of the properties can be set dynamically and are explained in the section Dynamic Configuration In CarbonData Using SET-RESET.Most of the properties that control the internal settings have reasonable default values.They are listed along with the properties along with explanation. --- End diff -- suggest removing this sentence : Some of the properties can be set dynamically and are explained in the section Dynamic Configuration In CarbonData Using SET-RESET ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8364/ ---
[GitHub] carbondata pull request #2592: [CARBONDATA-2915] Updated & enhanced Document...
Github user chenliang613 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2592#discussion_r215655761 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -470,15 +447,6 @@ */ @CarbonProperty public static final String CARBON_DATE_FORMAT = "carbon.date.format"; - /** - * STORE_LOCATION_HDFS - */ - @CarbonProperty - public static final String STORE_LOCATION_HDFS = "carbon.storelocation.hdfs"; --- End diff -- can you please explain, why need to remove : "STORE_LOCATION_HDFS" ? ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/294/ ---
[GitHub] carbondata issue #2697: [WIP] support "carbon.load.directWriteHdfs.enabled" ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2697 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/293/ ---
[GitHub] carbondata issue #2697: [WIP] support "carbon.load.directWriteHdfs.enabled" ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2697 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8363/ ---
[GitHub] carbondata issue #2678: [CARBONDATA-2909] Multi user support for SDK on S3
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2678 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8362/ ---
[GitHub] carbondata issue #2687: [CARBONDATA-2876]Fix Avro decimal datatype with prec...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2687 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/290/ ---
[GitHub] carbondata issue #2692: [CARBONDATA-2879] [CARBONDATA-2918] support sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2692 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/127/ ---
[GitHub] carbondata issue #2678: [CARBONDATA-2909] Multi user support for SDK on S3
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2678 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/292/ ---
[GitHub] carbondata issue #2687: [CARBONDATA-2876]Fix Avro decimal datatype with prec...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2687 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8360/ ---
[GitHub] carbondata issue #2692: [CARBONDATA-2879] [CARBONDATA-2918] support sort sco...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2692 @ravipesala : All comments are handled. please check. ---
[GitHub] carbondata pull request #2692: [CARBONDATA-2879] [CARBONDATA-2918] support s...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2692#discussion_r215634003
--- Diff:
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java
---
@@ -311,15 +326,15 @@ public CarbonWriterBuilder
withTableProperties(Map options) {
}
for (Map.Entry entry : options.entrySet()) {
- if (entry.getKey().equalsIgnoreCase("equalsIgnoreCase")) {
+ if (entry.getKey().equalsIgnoreCase("blocksize")) {
this.withBlockSize(Integer.parseInt(entry.getValue()));
} else if (entry.getKey().equalsIgnoreCase("blockletsize")) {
this.withBlockletSize(Integer.parseInt(entry.getValue()));
} else if
(entry.getKey().equalsIgnoreCase("localDictionaryThreshold")) {
this.localDictionaryThreshold(Integer.parseInt(entry.getValue()));
} else if (entry.getKey().equalsIgnoreCase("enableLocalDictionary"))
{
this.enableLocalDictionary((entry.getValue().equalsIgnoreCase("true")));
- } else {
+ } else if (entry.getKey().equalsIgnoreCase("sortcolumns")) {
--- End diff --
yes. changed it to same as table properties syntax
---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/126/ ---
[GitHub] carbondata pull request #2692: [CARBONDATA-2879] [CARBONDATA-2918] support s...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2692#discussion_r215628732
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/model/LoadOption.java
---
@@ -183,7 +184,6 @@
}
optionsFinal.put("single_pass", String.valueOf(singlePass));
-optionsFinal.put("sort_scope", "local_sort");
--- End diff --
same fix as above. reverted here.
---
[GitHub] carbondata pull request #2692: [CARBONDATA-2879] [CARBONDATA-2918] support s...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2692#discussion_r215628606
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/model/LoadOption.java
---
@@ -59,7 +59,8 @@
optionsFinal.put("fileheader", Maps.getOrDefault(options,
"fileheader", ""));
optionsFinal.put("commentchar", Maps.getOrDefault(options,
"commentchar", "#"));
optionsFinal.put("columndict", Maps.getOrDefault(options,
"columndict", null));
-
+optionsFinal.put("sort_scope",
+Maps.getOrDefault(options, "sort_scope",
CarbonCommonConstants.LOAD_SORT_SCOPE_DEFAULT));
--- End diff --
yes, it should not be set here.
now moved to
CarbonLoadModelBuilder.build() , so it will set only for SDK and
spark-file-format
---
[GitHub] carbondata issue #2697: [WIP] support "carbon.load.directWriteHdfs.enabled" ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2697 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/125/ ---
[GitHub] carbondata issue #2697: [WIP] support "carbon.load.directWriteHdfs.enabled" ...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2697 @KanakaKumar : please check this. do we need to change property name ? what if existing user using it ? ---
[GitHub] carbondata pull request #2697: [WIP] support "carbon.load.directWriteHdfs.en...
GitHub user ajantha-bhat opened a pull request: https://github.com/apache/carbondata/pull/2697 [WIP] support "carbon.load.directWriteHdfs.enabled" for S3 **problem** : Currently for s3, when the above carbon property is set. index file will not be written in the s3 store path due to bug in folder path. Solution: file separator used is wrong. Need to fix it. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted?NA - [ ] Document update required?NA - [ ] Testing done updated UT - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajantha-bhat/carbondata issue_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2697.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2697 commit 0ecec948a63d155b73ba6b8dd2a2a69382be3454 Author: ajantha-bhat Date: 2018-09-06T11:17:22Z support carbon.load.directWriteHdfs.enabled for S3 ---
[GitHub] carbondata issue #2678: [CARBONDATA-2909] Multi user support for SDK on S3
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2678 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/124/ ---
[GitHub] carbondata issue #2687: [CARBONDATA-2876]Fix Avro decimal datatype with prec...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2687 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/122/ ---
[GitHub] carbondata issue #2678: [CARBONDATA-2909] Multi user support for SDK on S3
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2678 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/123/ ---
[GitHub] carbondata issue #2678: [CARBONDATA-2909] Multi user support for SDK on S3
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2678 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/291/ ---
[GitHub] carbondata issue #2678: [CARBONDATA-2909] Multi user support for SDK on S3
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2678 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8361/ ---
[GitHub] carbondata pull request #2692: [CARBONDATA-2879] [CARBONDATA-2918] support s...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2692#discussion_r215612715
--- Diff:
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java
---
@@ -311,15 +326,15 @@ public CarbonWriterBuilder
withTableProperties(Map options) {
}
for (Map.Entry entry : options.entrySet()) {
- if (entry.getKey().equalsIgnoreCase("equalsIgnoreCase")) {
+ if (entry.getKey().equalsIgnoreCase("blocksize")) {
this.withBlockSize(Integer.parseInt(entry.getValue()));
} else if (entry.getKey().equalsIgnoreCase("blockletsize")) {
this.withBlockletSize(Integer.parseInt(entry.getValue()));
} else if
(entry.getKey().equalsIgnoreCase("localDictionaryThreshold")) {
this.localDictionaryThreshold(Integer.parseInt(entry.getValue()));
} else if (entry.getKey().equalsIgnoreCase("enableLocalDictionary"))
{
this.enableLocalDictionary((entry.getValue().equalsIgnoreCase("true")));
- } else {
+ } else if (entry.getKey().equalsIgnoreCase("sortcolumns")) {
--- End diff --
Please make sure all property names are the same as create table properties
---
[GitHub] carbondata issue #2687: [CARBONDATA-2876]Fix Avro decimal datatype with prec...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2687 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/121/ ---
[GitHub] carbondata pull request #2692: [CARBONDATA-2879] [CARBONDATA-2918] support s...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2692#discussion_r215609931
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/model/LoadOption.java
---
@@ -183,7 +184,6 @@
}
optionsFinal.put("single_pass", String.valueOf(singlePass));
-optionsFinal.put("sort_scope", "local_sort");
--- End diff --
Check for SDK flow
---
[GitHub] carbondata pull request #2692: [CARBONDATA-2879] [CARBONDATA-2918] support s...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2692#discussion_r215609784
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/model/LoadOption.java
---
@@ -59,7 +59,8 @@
optionsFinal.put("fileheader", Maps.getOrDefault(options,
"fileheader", ""));
optionsFinal.put("commentchar", Maps.getOrDefault(options,
"commentchar", "#"));
optionsFinal.put("columndict", Maps.getOrDefault(options,
"columndict", null));
-
+optionsFinal.put("sort_scope",
+Maps.getOrDefault(options, "sort_scope",
CarbonCommonConstants.LOAD_SORT_SCOPE_DEFAULT));
--- End diff --
Please check in the CrabonSession Load scenario we are restricting it or
not. It is not supposed to use per load . It should be only for SDK
---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8358/ ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/288/ ---
[GitHub] carbondata issue #2696: [CARBONDATA-2902][DataMap] Fix showing negative prun...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2696 LGTM ---
[GitHub] carbondata issue #2694: [CARBONDATA-2876]AVRO datatype support through SDK
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2694 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/286/ ---
[GitHub] carbondata issue #2694: [CARBONDATA-2876]AVRO datatype support through SDK
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2694 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8356/ ---
[GitHub] carbondata issue #2687: [CARBONDATA-2876]Fix Avro decimal datatype with prec...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2687 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/287/ ---
[GitHub] carbondata issue #2687: [CARBONDATA-2876]Fix Avro decimal datatype with prec...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2687 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8357/ ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/120/ ---
[GitHub] carbondata issue #2687: [CARBONDATA-2876]Fix Avro decimal datatype with prec...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2687 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/119/ ---
[GitHub] carbondata issue #2694: [CARBONDATA-2876]AVRO datatype support through SDK
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2694 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/118/ ---
[GitHub] carbondata issue #2694: [CARBONDATA-2876]AVRO datatype support through SDK
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2694 Retest this please ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/117/ ---
[GitHub] carbondata issue #2696: [CARBONDATA-2902][DataMap] Fix showing negative prun...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2696 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/284/ ---
[GitHub] carbondata issue #2696: [CARBONDATA-2902][DataMap] Fix showing negative prun...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2696 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8354/ ---
[GitHub] carbondata issue #2662: [CARBONDATA-2889]Add decoder based fallback mechanis...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2662 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8352/ ---
[GitHub] carbondata issue #2662: [CARBONDATA-2889]Add decoder based fallback mechanis...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2662 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/282/ ---
[GitHub] carbondata issue #2696: [CARBONDATA-2902][DataMap] Fix showing negative prun...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2696 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/116/ ---
[GitHub] carbondata pull request #2696: [CARBONDATA-2902][DataMap] Fix showing negati...
GitHub user kevinjmh opened a pull request: https://github.com/apache/carbondata/pull/2696 [CARBONDATA-2902][DataMap] Fix showing negative pruning result for explain command #2676 used method `ByteBuffer.getShort(int index)` to get number of blocklets in block, but it used wrong parameter. The `index` is index of byte instead of index of short. So it needs to multiply bytes of short type Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kevinjmh/carbondata explain_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2696.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2696 commit 1cdfc5c8e592f53170cf942bfbb45cc5e5d67719 Author: Manhua Date: 2018-09-06T09:09:21Z fix negative blocklet skipped when cache level is block ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/115/ ---
[GitHub] carbondata issue #2662: [CARBONDATA-2889]Add decoder based fallback mechanis...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2662 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/114/ ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/280/ ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8350/ ---
[GitHub] carbondata issue #2678: [CARBONDATA-2909] Multi user support for SDK on S3
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2678 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/281/ ---
[GitHub] carbondata issue #2678: [CARBONDATA-2909] Multi user support for SDK on S3
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2678 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8351/ ---
[GitHub] carbondata issue #2691: [CARBONDATA-2912] Support CSV table load csv data wi...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2691 @jackylk Please review it. master has updated, but branch-1.3 didn't fix it. ---
[GitHub] carbondata issue #2678: [CARBONDATA-2909] Multi user support for SDK on S3
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2678 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/113/ ---
[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2695 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8349/ ---
[GitHub] carbondata issue #2683: [CARBONDATA-2916] Add CarbonCli tool for data summar...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2683 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8348/ ---
[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2695 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/279/ ---
[GitHub] carbondata issue #2683: [CARBONDATA-2916] Add CarbonCli tool for data summar...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2683 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/278/ ---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2628 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/277/ ---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2628 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8347/ ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/112/ ---
[GitHub] carbondata issue #2694: [CARBONDATA-2876]AVRO datatype support through SDK
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2694 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/276/ ---
[GitHub] carbondata issue #2695: [CARBONDATA-2919] Support ingest from Kafka in Strea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2695 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/111/ ---
[GitHub] carbondata issue #2694: [CARBONDATA-2876]AVRO datatype support through SDK
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2694 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8346/ ---
[GitHub] carbondata issue #2683: [CARBONDATA-2916] Add CarbonCli tool for data summar...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2683 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/110/ ---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2628 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/109/ ---
[GitHub] carbondata pull request #2695: [CARBONDATA-2919] Support ingest from Kafka i...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2695#discussion_r215507635
--- Diff: examples/spark2/pom.xml ---
@@ -53,6 +53,11 @@
org.apache.spark
spark-sql_${scala.binary.version}
+
+ org.apache.spark
+ spark-sql-kafka-0-10_${scala.binary.version}
+ 2.2.1
--- End diff --
ok, fixed
---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user QiangCai commented on the issue: https://github.com/apache/carbondata/pull/2628 retest this please ---
