[GitHub] carbondata issue #2899: [CARBONDATA-3073] Support configure TableProperties ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2899 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1305/ ---
[GitHub] carbondata issue #2875: [CARBONDATA-3038] Refactor dynamic configuration
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2875 retest this please ---
[GitHub] carbondata issue #2902: [CARBONDATA-3083] Fixed data mismatch issue after up...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2902 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1516/ ---
[GitHub] carbondata issue #2899: [CARBONDATA-3073] Support configure TableProperties ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2899 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1517/ ---
[GitHub] carbondata issue #2900: [CARBONDATA-3078] Disable explain collector for coun...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2900 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1304/ ---
[GitHub] carbondata issue #2902: [CARBONDATA-3083] Fixed data mismatch issue after up...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2902 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1303/ ---
[GitHub] carbondata issue #2898: [CARBONDATA-3077] Fixed query failure in fileformat ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2898 @manishgupta88 it solves part of the problem (schema mismatch issue). But when you call getDataMaps it will give stale datamaps to you right. How those can be updated? ---
[GitHub] carbondata issue #2895: [HOTFIX] Fix NPE in spark, when same vector reads fi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2895 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9563/ ---
[GitHub] carbondata pull request #2903: [CARBONDATA-3084]dataload failure fix when fl...
GitHub user akashrn5 opened a pull request: https://github.com/apache/carbondata/pull/2903 [CARBONDATA-3084]dataload failure fix when float value exceeds the limit ### Problem: when the float value exceeds the range and we try to insert that data, data load fails. ### Analysis: when the value exceeds the range, the max is set as `Infinity`, so the decimal count of that value will be 0, so when decimal count is zero we go for CodecByAlgorithmForIntegral, so it fails ### Solution: when the value exceeds, and decimal count is zero , source datatype is float, then select DirectCompressCodec ### How this tested: test cases are added to validate the load and data Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/akashrn5/incubator-carbondata float Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2903.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2903 commit 798592441f77300dd89b1dec4dd51c15bc6b2c07 Author: akashrn5 Date: 2018-11-06T07:06:24Z dataload failure fix when float value exceeds the limit ---
[GitHub] carbondata issue #2898: [CARBONDATA-3077] Fixed query failure in fileformat ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2898 retest this please ---
[GitHub] carbondata pull request #2902: [CARBONDATA-3083] Fixed data mismatch issue a...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2902#discussion_r231012403
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
---
@@ -772,12 +772,33 @@ class UpdateCarbonTableTestCase extends QueryTest
with BeforeAndAfterAll {
sql("""drop table if exists iud.dest33_part""")
}
+ test("check data after update with row.filter pushdown as false") {
+CarbonProperties.getInstance().addProperty(CarbonCommonConstants
+ .CARBON_PUSH_ROW_FILTERS_FOR_VECTOR, "false")
+sql("""drop table if exists iud.dest33_flat""")
+sql(
+ """create table iud.dest33_part (c1 int,c2 string, c3 short) STORED
BY 'carbondata'"""
+.stripMargin)
+sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/IUD/negativevalue.csv'
INTO table iud
+ |.dest33_part options('header'='false')""".stripMargin)
+sql(
+ """update iud.dest33_part d set (c1) = (5) where d.c1 =
0""".stripMargin).show()
+checkAnswer(sql("select c3 from iud.dest33_part"), Seq(Row(-300),
Row(0), Row(-200), Row(700)
+ , Row(100), Row(-100), Row(null)))
+sql("""drop table if exists iud.dest33_part""")
+CarbonProperties.getInstance().addProperty(CarbonCommonConstants
+ .CARBON_PUSH_ROW_FILTERS_FOR_VECTOR, "true")
+ }
+
override def afterAll {
sql("use default")
sql("drop database if exists iud cascade")
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.isHorizontalCompactionEnabled ,
"true")
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.ENABLE_VECTOR_READER , "true")
+CarbonProperties.getInstance().addProperty(CarbonCommonConstants
+ .CARBON_PUSH_ROW_FILTERS_FOR_VECTOR, "false")
--- End diff --
instead of hard coding `"false"` use default value from constants
---
[GitHub] carbondata pull request #2902: [CARBONDATA-3083] Fixed data mismatch issue a...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2902#discussion_r231016130
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/compress/DirectCompressCodec.java
---
@@ -257,7 +265,13 @@ private void fillVector(ColumnPage columnPage,
CarbonColumnVector vector,
} else if (pageDataType == DataTypes.SHORT) {
short[] shortData = columnPage.getShortPage();
if (vectorDataType == DataTypes.SHORT) {
- vector.putShorts(0, pageSize, shortData, 0);
+ if (isUnderlyingVectorPresent) {
+for (int i = 0; i < pageSize; i++) {
+ vector.putShort(i, shortData[i]);
+}
+ } else {
+vector.putShorts(0, pageSize, shortData, 0);
--- End diff --
I think using `putShorts/putFloats` is common and unavoidable. In future
also any new encoding class can make use of these method and then again the
same problem can occur. Is it feasible to modify the vector classes
implementation methods itself just like an example below
`public void putShorts(int rowId, int count, short[] src, int srcIndex) {
for (int i = srcIndex; i < count; i++) {
putShort(rowId++, src[i]);
}
}`
This way it will be better
---
[GitHub] carbondata pull request #2902: [CARBONDATA-3083] Fixed data mismatch issue a...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2902#discussion_r231014429
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
---
@@ -772,12 +772,33 @@ class UpdateCarbonTableTestCase extends QueryTest
with BeforeAndAfterAll {
sql("""drop table if exists iud.dest33_part""")
}
+ test("check data after update with row.filter pushdown as false") {
+CarbonProperties.getInstance().addProperty(CarbonCommonConstants
+ .CARBON_PUSH_ROW_FILTERS_FOR_VECTOR, "false")
+sql("""drop table if exists iud.dest33_flat""")
+sql(
+ """create table iud.dest33_part (c1 int,c2 string, c3 short) STORED
BY 'carbondata'"""
+.stripMargin)
+sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/IUD/negativevalue.csv'
INTO table iud
+ |.dest33_part options('header'='false')""".stripMargin)
+sql(
+ """update iud.dest33_part d set (c1) = (5) where d.c1 =
0""".stripMargin).show()
+checkAnswer(sql("select c3 from iud.dest33_part"), Seq(Row(-300),
Row(0), Row(-200), Row(700)
+ , Row(100), Row(-100), Row(null)))
+sql("""drop table if exists iud.dest33_part""")
+CarbonProperties.getInstance().addProperty(CarbonCommonConstants
+ .CARBON_PUSH_ROW_FILTERS_FOR_VECTOR, "true")
--- End diff --
After test case completion we should set the default value for
`CARBON_PUSH_ROW_FILTERS_FOR_VECTOR`?...default property is false so I think at
the start of test case no need to modify the property value
---
[GitHub] carbondata issue #2902: [CARBONDATA-3083] Fixed data mismatch issue after up...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2902 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9559/ ---
[GitHub] carbondata issue #2895: [HOTFIX] Fix NPE in spark, when same vector reads fi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2895 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1515/ ---
[GitHub] carbondata issue #2898: [CARBONDATA-3077] Fixed query failure in fileformat ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2898 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9561/ ---
[GitHub] carbondata pull request #2900: [CARBONDATA-3078] Disable explain collector f...
Github user kevinjmh commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2900#discussion_r231014871
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
---
@@ -575,6 +576,8 @@ private BitSet setMatchedPartitions(String
partitionIds, Expression filter,
*/
public BlockMappingVO getBlockRowCount(Job job, CarbonTable table,
List partitions) throws IOException {
+// no useful information for count star query without filter, so
disable explain collector
+ExplainCollector.remove();
--- End diff --
OK
---
[GitHub] carbondata issue #2901: [CARBONDATA-3081] Fixed NPE for boolean type column ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2901 LGTM...please check the CI failure ---
[GitHub] carbondata issue #2902: [CARBONDATA-3083] Fixed data mismatch issue after up...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2902 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1512/ ---
[GitHub] carbondata issue #2901: [CARBONDATA-3081] Fixed NPE for boolean type column ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2901 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9560/ ---
[jira] [Created] (CARBONDATA-3084) data load with float datatype falis with internal error
Akash R Nilugal created CARBONDATA-3084: --- Summary: data load with float datatype falis with internal error Key: CARBONDATA-3084 URL: https://issues.apache.org/jira/browse/CARBONDATA-3084 Project: CarbonData Issue Type: Bug Reporter: Akash R Nilugal when data load is triggered for float datatype and data is exceeding the float max range, data load fails with following error java.lang.RuntimeException: internal error: FLOAT at org.apache.carbondata.core.datastore.page.encoding.DefaultEncodingFactory.fitMinMax(DefaultEncodingFactory.java:179) at org.apache.carbondata.core.datastore.page.encoding.DefaultEncodingFactory.selectCodecByAlgorithmForIntegral(DefaultEncodingFactory.java:259) at org.apache.carbondata.core.datastore.page.encoding.DefaultEncodingFactory.selectCodecByAlgorithmForFloating(DefaultEncodingFactory.java:337) at org.apache.carbondata.core.datastore.page.encoding.DefaultEncodingFactory.createEncoderForMeasureOrNoDictionaryPrimitive(DefaultEncodingFactory.java:130) at org.apache.carbondata.core.datastore.page.encoding.DefaultEncodingFactory.createEncoder(DefaultEncodingFactory.java:66) at org.apache.carbondata.processing.store.TablePage.encodeAndCompressMeasures(TablePage.java:385) at org.apache.carbondata.processing.store.TablePage.encode(TablePage.java:372) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.processDataRows(CarbonFactDataHandlerColumnar.java:285) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.access$500(CarbonFactDataHandlerColumnar.java:59) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar$Producer.call(CarbonFactDataHandlerColumnar.java:583) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar$Producer.call(CarbonFactDataHandlerColumnar.java:560) Steps to reproduce are create table datatype_floa_byte(f float, b byte) using carbon; insert into datatype_floa_byte select 123.123,127; insert into datatype_floa_byte select "1.7976931348623157E308",-127; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2900: [CARBONDATA-3078] Disable explain collector f...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2900#discussion_r231012866
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
---
@@ -575,6 +576,8 @@ private BitSet setMatchedPartitions(String
partitionIds, Expression filter,
*/
public BlockMappingVO getBlockRowCount(Job job, CarbonTable table,
List partitions) throws IOException {
+// no useful information for count star query without filter, so
disable explain collector
+ExplainCollector.remove();
--- End diff --
oh... I understand.
The current implementation of pruning collector may has bugs. Based on the
current implementation, your modification is OK...
---
[GitHub] carbondata pull request #2900: [CARBONDATA-3078] Disable explain collector f...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2900#discussion_r231012917
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
---
@@ -575,6 +576,8 @@ private BitSet setMatchedPartitions(String
partitionIds, Expression filter,
*/
public BlockMappingVO getBlockRowCount(Job job, CarbonTable table,
List partitions) throws IOException {
+// no useful information for count star query without filter, so
disable explain collector
+ExplainCollector.remove();
--- End diff --
please add comments for your modification in the code for better
understanding
---
[GitHub] carbondata issue #2804: [CARBONDATA-2996] CarbonSchemaReader support read sc...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2804 @KanakaKumar @kunal642 CI pass, please check it. ---
[GitHub] carbondata issue #2890: [CARBONDATA-3002] Fix some spell error
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2890 @chenliang613 @jackylk How to handle this PR? ---
[GitHub] carbondata issue #2805: [Documentation] Local dictionary Data which are not ...
Github user sgururajshetty commented on the issue: https://github.com/apache/carbondata/pull/2805 @sraghunandan kindly review ---
[GitHub] carbondata issue #2805: [Documentation] Local dictionary Data which are not ...
Github user sgururajshetty commented on the issue: https://github.com/apache/carbondata/pull/2805 @sraghunandan kindly review and help me to merge my changes ---
[GitHub] carbondata issue #2901: [CARBONDATA-3081] Fixed NPE for boolean type column ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2901 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1513/ ---
[GitHub] carbondata pull request #2902: [CARBONDATA-3083] Fixed data mismatch issue a...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2902#discussion_r231009938
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -1734,7 +1734,7 @@ private CarbonCommonConstants() {
public static final String CARBON_PUSH_ROW_FILTERS_FOR_VECTOR =
"carbon.push.rowfilters.for.vector";
- public static final String CARBON_PUSH_ROW_FILTERS_FOR_VECTOR_DEFAULT =
"false";
+ public static final String CARBON_PUSH_ROW_FILTERS_FOR_VECTOR_DEFAULT =
"true";
--- End diff --
Any specific reason for changing the default value?
---
[GitHub] carbondata pull request #2900: [CARBONDATA-3078] Disable explain collector f...
Github user kevinjmh commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2900#discussion_r231010531
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
---
@@ -575,6 +576,8 @@ private BitSet setMatchedPartitions(String
partitionIds, Expression filter,
*/
public BlockMappingVO getBlockRowCount(Job job, CarbonTable table,
List partitions) throws IOException {
+// no useful information for count star query without filter, so
disable explain collector
+ExplainCollector.remove();
--- End diff --
No remove. Its implementation is disable.
---
[GitHub] carbondata issue #2899: [CARBONDATA-3073] Support configure TableProperties ...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2899 retest this please ---
[GitHub] carbondata pull request #2900: [CARBONDATA-3078] Disable explain collector f...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2900#discussion_r231010174
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
---
@@ -575,6 +576,8 @@ private BitSet setMatchedPartitions(String
partitionIds, Expression filter,
*/
public BlockMappingVO getBlockRowCount(Job job, CarbonTable table,
List partitions) throws IOException {
+// no useful information for count star query without filter, so
disable explain collector
+ExplainCollector.remove();
--- End diff --
That's the weird part -- We are trying to remove the pruning collector even
the pruning info is not initialized.
I think you can add a flag for the collector to identify whether it is
initialized. And this flag is used where carbon what to record the info. If you
are planing to work like this, please add a comment for the scenario of this
variable.
---
[GitHub] carbondata issue #2875: [CARBONDATA-3038] Refactor dynamic configuration
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2875 retest thi please ---
[GitHub] carbondata issue #2895: [HOTFIX] Fix NPE in spark, when same vector reads fi...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2895 LGTM..can be merged once build is passed ---
[GitHub] carbondata issue #2895: [HOTFIX] Fix NPE in spark, when same vector reads fi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2895 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1302/ ---
[GitHub] carbondata issue #2895: [HOTFIX] Fix NPE in spark, when same vector reads fi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2895 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9562/ ---
[GitHub] carbondata issue #2898: [CARBONDATA-3077] Fixed query failure in fileformat ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2898 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1300/ ---
[GitHub] carbondata issue #2902: [CARBONDATA-3083] Fixed data mismatch issue after up...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2902 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1298/ ---
[GitHub] carbondata issue #2901: [CARBONDATA-3081] Fixed NPE for boolean type column ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2901 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1299/ ---
[GitHub] carbondata issue #2898: [CARBONDATA-3077] Fixed query failure in fileformat ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2898 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1511/ ---
[GitHub] carbondata issue #2895: [HOTFIX] Fix NPE in spark, when same vector reads fi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2895 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1301/ ---
[jira] [Updated] (CARBONDATA-3075) Select Filter fails for Legacy store if DirectVectorFill is enabled
[ https://issues.apache.org/jira/browse/CARBONDATA-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi Muthumurugesh updated CARBONDATA-3075: - Description: Please find below steps to reproduce the issue: # Create table and load data in legacy store # In new store, with Direct Vector filling Enabled, execute filter query and find below Exception | |*This operation is not supported in this reader org.apache.carbondata.core.datastore.chunk.reader.dimension.v2.CompressedDimensionChunkFileBasedReaderV2*| | > Select Filter fails for Legacy store if DirectVectorFill is enabled > --- > > Key: CARBONDATA-3075 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3075 > Project: CarbonData > Issue Type: Improvement >Reporter: Indhumathi Muthumurugesh >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > Please find below steps to reproduce the issue: > # Create table and load data in legacy store > # In new store, with Direct Vector filling Enabled, execute filter query and > find below Exception > | > |*This operation is not supported in this reader > org.apache.carbondata.core.datastore.chunk.reader.dimension.v2.CompressedDimensionChunkFileBasedReaderV2*| > | -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-3083) Null values are getting replaced by 0 after update operation.
Kunal Kapoor created CARBONDATA-3083:
Summary: Null values are getting replaced by 0 after update
operation.
Key: CARBONDATA-3083
URL: https://issues.apache.org/jira/browse/CARBONDATA-3083
Project: CarbonData
Issue Type: Bug
Affects Versions: 1.5.1
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor
Fix For: 1.5.1
create table negativeTable(intCol int, stringCol string, shortCol short) stored
by 'carbondata'
load data inpath 'hdfs://hacluster/user/dataWithNegativeValues.csv' into table
negativeTable
options('delimiter'=',','fileheader'='intCol,stringCol,shortCol','bad_records_action'='force')
select * from negativeTable
insert into negativeTable select 0,null,-10
insert into negativeTable select null,'inserted',20
select * from negativeTable
update negativeTable set (intCol) = (5) where intCol=0
select * from negativeTable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #2895: [HOTFIX] Fix NPE in spark, when same vector reads fi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2895 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1514/ ---
[GitHub] carbondata pull request #2902: [WIP] Fixed data mismatch issue after update
GitHub user kunal642 opened a pull request: https://github.com/apache/carbondata/pull/2902 [WIP] Fixed data mismatch issue after update Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kunal642/carbondata update_data_mismatch_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2902.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2902 commit bbd3dc87ac84c1d4005379dd445dec30f31f24aa Author: kunal642 Date: 2018-11-06T05:21:00Z fixed data mismatch issue after update ---
[GitHub] carbondata pull request #2895: [HOTFIX] Fix NPE in spark, when same vector r...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2895#discussion_r231001948
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/LocalDictDimensionDataChunkStore.java
---
@@ -61,10 +61,7 @@ public void fillVector(int[] invertedIndex, int[]
invertedIndexReverse, byte[] d
int columnValueSize = dimensionDataChunkStore.getColumnValueSize();
int rowsNum = data.length / columnValueSize;
CarbonColumnVector vector = vectorInfo.vector;
-if (!dictionary.isDictionaryUsed()) {
- vector.setDictionary(dictionary);
- dictionary.setDictionaryUsed();
-}
+vector.setDictionary(dictionary);
--- End diff --
done
---
[GitHub] carbondata pull request #2901: [CARBONDATA-3081] Fixed NPE for boolean type ...
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2901#discussion_r231001835
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/util/CarbonVectorizedRecordReader.java
---
@@ -171,13 +171,20 @@ public Object getCurrentValue() throws IOException,
InterruptedException {
rowCount += 1;
Object[] row = new Object[carbonColumnarBatch.columnVectors.length];
for (int i = 0; i < carbonColumnarBatch.columnVectors.length; i ++) {
+ Object data = carbonColumnarBatch.columnVectors[i].getData(batchIdx
- 1);
if (carbonColumnarBatch.columnVectors[i].getType() ==
DataTypes.STRING
|| carbonColumnarBatch.columnVectors[i].getType() ==
DataTypes.VARCHAR) {
-byte[] data = (byte[])
carbonColumnarBatch.columnVectors[i].getData(batchIdx - 1);
-row[i] = ByteUtil.toString(data, 0, data.length);
+if (data == null) {
+ row[i] = null;
+} else {
+ row[i] = ByteUtil.toString((byte[]) data, 0, (((byte[])
data).length));
+}
} else if (carbonColumnarBatch.columnVectors[i].getType() ==
DataTypes.BOOLEAN) {
-byte data = (byte)
carbonColumnarBatch.columnVectors[i].getData(batchIdx - 1);
-row[i] = ByteUtil.toBoolean(data);
+if (data == null) {
+ row[i] = null;
+} else {
+ row[i] = ByteUtil.toBoolean((byte) data);
+}
--- End diff --
getData is already has a check for null values. Because here explicit
conversion is required therefore null check had to be added.
---
[GitHub] carbondata pull request #2901: [CARBONDATA-3081] Fixed NPE for boolean type ...
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2901#discussion_r231001760
--- Diff:
store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
@@ -1844,4 +1844,53 @@ public void testVectorReader() {
}
}
+ @Test
+ public void testReadingNullValues() {
+String path = "./testWriteFiles";
+try {
+ FileUtils.deleteDirectory(new File(path));
+
+ Field[] fields = new Field[2];
+ fields[0] = new Field("stringField", DataTypes.STRING);
+ fields[1] = new Field("shortField", DataTypes.BOOLEAN);
--- End diff --
done
---
[GitHub] carbondata issue #2804: [CARBONDATA-2996] CarbonSchemaReader support read sc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2804 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9558/ ---
[GitHub] carbondata pull request #2895: [HOTFIX] Fix NPE in spark, when same vector r...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2895#discussion_r230999455
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/LocalDictDimensionDataChunkStore.java
---
@@ -61,10 +61,7 @@ public void fillVector(int[] invertedIndex, int[]
invertedIndexReverse, byte[] d
int columnValueSize = dimensionDataChunkStore.getColumnValueSize();
int rowsNum = data.length / columnValueSize;
CarbonColumnVector vector = vectorInfo.vector;
-if (!dictionary.isDictionaryUsed()) {
- vector.setDictionary(dictionary);
- dictionary.setDictionaryUsed();
-}
+vector.setDictionary(dictionary);
--- End diff --
Both the method are called in carbon flow for vector filling. One is direct
fill case and the other one is old vector fill flow. Please cross check once
---
[GitHub] carbondata issue #2875: [CARBONDATA-3038] Refactor dynamic configuration
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2875 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9557/ ---
[jira] [Updated] (CARBONDATA-3082) In open source documentation load_min_size_inmb parameter mentioned incorrectly as configuration instead of create table-load data DDL parameter
[ https://issues.apache.org/jira/browse/CARBONDATA-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Bhat updated CARBONDATA-3082: Summary: In open source documentation load_min_size_inmb parameter mentioned incorrectly as configuration instead of create table-load data DDL parameter (was: load_min_size_inmb parameter mentioned incorrectly as configuration instead of create table-load data DDL parameter) > In open source documentation load_min_size_inmb parameter mentioned > incorrectly as configuration instead of create table-load data DDL parameter > > > Key: CARBONDATA-3082 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3082 > Project: CarbonData > Issue Type: Bug > Components: docs >Affects Versions: 1.5.1 > Environment: Spark 2.1,2.2,2.3 >Reporter: Chetan Bhat >Assignee: Gururaj Shetty >Priority: Minor > > In the [https://carbondata.apache.org/configuration-parameters.html] link the > load_min_size_inmb parameter is mentioned incorrectly. Currently the > parameter is not supported as a configuration and hence the > load_min_size_inmb parameter information can be removed from the link. > Instead it needs to be updated in CREATE TABLE and LOAD DATA links as its > supported in create table table properties and load data options parameter > now. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2875: [CARBONDATA-3038] Refactor dynamic configuration
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2875 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1510/ ---
[GitHub] carbondata issue #2899: [CARBONDATA-3073] Support configure TableProperties ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2899 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9556/ ---
[GitHub] carbondata pull request #2895: [HOTFIX] Fix NPE in spark, when same vector r...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2895#discussion_r230997947
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/LocalDictDimensionDataChunkStore.java
---
@@ -61,10 +61,7 @@ public void fillVector(int[] invertedIndex, int[]
invertedIndexReverse, byte[] d
int columnValueSize = dimensionDataChunkStore.getColumnValueSize();
int rowsNum = data.length / columnValueSize;
CarbonColumnVector vector = vectorInfo.vector;
-if (!dictionary.isDictionaryUsed()) {
- vector.setDictionary(dictionary);
- dictionary.setDictionaryUsed();
-}
+vector.setDictionary(dictionary);
--- End diff --
I have checked this while coding, fill row is our method. So, no issues.
Only this vector is spark vector and dictionary needs to clear for it.
---
[GitHub] carbondata issue #2804: [CARBONDATA-2996] CarbonSchemaReader support read sc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2804 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1509/ ---
[GitHub] carbondata issue #2901: [CARBONDATA-3081] Fixed NPE for boolean type column ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2901 Add PR description ---
[GitHub] carbondata issue #2898: [CARBONDATA-3077] Fixed query failure in fileformat ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2898 retest this please ---
[GitHub] carbondata issue #2898: [CARBONDATA-3077] Fixed query failure in fileformat ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2898 @xuchuanyin ...yes this scenario will work fine. In case of dropping normal table it will go through CarbonSession flow and drop table command is already taking care of clearing the datamaps. In case of fileFormat table drop, if the clear dataMap API is not integrated by customer in that case the changes done in this PR will take care of referring only to latest carbon table ---
[GitHub] carbondata issue #2804: [CARBONDATA-2996] CarbonSchemaReader support read sc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2804 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1297/ ---
[GitHub] carbondata issue #2875: [CARBONDATA-3038] Refactor dynamic configuration
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2875 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1296/ ---
[GitHub] carbondata issue #2899: [CARBONDATA-3073] Support configure TableProperties ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2899 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1295/ ---
[GitHub] carbondata issue #2899: [CARBONDATA-3073] Support configure TableProperties ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2899 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1508/ ---
[GitHub] carbondata issue #2875: [CARBONDATA-3038] Refactor dynamic configuration
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2875 test it by âInserting table: support boolean and other data type, big fileâ test case in org.apache.carbondata.spark.testsuite.booleantype.BooleanDataTypesBigFileTest. when run "insert into boolean_table2 select * from boolean_table", it will match org.apache.spark.sql.execution.strategy.CarbonLateDecodeStrategy#pruneFilterProjectRaw ---
[GitHub] carbondata issue #2900: [CARBONDATA-3078] Disable explain collector for coun...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2900 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9555/ ---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230991320
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -1744,89 +2021,101 @@ private CarbonCommonConstants() {
* BYTE_ENCODING
*/
public static final String BYTE_ENCODING = "ISO-8859-1";
+
--- End diff --
support configure carbon.push.rowfilters.for.vector in set command, dynamic
configurable in this PR
---
[GitHub] carbondata issue #2900: [CARBONDATA-3078] Disable explain collector for coun...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2900 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1507/ ---
[GitHub] carbondata pull request #2804: [CARBONDATA-2996] CarbonSchemaReader support ...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2804#discussion_r230983007
--- Diff:
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonSchemaReader.java
---
@@ -61,14 +65,121 @@ public static Schema readSchemaInSchemaFile(String
schemaFilePath) throws IOExce
return new Schema(schemaList);
}
+ /**
+ * get carbondata/carbonindex file in path
+ *
+ * @param path carbon file path
+ * @return CarbonFile array
+ */
+ private static CarbonFile[] getCarbonFile(String path, final String
extension) {
+String dataFilePath = path;
+if (!(dataFilePath.contains(extension))) {
+ CarbonFile[] carbonFiles = FileFactory
+ .getCarbonFile(path)
+ .listFiles(new CarbonFileFilter() {
+@Override
+public boolean accept(CarbonFile file) {
+ if (file == null) {
+return false;
+ }
+ return file.getName().endsWith(extension);
+}
+ });
+ if (carbonFiles == null || carbonFiles.length < 1) {
+throw new RuntimeException("Carbon file not exists.");
--- End diff --
ok, done
---
[GitHub] carbondata pull request #2804: [CARBONDATA-2996] CarbonSchemaReader support ...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2804#discussion_r230982799
--- Diff:
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonSchemaReader.java
---
@@ -61,14 +65,121 @@ public static Schema readSchemaInSchemaFile(String
schemaFilePath) throws IOExce
return new Schema(schemaList);
}
+ /**
+ * get carbondata/carbonindex file in path
+ *
+ * @param path carbon file path
+ * @return CarbonFile array
+ */
+ private static CarbonFile[] getCarbonFile(String path, final String
extension) {
+String dataFilePath = path;
+if (!(dataFilePath.contains(extension))) {
+ CarbonFile[] carbonFiles = FileFactory
+ .getCarbonFile(path)
+ .listFiles(new CarbonFileFilter() {
+@Override
+public boolean accept(CarbonFile file) {
+ if (file == null) {
+return false;
+ }
+ return file.getName().endsWith(extension);
+}
+ });
+ if (carbonFiles == null || carbonFiles.length < 1) {
+throw new RuntimeException("Carbon file not exists.");
+ }
+ return carbonFiles;
+}
+return null;
+ }
+
+ /**
+ * read schema from path,
+ * path can be folder path, carbonindex file path, and carbondata file
path
+ * and will not check all files schema
+ *
+ * @param path file/folder path
+ * @return schema
+ * @throws IOException
+ */
+ public static Schema readSchema(String path) throws IOException {
+return readSchema(path, false);
+ }
+
+ /**
+ * read schema from path,
+ * path can be folder path, carbonindex file path, and carbondata file
path
+ * and user can decide whether check all files schema
+ *
+ * @param path file/folder path
+ * @param checkFilesSchema whether check all files schema
+ * @return schema
+ * @throws IOException
+ */
+ public static Schema readSchema(String path, boolean checkFilesSchema)
throws IOException {
--- End diff --
when user only want to check schema and no need to query data, they can use
readSchema. and readSchema also will faster.
---
[GitHub] carbondata pull request #2804: [CARBONDATA-2996] CarbonSchemaReader support ...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2804#discussion_r230982638
--- Diff:
store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonSchemaReaderTest.java
---
@@ -101,18 +104,30 @@ public boolean accept(CarbonFile file) {
String dataFilePath = carbonFiles[0].getAbsolutePath();
Schema schema = CarbonSchemaReader
- .readSchemaInDataFile(dataFilePath)
+ .readSchema(dataFilePath)
.asOriginOrder();
assertEquals(schema.getFieldsLength(), 12);
checkSchema(schema);
+} catch (Throwable e) {
+ e.printStackTrace();
--- End diff --
ok, done,added Assert.fail();
---
[GitHub] carbondata pull request #2804: [CARBONDATA-2996] CarbonSchemaReader support ...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2804#discussion_r230982398 --- Diff: docs/sdk-guide.md --- @@ -685,6 +685,31 @@ Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/ */ public static Schema readSchemaInIndexFile(String indexFilePath); ``` +``` + /** + * read schema from path, + * path can be folder path,carbonindex file path, and carbondata file path + * and will not check all files schema + * + * @param path file/folder path + * @return schema + * @throws IOException + */ + public static Schema readSchema(String path); +``` +``` + /** + * read schema from path, + * path can be folder path,carbonindex file path, and carbondata file path + * and user can decide whether check all files schema + * + * @param path file/folder path + * @param checkFilesSchema whether check all files schema + * @return schema + * @throws IOException + */ + public static Schema readSchema(String path, boolean checkFilesSchema); --- End diff -- ok, done ---
[GitHub] carbondata issue #2818: [CARBONDATA-3011] Add carbon property to configure v...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2818 why haven't test case in this PR? ---
[GitHub] carbondata pull request #2818: [CARBONDATA-3011] Add carbon property to conf...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2818#discussion_r230977735
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java ---
@@ -1558,4 +1564,6 @@ private void validateStringCharacterLimit() {
CarbonCommonConstants.CARBON_MINMAX_ALLOWED_BYTE_COUNT_DEFAULT);
}
}
+
+
--- End diff --
no need two empty lines
---
[GitHub] carbondata issue #2900: [CARBONDATA-3078] Disable explain collector for coun...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2900 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1294/ ---
[jira] [Resolved] (CARBONDATA-3003) Suppor read batch row in CSDK
[ https://issues.apache.org/jira/browse/CARBONDATA-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-3003. -- Resolution: Fixed Fix Version/s: 1.5.1 > Suppor read batch row in CSDK > - > > Key: CARBONDATA-3003 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3003 > Project: CarbonData > Issue Type: Sub-task >Affects Versions: 1.5.0 >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Fix For: 1.5.1 > > Time Spent: 26h > Remaining Estimate: 0h > > Suppor read batch row in CSDK -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2816: [CARBONDATA-3003] Support read batch row in C...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2816 ---
[GitHub] carbondata issue #2816: [CARBONDATA-3003] Support read batch row in CSDK
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2816 LGTM ---
[GitHub] carbondata issue #2899: [CARBONDATA-3073] Support configure TableProperties ...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2899 @KanakaKumar @ajantha-bhat @jackylk Please help to review it. ---
[GitHub] carbondata issue #2898: [CARBONDATA-3077] Fixed query failure in fileformat ...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2898 @manishgupta88 What if the user use fileformat carbontable and normal carbontable at the same time? For example, creating/using/droping fileformat table and then creating/using/droping normal carbon table, these tables are with the same name. Will this be OK? ---
[GitHub] carbondata pull request #2900: [CARBONDATA-3078] Disable explain collector f...
Github user kevinjmh commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2900#discussion_r230972236
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
---
@@ -575,6 +576,8 @@ private BitSet setMatchedPartitions(String
partitionIds, Expression filter,
*/
public BlockMappingVO getBlockRowCount(Job job, CarbonTable table,
List partitions) throws IOException {
+// no useful information for count star query without filter, so
disable explain collector
+ExplainCollector.remove();
--- End diff --
You are right.
Normal query flow goes to `CarbonInputFormat#getPrunedBlocklets` and
initialize the pruning info for table we queried. But count star query without
filter use a different query plan, it does not go into that method, so no
pruning info does not init. When it calls default data map to prune(with a null
filter), exception will occur during settingg pruning info.
One solution is to init the pruning info for this type of query in mrthod
`getBlockRowCount`. But considering
no useful information about block/blocklet pruning for such query(actually
no pruning), I choose to disable the expalin collector instead.
---
[GitHub] carbondata pull request #2900: [CARBONDATA-3078] Disable explain collector f...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2900#discussion_r230970553
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
---
@@ -575,6 +576,8 @@ private BitSet setMatchedPartitions(String
partitionIds, Expression filter,
*/
public BlockMappingVO getBlockRowCount(Job job, CarbonTable table,
List partitions) throws IOException {
+// no useful information for count star query without filter, so
disable explain collector
+ExplainCollector.remove();
--- End diff --
I think this modification just try to avoid the problem but don't actually
solve the problem.
Can you explain what is the root cause of that problem?
---
[GitHub] carbondata issue #2900: [CARBONDATA-3078] Disable explain collector for coun...
Github user kevinjmh commented on the issue: https://github.com/apache/carbondata/pull/2900 retest this please ---
[GitHub] carbondata issue #2896: [CARBONDATA-3075] Select Filter fails for Legacy sto...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2896 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1506/ ---
[GitHub] carbondata issue #2896: [CARBONDATA-3075] Select Filter fails for Legacy sto...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2896 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9554/ ---
[GitHub] carbondata issue #2896: [CARBONDATA-3075] Select Filter fails for Legacy sto...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2896 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1293/ ---
[GitHub] carbondata issue #2896: [CARBONDATA-3075] Select Filter fails for Legacy sto...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2896 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1505/ ---
[GitHub] carbondata issue #2896: [CARBONDATA-3075] Select Filter fails for Legacy sto...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2896 Retest this please ---
[GitHub] carbondata issue #2896: [CARBONDATA-3075] Select Filter fails for Legacy sto...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2896 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9553/ ---
[GitHub] carbondata issue #2896: [CARBONDATA-3075] Select Filter fails for Legacy sto...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2896 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1292/ ---
[GitHub] carbondata issue #2895: [HOTFIX] Fix NPE in spark, when same vector reads fi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2895 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9552/ ---
[GitHub] carbondata issue #2895: [HOTFIX] Fix NPE in spark, when same vector reads fi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2895 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1291/ ---
[GitHub] carbondata issue #2898: [CARBONDATA-3077] Fixed query failure in fileformat ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2898 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1289/ ---
[GitHub] carbondata issue #2898: [CARBONDATA-3077] Fixed query failure in fileformat ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2898 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9550/ ---
[GitHub] carbondata issue #2901: [CARBONDATA-3081] Fixed NPE for boolean type column ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2901 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1288/ ---
[GitHub] carbondata issue #2898: [CARBONDATA-3077] Fixed query failure in fileformat ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2898 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1504/ ---
[GitHub] carbondata issue #2901: [CARBONDATA-3081] Fixed NPE for boolean type column ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2901 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9549/ ---
[GitHub] carbondata issue #2896: [CARBONDATA-3075] Select Filter fails for Legacy sto...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2896 Retest this please ---
[GitHub] carbondata issue #2816: [CARBONDATA-3003] Support read batch row in CSDK
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2816 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9548/ ---
[GitHub] carbondata issue #2895: [HOTFIX] Fix NPE in spark, when same vector reads fi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2895 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1503/ ---
[GitHub] carbondata pull request #2901: [CARBONDATA-3081] Fixed NPE for boolean type ...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2901#discussion_r230818772
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/util/CarbonVectorizedRecordReader.java
---
@@ -171,13 +171,20 @@ public Object getCurrentValue() throws IOException,
InterruptedException {
rowCount += 1;
Object[] row = new Object[carbonColumnarBatch.columnVectors.length];
for (int i = 0; i < carbonColumnarBatch.columnVectors.length; i ++) {
+ Object data = carbonColumnarBatch.columnVectors[i].getData(batchIdx
- 1);
if (carbonColumnarBatch.columnVectors[i].getType() ==
DataTypes.STRING
|| carbonColumnarBatch.columnVectors[i].getType() ==
DataTypes.VARCHAR) {
-byte[] data = (byte[])
carbonColumnarBatch.columnVectors[i].getData(batchIdx - 1);
-row[i] = ByteUtil.toString(data, 0, data.length);
+if (data == null) {
+ row[i] = null;
+} else {
+ row[i] = ByteUtil.toString((byte[]) data, 0, (((byte[])
data).length));
+}
} else if (carbonColumnarBatch.columnVectors[i].getType() ==
DataTypes.BOOLEAN) {
-byte data = (byte)
carbonColumnarBatch.columnVectors[i].getData(batchIdx - 1);
-row[i] = ByteUtil.toBoolean(data);
+if (data == null) {
+ row[i] = null;
+} else {
+ row[i] = ByteUtil.toBoolean((byte) data);
+}
--- End diff --
For other dataTypes is the same handling of null required?...If required
then you can move the if check for `data == null` before first if check and set
the row to null if data is null and continue
---
