[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r162290297 --- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java --- @@ -196,6 +201,10 @@ private DataMapRowImpl loadToUnsafe(DataFileFooter fileFooter, unsafeMemorySummaryDMStore.getSchema()[taskMinMaxOrdinal], maxValues, TASK_MAX_VALUES_INDEX, false); ordinal++; + taskMinMaxOrdinal++; + + summaryRow.setByteArray(minMaxIndex.getNullValues().toByteArray(), taskMinMaxOrdinal); --- End diff -- Added new routine addTaskNullValues which is going to OR all the Null Bitsets from all the blocklets and place the final value into the summary row. Like the one done for minMax. ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162290027
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeGrtrThanEquaToFilterExecuterImpl.java
---
@@ -149,13 +154,17 @@ private boolean isScanRequired(byte[] blockMaxValue,
byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] maxValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] maxValue, boolean nullValue,
Object[] filterValue,
DataType dataType) {
Object value = DataTypeUtil.getMeasureObjectFromDataType(maxValue,
dataType);
for (int i = 0; i < filterValue.length; i++) {
// TODO handle min and max for null values.
if (filterValue[i] == null) {
-return true;
+if (nullValue == true) {
+ return true;
+} else {
+ return false;
+}
--- End diff --
Done
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162289986
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanEqualFilterExecuterImpl.java
---
@@ -150,14 +155,17 @@ private boolean isScanRequired(byte[] blockMinValue,
byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] minValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] minValue, boolean nullValue,
Object[] filterValue,
DataType dataType) {
Object value =
DataTypeUtil.getMeasureObjectFromDataType(minValue, dataType);
for (int i = 0; i < filterValue.length; i++) {
- // TODO handle min and max for null values.
if (filterValue[i] == null) {
-return true;
+if (nullValue == true) {
+ return true;
+} else {
+ return false;
+}
--- End diff --
Done
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162289956
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanFiterExecuterImpl.java
---
@@ -153,13 +158,17 @@ private boolean isScanRequired(byte[] blockMinValue,
byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] minValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] minValue, boolean nullValue,
Object[] filterValue,
DataType dataType) {
Object value = DataTypeUtil.getMeasureObjectFromDataType(minValue,
dataType);
for (int i = 0; i < filterValue.length; i++) {
// TODO handle min and max for null values.
if (filterValue[i] == null) {
-return true;
+if (nullValue == true) {
+ return true;
+} else {
+ return false;
+}
--- End diff --
Done
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162289924
--- Diff:
hadoop/src/test/java/org/apache/carbondata/hadoop/test/util/StoreCreator.java
---
@@ -520,7 +522,19 @@ public static String readCurrentTime() {
}
public static void main(String[] args) {
-StoreCreator.createCarbonStore();
+
+SimpleDateFormat parser = null;
+parser = new SimpleDateFormat("-MM-dd");
+parser.setTimeZone(TimeZone.getTimeZone("IST"));
+
+String v = "2011-11-11";
+try {
+ long value = parser.parse(v).getTime() * 1000L;
+ System.out.print(value);
+} catch (ParseException pe) {
+
+}
+// StoreCreator.createCarbonStore();
--- End diff --
Removed
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162126330
--- Diff:
hadoop/src/test/java/org/apache/carbondata/hadoop/test/util/StoreCreator.java
---
@@ -520,7 +522,19 @@ public static String readCurrentTime() {
}
public static void main(String[] args) {
-StoreCreator.createCarbonStore();
+
+SimpleDateFormat parser = null;
+parser = new SimpleDateFormat("-MM-dd");
+parser.setTimeZone(TimeZone.getTimeZone("IST"));
+
+String v = "2011-11-11";
+try {
+ long value = parser.parse(v).getTime() * 1000L;
+ System.out.print(value);
+} catch (ParseException pe) {
+
+}
+// StoreCreator.createCarbonStore();
--- End diff --
why this class i changed
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162125695
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanFiterExecuterImpl.java
---
@@ -153,13 +158,17 @@ private boolean isScanRequired(byte[] blockMinValue,
byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] minValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] minValue, boolean nullValue,
Object[] filterValue,
DataType dataType) {
Object value = DataTypeUtil.getMeasureObjectFromDataType(minValue,
dataType);
for (int i = 0; i < filterValue.length; i++) {
// TODO handle min and max for null values.
if (filterValue[i] == null) {
-return true;
+if (nullValue == true) {
+ return true;
+} else {
+ return false;
+}
--- End diff --
just return nullValue
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162124981
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanEqualFilterExecuterImpl.java
---
@@ -150,14 +155,17 @@ private boolean isScanRequired(byte[] blockMinValue,
byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] minValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] minValue, boolean nullValue,
Object[] filterValue,
DataType dataType) {
Object value =
DataTypeUtil.getMeasureObjectFromDataType(minValue, dataType);
for (int i = 0; i < filterValue.length; i++) {
- // TODO handle min and max for null values.
if (filterValue[i] == null) {
-return true;
+if (nullValue == true) {
+ return true;
+} else {
+ return false;
+}
--- End diff --
Just return nullValue no need if condition
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162124619
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeGrtrThanEquaToFilterExecuterImpl.java
---
@@ -149,13 +154,17 @@ private boolean isScanRequired(byte[] blockMaxValue,
byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] maxValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] maxValue, boolean nullValue,
Object[] filterValue,
DataType dataType) {
Object value = DataTypeUtil.getMeasureObjectFromDataType(maxValue,
dataType);
for (int i = 0; i < filterValue.length; i++) {
// TODO handle min and max for null values.
if (filterValue[i] == null) {
-return true;
+if (nullValue == true) {
+ return true;
+} else {
+ return false;
+}
--- End diff --
you no need use if condition, just return nullValue
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r162053147 --- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java --- @@ -196,6 +201,10 @@ private DataMapRowImpl loadToUnsafe(DataFileFooter fileFooter, unsafeMemorySummaryDMStore.getSchema()[taskMinMaxOrdinal], maxValues, TASK_MAX_VALUES_INDEX, false); ordinal++; + taskMinMaxOrdinal++; + + summaryRow.setByteArray(minMaxIndex.getNullValues().toByteArray(), taskMinMaxOrdinal); --- End diff -- We need to calculate null bits based on whole blocks and then set to the summary ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r161963305 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java --- @@ -25,6 +25,8 @@ Object getMax(); + byte getNull(); --- End diff -- We can do it. But in later phase of execution the Boolean will again get converted to Byte[]. So it doesn't make that much difference. ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431619
--- Diff: format/src/main/thrift/carbondata.thrift ---
@@ -45,6 +45,7 @@ struct BlockletBTreeIndex{
struct BlockletMinMaxIndex{
1: required list min_values; //Min value of all columns of one
blocklet Bit-Packed
2: required list max_values; //Max value of all columns of one
blocklet Bit-Packed
+3: optional list null_value; // Null Bool turn on if null
value present.
--- End diff --
Done. Stored as a Bitset
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431581
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/ExampleUtils.scala
---
@@ -39,8 +39,8 @@ object ExampleUtils {
val warehouse = s"$rootPath/examples/spark2/target/warehouse"
CarbonProperties.getInstance()
- .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"/MM/dd HH:mm:ss")
- .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "/MM/dd")
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"-MM-dd HH:mm:ss")
--- End diff --
Done
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431568
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala
---
@@ -62,9 +62,9 @@ object CarbonSessionExample {
spark.sql(
s"""
- | SELECT *
- | FROM carbon_table
- | WHERE stringfield = 'spark' AND decimalField > 40
+| SELECT *
--- End diff --
Rectified
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r161431317 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java --- @@ -34,7 +34,7 @@ * @return */ BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minValue, - String uniqueBlockPath); + byte[] nullValue, String uniqueBlockPath); --- End diff -- Done ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431335
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/TablePageStatistics.java
---
@@ -71,6 +82,10 @@ private void updateMeasureMinMax(EncodedColumnPage[]
measures) {
return dimensionMaxValue;
}
+ public byte[] getDimentionNullValue() {
--- End diff --
Rectified
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r161431288 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/FilterExecuter.java --- @@ -46,7 +46,7 @@ boolean applyFilter(RowIntf value, int dimOrdinalMax) * @param blockMinValue * @return BitSet */ - BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue); + BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue, byte[] blockNullValue); --- End diff -- Done ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r161431299 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java --- @@ -43,5 +43,6 @@ BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minVal * * @return */ - Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue); + Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue, --- End diff -- Done ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431217
--- Diff:
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataRefNodeWrapper.java
---
@@ -99,6 +99,10 @@ public BlockletDataRefNodeWrapper(List
blockInfos, int index,
return null;
}
+ @Override public byte[] getColumnsNullValue() {
--- End diff --
Done
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431169
--- Diff:
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java
---
@@ -728,9 +746,9 @@ private DataMapRow convertToRow(IndexKey key) {
private List getPartitions() {
DataMapRow unsafeRow = unsafeMemorySummaryDMStore.getUnsafeRow(0);
-if (unsafeRow.getColumnCount() > 2) {
+if (unsafeRow.getColumnCount() > 3) {
List partitions = new ArrayList<>();
- DataMapRow row = unsafeRow.getRow(2);
+ DataMapRow row = unsafeRow.getRow(3);
--- End diff --
Done
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r161430749 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java --- @@ -25,6 +25,8 @@ Object getMax(); + byte getNull(); --- End diff -- getNull() will returns Null value present in page or not. ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r161366923 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java --- @@ -25,6 +25,8 @@ Object getMax(); + byte getNull(); --- End diff -- SimpleStatsResult is the statistics for a ColumnPage, what does getNull() means for a page? ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r161366903 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java --- @@ -25,6 +25,8 @@ Object getMax(); + byte getNull(); --- End diff -- SimpleStatsResult is the statistics for a ColumnPage, what does getNull() means for a page? ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r160157245
--- Diff: format/src/main/thrift/carbondata.thrift ---
@@ -45,6 +45,7 @@ struct BlockletBTreeIndex{
struct BlockletMinMaxIndex{
1: required list min_values; //Min value of all columns of one
blocklet Bit-Packed
2: required list max_values; //Max value of all columns of one
blocklet Bit-Packed
+3: optional list null_value; // Null Bool turn on if null
value present.
--- End diff --
we can use PresenceMeta to store null value
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r159832789 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java --- @@ -34,7 +34,7 @@ * @return */ BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minValue, - String uniqueBlockPath); + byte[] nullValue, String uniqueBlockPath); --- End diff -- Here also use BlockletStats object ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r159832781 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java --- @@ -43,5 +43,6 @@ BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minVal * * @return */ - Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue); + Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue, --- End diff -- Here also use BlockletStats object ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r159832676 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/FilterExecuter.java --- @@ -46,7 +46,7 @@ boolean applyFilter(RowIntf value, int dimOrdinalMax) * @param blockMinValue * @return BitSet */ - BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue); + BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue, byte[] blockNullValue); --- End diff -- Better wrap these parameters to new BlockletStats object ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159832035
--- Diff:
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java
---
@@ -728,9 +746,9 @@ private DataMapRow convertToRow(IndexKey key) {
private List getPartitions() {
DataMapRow unsafeRow = unsafeMemorySummaryDMStore.getUnsafeRow(0);
-if (unsafeRow.getColumnCount() > 2) {
+if (unsafeRow.getColumnCount() > 3) {
List partitions = new ArrayList<>();
- DataMapRow row = unsafeRow.getRow(2);
+ DataMapRow row = unsafeRow.getRow(3);
--- End diff --
Please add constant to partition index
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159832127
--- Diff:
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataRefNodeWrapper.java
---
@@ -99,6 +99,10 @@ public BlockletDataRefNodeWrapper(List
blockInfos, int index,
return null;
}
+ @Override public byte[] getColumnsNullValue() {
--- End diff --
Change byte[] to BitSet
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159831719
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/TablePageStatistics.java
---
@@ -71,6 +82,10 @@ private void updateMeasureMinMax(EncodedColumnPage[]
measures) {
return dimensionMaxValue;
}
+ public byte[] getDimentionNullValue() {
--- End diff --
typo getDimensionNullValue
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r159829745 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java --- @@ -25,6 +25,8 @@ Object getMax(); + byte getNull(); --- End diff -- Better use boolean and change method to isNull ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159829285
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala
---
@@ -62,9 +62,9 @@ object CarbonSessionExample {
spark.sql(
s"""
- | SELECT *
- | FROM carbon_table
- | WHERE stringfield = 'spark' AND decimalField > 40
+| SELECT *
--- End diff --
Don't change this example.
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159829185
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/ExampleUtils.scala
---
@@ -39,8 +39,8 @@ object ExampleUtils {
val warehouse = s"$rootPath/examples/spark2/target/warehouse"
CarbonProperties.getInstance()
- .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"/MM/dd HH:mm:ss")
- .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "/MM/dd")
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"-MM-dd HH:mm:ss")
--- End diff --
Don't change the example class
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159829085
--- Diff: format/src/main/thrift/carbondata.thrift ---
@@ -45,6 +45,7 @@ struct BlockletBTreeIndex{
struct BlockletMinMaxIndex{
1: required list min_values; //Min value of all columns of one
blocklet Bit-Packed
2: required list max_values; //Max value of all columns of one
blocklet Bit-Packed
+3: optional list null_value; // Null Bool turn on if null
value present.
--- End diff --
Better use binary and try to use BitSet for null values
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r159368019 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/chunk/AbstractRawColumnChunk.java --- @@ -29,6 +29,8 @@ private byte[][] maxValues; + private byte[] nullValues; --- End diff -- please add comment ---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1751#discussion_r159368023 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/DataRefNode.java --- @@ -71,6 +71,8 @@ */ byte[][] getColumnsMinValue(); + byte[] getColumnsNullValue(); --- End diff -- please add comment ---
