[GitHub] carbondata issue #2996: [WIP] Fix Rename-Fail & Datamap-creation-Fail
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2996 retest this please ---
[GitHub] carbondata pull request #3028: [CARBONDATA-3205]Fix Get Local Dictionary for...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/3028 [CARBONDATA-3205]Fix Get Local Dictionary for empty Array of Struct Problem: When a table contains a nested complex schema with Array of Struct as empty data, get dictionary for empty Array of Struct fails. Solution: Check if usedDictionary is empty, then return null - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Added a test case - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata nested Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/3028.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3028 commit 99e6233dcdb4b488d0f0366ac5966d4e9c4f895c Author: Indhumathi27 Date: 2018-12-27T12:02:48Z [CARBONDATA-3205]Fix Get Local Dictionary for empty Array of Struct ---
[GitHub] carbondata pull request #2998: [WIP] Test Load using carbondata
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2998 [WIP] Test Load using carbondata Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata testload Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2998.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2998 commit b113f3aba594fac43bef4bfecc6e92ad270e0327 Author: Indhumathi27 Date: 2018-12-18T10:38:42Z test using carbondata ---
[GitHub] carbondata pull request #2989: [CARBONDATA-3175]Fix Testcase failures in com...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2989 [CARBONDATA-3175]Fix Testcase failures in complex deleimiters Fix Testcase failures in complex deleimiters: Changed the complex delimiter in testcases - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata fixsdv Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2989.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2989 commit ccd2347f6413f0d92e326b4c1013b415eabafbc1 Author: Indhumathi27 Date: 2018-12-14T10:56:00Z [CARBONDATA-3175]Fix Testcase failures in complex deleimiters ---
[GitHub] carbondata pull request #2971: [TEST] Test loading performance of range_sort
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2971#discussion_r238184022
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessorStepOnSpark.scala
---
@@ -305,4 +307,107 @@ object DataLoadProcessorStepOnSpark {
e)
}
}
+
+ def sortAdnWriteFunc(
--- End diff --
Please change the method name from sortAdnWriteFunc to sortAndWriteFunc
---
[GitHub] carbondata pull request #2821: [CARBONDATA-3017] Map DDL Support
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2821#discussion_r238161031
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateDDLForComplexMapType.scala
---
@@ -0,0 +1,330 @@
+/*
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+*
+http://www.apache.org/licenses/LICENSE-2.0
+*
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+package
org.apache.carbondata.spark.testsuite.createTable.TestCreateDDLForComplexMapType
+
+import java.io.File
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestCreateDDLForComplexMapType extends QueryTest with
BeforeAndAfterAll {
+ private val conf: Configuration = new Configuration(false)
+
+ val rootPath = new File(this.getClass.getResource("/").getPath
+ + "../../../..").getCanonicalPath
+
+ val path =
s"$rootPath/examples/spark2/src/main/resources/mapDDLTestData.csv"
+
+ override def beforeAll(): Unit = {
+sql("DROP TABLE IF EXISTS carbon")
+ }
+
+ test("Single Map One Level") {
+sql("DROP TABLE IF EXISTS carbon")
+sql(
+ s"""
+ | CREATE TABLE carbon(
+ | mapField map
+ | )
+ | STORED BY 'carbondata'
+ | """
+.stripMargin)
+val desc = sql(
+ s"""
+ | Describe Formatted
+ | carbon
+ | """.stripMargin).collect()
+
assert(desc(0).get(1).asInstanceOf[String].trim.equals("map"))
+ }
+
+ test("Single Map with Two Nested Level") {
+sql("DROP TABLE IF EXISTS carbon")
+sql(
+ s"""
+ | CREATE TABLE carbon(
+ | mapField map>
+ | )
+ | STORED BY
+ |'carbondata'
+ |"""
+.stripMargin)
+val desc = sql(
+ s"""
+ | Describe Formatted
+ | carbon
+ | """.stripMargin).collect()
+
assert(desc(0).get(1).asInstanceOf[String].trim.equals("map>"))
+ }
+
+ test("Map Type with array type as value") {
+sql("DROP TABLE IF EXISTS carbon")
+sql(
+ s"""
+ | CREATE TABLE carbon(
+ | mapField map>
+ | )
+ | STORED BY 'carbondata'
+ |
+ """
+.stripMargin)
+val desc = sql(
+ s"""
+ | Describe Formatted
+ | carbon
+ | """.stripMargin).collect()
+
assert(desc(0).get(1).asInstanceOf[String].trim.equals("map>"))
+ }
+
+ test("Map Type with struct type as value") {
+sql("DROP TABLE IF EXISTS carbon")
+sql(
+ s"""
+ | CREATE TABLE carbon(
+ | mapField map>
+ | )
+ | STORED BY
+ | 'carbondata'
+ | """
+.stripMargin)
+val desc = sql(
+ s"""
+ | Describe Formatted
+ | carbon
+ | """.stripMargin).collect()
+assert(desc(0).get(1).asInstanceOf[String].trim
+ .equals("map>"))
+ }
+
+ test("Map Type as child to struct type") {
+sql("DROP TABLE IF EXISTS carbon")
+sql(
+ s"""
+ | CREATE TABLE carbon(
+ | mapField struct>
+ | )
+ | STORED BY
+ |'carbondata' """
+.stripMargin)
[GitHub] carbondata pull request #2821: [CARBONDATA-3017] Map DDL Support
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2821#discussion_r238160123
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateDDLForComplexMapType.scala
---
@@ -0,0 +1,330 @@
+/*
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+*
+http://www.apache.org/licenses/LICENSE-2.0
+*
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+package
org.apache.carbondata.spark.testsuite.createTable.TestCreateDDLForComplexMapType
+
+import java.io.File
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestCreateDDLForComplexMapType extends QueryTest with
BeforeAndAfterAll {
+ private val conf: Configuration = new Configuration(false)
+
+ val rootPath = new File(this.getClass.getResource("/").getPath
+ + "../../../..").getCanonicalPath
+
+ val path =
s"$rootPath/examples/spark2/src/main/resources/mapDDLTestData.csv"
+
+ override def beforeAll(): Unit = {
+sql("DROP TABLE IF EXISTS carbon")
+ }
+
+ test("Single Map One Level") {
+sql("DROP TABLE IF EXISTS carbon")
+sql(
+ s"""
+ | CREATE TABLE carbon(
+ | mapField map
+ | )
+ | STORED BY 'carbondata'
+ | """
+.stripMargin)
+val desc = sql(
+ s"""
+ | Describe Formatted
+ | carbon
+ | """.stripMargin).collect()
+
assert(desc(0).get(1).asInstanceOf[String].trim.equals("map"))
+ }
+
+ test("Single Map with Two Nested Level") {
+sql("DROP TABLE IF EXISTS carbon")
+sql(
+ s"""
+ | CREATE TABLE carbon(
+ | mapField map>
+ | )
+ | STORED BY
+ |'carbondata'
+ |"""
+.stripMargin)
+val desc = sql(
+ s"""
+ | Describe Formatted
+ | carbon
+ | """.stripMargin).collect()
+
assert(desc(0).get(1).asInstanceOf[String].trim.equals("map>"))
+ }
+
+ test("Map Type with array type as value") {
+sql("DROP TABLE IF EXISTS carbon")
+sql(
+ s"""
+ | CREATE TABLE carbon(
+ | mapField map>
+ | )
+ | STORED BY 'carbondata'
+ |
+ """
+.stripMargin)
+val desc = sql(
+ s"""
+ | Describe Formatted
+ | carbon
+ | """.stripMargin).collect()
+
assert(desc(0).get(1).asInstanceOf[String].trim.equals("map>"))
+ }
+
+ test("Map Type with struct type as value") {
+sql("DROP TABLE IF EXISTS carbon")
+sql(
+ s"""
+ | CREATE TABLE carbon(
+ | mapField map>
+ | )
+ | STORED BY
+ | 'carbondata'
+ | """
+.stripMargin)
+val desc = sql(
+ s"""
+ | Describe Formatted
+ | carbon
+ | """.stripMargin).collect()
+assert(desc(0).get(1).asInstanceOf[String].trim
+ .equals("map>"))
+ }
+
+ test("Map Type as child to struct type") {
+sql("DROP TABLE IF EXISTS carbon")
+sql(
+ s"""
+ | CREATE TABLE carbon(
+ | mapField struct>
+ | )
+ | STORED BY
+ |'carbondata' """
+.stripMargin)
[GitHub] carbondata pull request #2963: [CARBONDATA-3139] Fix bugs in MinMaxDataMap e...
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2963#discussion_r238156188
--- Diff:
integration/spark2/src/test/scala/org/apache/carbondata/datamap/minmax/MinMaxDataMapFunctionSuite.scala
---
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.minmax
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class MinMaxDataMapFunctionSuite extends QueryTest with BeforeAndAfterAll {
+ private val minmaxDataMapFactoryName =
"org.apache.carbondata.datamap.minmax.MinMaxDataMapFactory"
+ var originalStatEnabled = CarbonProperties.getInstance().getProperty(
+CarbonCommonConstants.ENABLE_QUERY_STATISTICS,
+CarbonCommonConstants.ENABLE_QUERY_STATISTICS_DEFAULT)
+
+ override protected def beforeAll(): Unit = {
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.ENABLE_QUERY_STATISTICS, "true")
+
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+ "-MM-dd")
+
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+ "-MM-dd HH:mm:ss")
--- End diff --
Please check whether we can use the default timestamp/date format here.
org.apache.carbondata.core.constants.CarbonCommonConstants#CARBON_TIMESTAMP_DEFAULT_FORMAT
---
[GitHub] carbondata pull request #2963: [CARBONDATA-3139] Fix bugs in MinMaxDataMap e...
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2963#discussion_r238152299
--- Diff:
datamap/example/src/main/java/org/apache/carbondata/datamap/minmax/AbstractMinMaxDataMapWriter.java
---
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.minmax;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.List;
+
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datamap.Segment;
+import org.apache.carbondata.core.datamap.dev.DataMapWriter;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.datastore.page.ColumnPage;
+import
org.apache.carbondata.core.datastore.page.encoding.bool.BooleanConvert;
+import
org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatsCollector;
+import
org.apache.carbondata.core.datastore.page.statistics.KeyPageStatsCollector;
+import
org.apache.carbondata.core.datastore.page.statistics.PrimitivePageStatsCollector;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.encoder.Encoding;
+import
org.apache.carbondata.core.metadata.schema.table.column.CarbonColumn;
+import org.apache.carbondata.core.util.CarbonUtil;
+import org.apache.carbondata.core.util.DataTypeUtil;
+
+import org.apache.log4j.Logger;
+
+/**
+ * We will record the min & max value for each index column in each
blocklet.
+ * Since the size of index is quite small, we will combine the index for
all index columns
+ * in one file.
+ */
+public abstract class AbstractMinMaxDataMapWriter extends DataMapWriter {
+ private static final Logger LOGGER = LogServiceFactory.getLogService(
+ AbstractMinMaxDataMapWriter.class.getName());
+
+ private ColumnPageStatsCollector[] indexColumnMinMaxCollectors;
+ protected int currentBlockletId;
+ private String currentIndexFile;
+ private DataOutputStream currentIndexFileOutStream;
+
+ public AbstractMinMaxDataMapWriter(String tablePath, String dataMapName,
+ List indexColumns, Segment segment, String shardName)
throws IOException {
+super(tablePath, dataMapName, indexColumns, segment, shardName);
+initStatsCollector();
+initDataMapFile();
+ }
+
+ private void initStatsCollector() {
+indexColumnMinMaxCollectors = new
ColumnPageStatsCollector[indexColumns.size()];
+CarbonColumn indexCol;
+for (int i = 0; i < indexColumns.size(); i++) {
+ indexCol = indexColumns.get(i);
+ if (indexCol.isMeasure()
+ || (indexCol.isDimension()
+ && DataTypeUtil.isPrimitiveColumn(indexCol.getDataType())
+ && !indexCol.hasEncoding(Encoding.DICTIONARY)
+ && !indexCol.hasEncoding(Encoding.DIRECT_DICTIONARY))) {
+indexColumnMinMaxCollectors[i] =
PrimitivePageStatsCollector.newInstance(
+indexColumns.get(i).getDataType());
+ } else {
+indexColumnMinMaxCollectors[i] =
KeyPageStatsCollector.newInstance(DataTypes.BYTE_ARRAY);
+ }
+}
+ }
+
+ private void initDataMapFile() throws IOException {
+if (!FileFactory.isFileExist(dataMapPath) &&
+!FileFactory.mkdirs(dataMapPath,
FileFactory.getFileType(dataMapPath))) {
+ throw new IOException("Failed to create directory " + dataMapPath);
+}
+
+try {
+ currentIndexFile = MinMaxIndexDataMap.getIndexFile(dataMapPath,
+ MinMaxIndexHolder.MINMAX_INDEX_PREFFIX + indexColumns.size());
+ FileFactory.createNewFile(currentIndexFile,
FileFactory
[GitHub] carbondata pull request #2960: [WIP] Update the Project List
Github user Indhumathi27 closed the pull request at: https://github.com/apache/carbondata/pull/2960 ---
[GitHub] carbondata pull request #2960: [WIP] Update the Project List
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2960 [WIP] Update the Project List Update the Project List - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata scan Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2960.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2960 commit 20f9e53e7eb11fe7ce850f30630d77d9b377b63a Author: Indhumathi27 Date: 2018-11-28T10:48:13Z [CARBONDATA-3137] Update the Project List ---
[GitHub] carbondata issue #2937: [CARBONDATA-3114]Remove Null Values for a Dictionary...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2937 Retest this please ---
[GitHub] carbondata pull request #2939: [CARBONDATA-3115] Fix CodeGen error in preagg...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2939 [CARBONDATA-3115] Fix CodeGen error in preaggregate table and codegen display issue in oldstores **Problem:** 1. While querying a preaggregate table, codegen error is displayed. 2. In old stores, code is getting displayed while executing queries. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata codegen Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2939.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2939 commit b1c1cd2bb354afc8566774c6b002d511c4a99f65 Author: Indhumathi27 Date: 2018-11-21T11:53:25Z [CARBONDATA-3115]Fix Codegen issue ---
[GitHub] carbondata pull request #2937: [CARBONDATA-3114]Remove Null Values in all ty...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2937 [CARBONDATA-3114]Remove Null Values in all types of columns for Range Filters **Problem:** Null Values are not removed in case of RangeFilters, if column is a dictionary and no_inverted_index column. **Solution:** Remove NULL values in case of RangeFilters for columns of all types. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata timestamp_di Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2937.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2937 commit fb0912fc938ed0be3c9451ce21bde8d59505d477 Author: Indhumathi27 Date: 2018-11-21T09:51:49Z [CARBONDATA-3114]Remove Null Values in all types of columns for RangeFilters ---
[GitHub] carbondata issue #2932: [HOTFIX]Fix Describe Formatted Testcases
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2932 Retest this please ---
[GitHub] carbondata issue #2932: [HOTFIX]Fix Describe Formatted Testcases
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2932 Retest this please ---
[GitHub] carbondata issue #2932: [HOTFIX]Fix Describe Formatted Testcases
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2932 retest this please ---
[GitHub] carbondata pull request #2932: [HOTFIX]Fix Describe Formatted Testcases
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2932 [HOTFIX]Fix Describe Formatted Testcases - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata testcasefix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2932.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2932 commit 6123f3008c62cd3f60ec391f5c16f141d9b00428 Author: Indhumathi27 Date: 2018-11-20T09:02:00Z Fix Describe Formatted Testcases ---
[GitHub] carbondata pull request #2928: [CARBONDATA-3106] Written_by_APPNAME not seri...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2928 [CARBONDATA-3106] Written_by_APPNAME not serialized in executor with global sort **Problem:** Written_By_APPNAME when added in carbonproperty is not serialized in executor with global sort **Solution:** Add Written_by_APPNAME in hadoop conf and in executor get it from configuration and add to carbonproperty - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata global_sort Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2928.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2928 commit d956df1340e2dd6309ed688dc71b4046afac336e Author: Indhumathi27 Date: 2018-11-16T16:19:16Z [CARBONDATA-3106] WrittenbyAPI not serialized in executor with globalsort ---
[GitHub] carbondata issue #2916: [CARBONDATA-3096] Wrong records size on the input me...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2916 retest this please ---
[GitHub] carbondata issue #2849: [CARBONDATA-2896] Added TestCases for Adaptive encod...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2849 Retest this please ---
[GitHub] carbondata issue #2896: [CARBONDATA-3075] Select Filter fails for Legacy sto...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2896 Retest this please ---
[GitHub] carbondata issue #2896: [CARBONDATA-3075] Select Filter fails for Legacy sto...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2896 Retest this please ---
[GitHub] carbondata issue #2896: [CARBONDATA-3075] Select Filter fails for Legacy sto...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2896 Retest this please ---
[GitHub] carbondata pull request #2896: [CARBONDATA-3075] Select Filter fails for Leg...
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2896#discussion_r230731270
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/executor/impl/AbstractQueryExecutor.java
---
@@ -230,6 +230,11 @@ protected void initQuery(QueryModel queryModel) throws
IOException {
updateColumns(queryModel, fileFooter.getColumnInTable(),
blockInfo.getFilePath());
filePathToSegmentPropertiesMap.put(blockInfo.getFilePath(),
segmentProperties);
}
+//In case of Legacy store, set setDirectVectorFill to false,
+// as filling vector directly is not supported for V2 store
--- End diff --
okay
---
[GitHub] carbondata pull request #2896: [WIP] Select Filter fails for Legacy store if...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2896 [WIP] Select Filter fails for Legacy store if DirectVecorFill is enabled **Why this PR?** When **isDirectVectorFill** is set to true, Select filter fails in Legacy Store throwing UnsupportedOperationException **Solution:** Set **isDirectVectorFill** to false for Legacy store - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata scan_comp Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2896.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2896 commit c2af1dc8ed8ea9ec5671d3fa107007c338ecbbef Author: Indhumathi27 Date: 2018-11-05T10:42:24Z [CARBONDATA-3075] Select Filter fails for Legacy store if DirectVectorFill is enabled ---
[GitHub] carbondata pull request #2883: [CARBONDATA-3062] Fix Compatibility issue wit...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2883 [CARBONDATA-3062] Fix Compatibility issue with cache_level as blocklet **Why this PR for?** In case of hybrid store we can have block as well as blocklet schema. Scenario: When there is a hybrid store in which few loads are from legacy store which do not contain the blocklet information and hence they will be, by default have cache_level as BLOCK and few loads with latest store which contain the BLOCKLET information and have cache_level BLOCKLET. For these type of scenarios we need to have separate task and footer schemas. For all loads with/without blocklet info there will not be any additional cost of maintaining 2 variables - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata column_comp Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2883.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2883 ---
[GitHub] carbondata pull request #2852: [WIP]Column Schema objects are present in Dri...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2852 [WIP]Column Schema objects are present in Driver even after dropping table **Problem:** Column Schema objects are present in Driver even after dropping table. **Solution:** After dropping table, remove entry of tableInfo from CarbonMetaDataInstance. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata memory_leak_driver Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2852.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2852 commit 182acdf3b2edd09403303fb15959f4d058e2c759 Author: Indhumathi27 Date: 2018-10-25T04:46:32Z Column Schema objects are present in Driver even after dropping table ---
[GitHub] carbondata pull request #2845: [WIP] Rand function issue
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2845 [WIP] Rand function issue Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata rand Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2845.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2845 commit 5b21078335ee9cd1962b4744d04069526dce08cc Author: Indhumathi27 Date: 2018-10-23T05:59:51Z Check rand() ---
[GitHub] carbondata issue #2785: [CARBONDATA-2986] Table Properties are lost when mul...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2785 retest this please ---
[GitHub] carbondata issue #2734: [CARBONDATA-2946] Unify conversion while writing to ...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2734 @xuchuanyin Please check this PR. The conversion on filter value is now avoided. ---
[GitHub] carbondata issue #2756: [CARBONDATA-2966]Update Documentation For Avro DataT...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2756 Retest this please ---
[GitHub] carbondata pull request #2756: [CARBONDATA-2966]Update Documentation For Avr...
Github user Indhumathi27 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2756#discussion_r220429435 --- Diff: docs/configuration-parameters.md --- @@ -42,6 +42,7 @@ This section provides the details of all the configurations required for the Car | carbon.lock.type | LOCALLOCK | This configuration specifies the type of lock to be acquired during concurrent operations on table. There are following types of lock implementation: - LOCALLOCK: Lock is created on local file system as file. This lock is useful when only one spark driver (thrift server) runs on a machine and no other CarbonData spark application is launched concurrently. - HDFSLOCK: Lock is created on HDFS file system as file. This lock is useful when multiple CarbonData spark applications are launched and no ZooKeeper is running on cluster and HDFS supports file based locking. | | carbon.lock.path | TABLEPATH | This configuration specifies the path where lock files have to be created. Recommended to configure zookeeper lock type or configure HDFS lock path(to this property) in case of S3 file system as locking is not feasible on S3. | | carbon.unsafe.working.memory.in.mb | 512 | CarbonData supports storing data in off-heap memory for certain operations during data loading and query.This helps to avoid the Java GC and thereby improve the overall performance.The Minimum value recommeded is 512MB.Any value below this is reset to default value of 512MB.**NOTE:** The below formulas explain how to arrive at the off-heap size required.Memory Required For Data Loading:(*carbon.number.of.cores.while.loading*) * (Number of tables to load in parallel) * (*offheap.sort.chunk.size.inmb* + *carbon.blockletgroup.size.in.mb* + *carbon.blockletgroup.size.in.mb*/3.5 ). Memory required for Query:SPARK_EXECUTOR_INSTANCES * (*carbon.blockletgroup.size.in.mb* + *carbon.blockletgroup.size.in.mb* * 3.5) * spark.executor.cores | +| carbon.unsafe.driver.working.memory.in.mb | 60% of JVM Heap Memory | CarbonData supports storing data in unsafe on-heap memory in driver for certain operations like insert into, query for loading datamap cache. The Minimum value recommended is 512MB. | --- End diff -- Okay. I think parameter description has covered all the questions which are applicable ---
[GitHub] carbondata pull request #2752: [WIP]Even after carbon file is copied to targ...
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2752#discussion_r220142943
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -418,6 +423,24 @@ protected void closeExecutorService() throws
CarbonDataWriterException {
CarbonDataWriterException exception = null;
try {
listener.finish();
+ String tableName = this.model.getTableName();
+ // Delete CarbonData and CarbonIndex files after copying from temp
location
+ if (null != carbonDataFileTempPath && null ==
model.getTableSpec().getCarbonTable()
+ .getBucketingInfo(tableName)) {
+String deleteDataAndIndexInTempPath =
--- End diff --
Changed the code fix. Delete carbon index and carbon data files in temp
location, once it is copied from temp to target path
---
[GitHub] carbondata pull request #2756: [CARBONDATA-2966]Update Documentation For Avr...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2756 [CARBONDATA-2966]Update Documentation For Avro DataType conversion Updated document for Avro DataType conversion to carbon - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata doc_avro Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2756.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2756 commit f30d1489dbdbe3fecd9eacb96ddff3658904b691 Author: Indhumathi27 Date: 2018-09-24T18:04:04Z [CARBONDATA-2966]Update Documentation For Avro DataType conversion ---
[GitHub] carbondata pull request #2752: [WIP]Even after carbon file is copied to targ...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2752 [WIP]Even after carbon file is copied to targetfolder(local/hdfs), carbon files is not deleted from temp directory **Problem:** Even after carbon file is copied to targetfolder(local/hdfs), carbon files is not deleted from temp directory. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata deletetemp Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2752.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2752 commit d47beb2990791feb923215e4ab6bd8b8ef005068 Author: Indhumathi27 Date: 2018-09-24T06:58:47Z Even after carbon file is copied to targetfolder(local/hdfs), carbon files is not deleted from temp directory ---
[GitHub] carbondata issue #2745: [CARBONDATA-2958] Compaction with CarbonProperty 'ca...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2745 Retest this please ---
[GitHub] carbondata pull request #2745: [CARBONDATA-2958] Compaction with CarbonPrope...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2745 [CARBONDATA-2958] Compaction with CarbonProperty 'carbon.enable.page.level.reader.in.compaction' enabled fails as Compressor is null **Problem:** When CarbonProperty 'carbon.enable.page.level.reader.in.compaction' is enabled, compaction fails throwing Null Pointer Exception as compressor is Null **Solution:** Set compressor from pageMetaData - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Test case added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata page_compaction Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2745.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2745 commit 1c0a6fc8435c582cefa00bcd8d6979e89f1d886d Author: Indhumathi27 Date: 2018-09-21T09:54:39Z [CARBONDATA-2958] Compaction with CarbonProperty 'carbon.enable.page.level.reader.in.compaction' enabled fails as Compressor is null ---
[GitHub] carbondata pull request #2727: [CARBONDATA-2940]Fix BufferUnderFlowException...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2727 [CARBONDATA-2940]Fix BufferUnderFlowException for ComplexPushDown **Problem:** BufferUnderFlow Exception was thrown, as buffer data was filled with wrong values. **Solution:** Check for if column is of Primitive Dimension column with ComplexParentOrdinal = -1 and not Complex and increment Buffer index for NoDictionaryKeys. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Added testcase scenario - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata complex_bue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2727.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2727 commit 833f4b0158be0a3a643d5b862acc41bf612a31b8 Author: Indhumathi27 Date: 2018-09-17T07:02:49Z [CARBONDATA-2940]Fix BufferUnderFlowException for ComplexPushDown ---
[GitHub] carbondata issue #2694: [CARBONDATA-2876]AVRO datatype support through SDK
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2694 Retest this please ---
[GitHub] carbondata pull request #2694: [CARBONDATA-2876]AVRO datatype support throug...
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2694#discussion_r215500532
--- Diff:
store/sdk/src/main/java/org/apache/carbondata/sdk/file/AvroCarbonWriter.java ---
@@ -445,17 +445,13 @@ private static Field prepareFields(Schema.Field
avroField) {
if (logicalType instanceof LogicalTypes.Date) {
--- End diff --
Added a comment
---
[GitHub] carbondata pull request #2694: [CARBONDATA-2876]AVRO datatype support throug...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2694 [CARBONDATA-2876]AVRO datatype support through SDK This PR supports following Avro DataTypes to carbon format through SDK. Avro datatypes include, 1. Avro Logical type TimeMillis 2. Avro Logical type TimeMicros - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done test cases added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata avro_support_sdk1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2694.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2694 commit 2738b29b4cb3439fec40b80da0ec31776c5944a6 Author: Indhumathi27 Date: 2018-09-03T04:47:20Z [CARBONDATA-2876]AVRO datatype support through SDK ---
[GitHub] carbondata issue #2687: [CARBONDATA-2876]Fix Avro decimal datatype with prec...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2687 Retest this please ---
[GitHub] carbondata pull request #2687: [CARBONDATA-2876]Fix Avro decimal datatype wi...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2687 [CARBONDATA-2876]Fix Avro decimal datatype with precision and scale **What is PR for?** Add precision and scale for fieldvalue for Avro Decimal logical type - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata decimal_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2687.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2687 commit a6e5149c12a7427a00fdc634cd0d35aca077 Author: Indhumathi27 Date: 2018-09-03T12:05:01Z Fix decimal type for Avro ---
[GitHub] carbondata pull request #2671: [WIP]AVRO datatype support through SDK
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2671 [WIP]AVRO datatype support through SDK This PR supports following Avro DataTypes to carbon format through SDK. Avro datatypes include, 1. Avro Union 2. Avro Enum 3. Avro Logical type Decimal Please refer JIRA CARBONDATA-2876 for further detail. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Test file has been added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata avro_support_sdk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2671.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2671 commit eae87521707fa86337a45d5677cc7d0f1f1fbfbc Author: Indhumathi27 Date: 2018-08-29T14:18:21Z Support Avro datatype conversion through SDK ---
[GitHub] carbondata pull request #2621: [CARBONDATA-2840] Added SDV testcases for Com...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2621 [CARBONDATA-2840] Added SDV testcases for Complex DataType Support **Added SDV testcases for Complex DataType Support** - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata complexsdv Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2621.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2621 commit f9bd3e83720b6657654d38d96f1b3d07f9172192 Author: Indhumathi27 Date: 2018-08-08T10:03:36Z [CARBONDATA-2840] Added SDV testcases for Complex DataType Support ---
[GitHub] carbondata issue #2578: Fix Dictionary_Include for ComplexDataType
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2578 Retest this please ---
[GitHub] carbondata pull request #2578: Fix Dictionary_Include for ComplexDataType
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2578 Fix Dictionary_Include for ComplexDataType Why this PR for? Check if a column is given as Dictionary_Include and fill cardinality for that column - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata dict_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2578.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2578 commit bab79be62af7ffecb9dc1fcfe520fd0288c0281a Author: Indhumathi27 Date: 2018-07-30T08:48:44Z Fix Dictionary_Include for ComplexDataType ---
[GitHub] carbondata issue #2573: [CARBONDATA-2794]Distinct count fails on ArrayOfStru...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2573 Retest this please ---
[GitHub] carbondata pull request #2573: [CARBONDATA-2794]Distinct count fails on Arra...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2573 [CARBONDATA-2794]Distinct count fails on ArrayOfStruct This PR fixes Code Generator Error thrown when Select filter contains more than one count of distinct of ArrayofStruct with group by Clause - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Testcase added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata codegeb Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2573.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2573 commit e9e1cd6e7fcd9603bc35a74ad7981853502fef51 Author: Indhumathi27 Date: 2018-07-27T18:09:50Z [CARBONDATA-2794]Distinct count fails on ArrayOfStruct ---
[GitHub] carbondata issue #2569: [CARBONDATA-2791]Fix Encoding for Double if exceeds ...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2569 Retest this please ---
[GitHub] carbondata pull request #2569: [CARBONDATA-2791]Fix Adaptive Encoding for Do...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2569 [CARBONDATA-2791]Fix Adaptive Encoding for Double if exceeds LONG.Max_value **What this PR for?** If Factor(decimalcount) * absMaxValue exceeds LONG.MAX_VALUE, then go for direct compression. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Testcases added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata adapfix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2569.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2569 commit c26897b2d8d37ba4e5361da7585dfe6942c42f5a Author: Indhumathi27 Date: 2018-07-27T06:52:25Z [CARBONDATA-2791]Fix Adaptive Encoding for Double if exceeds LONG.Max_value ---
[GitHub] carbondata issue #2555: [CARBONDATA-2753][Compatibility] Row count of page i...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2555 retest sdv please ---
[GitHub] carbondata issue #2535: [CARBONDATA-2606]Fix Complex array Pushdown
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2535 Retest this please ---
[GitHub] carbondata issue #2535: [CARBONDATA-2606]Fix Complex array Pushdown
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2535 Retest this please ---
[GitHub] carbondata issue #2535: [CARBONDATA-2606]Fix Complex array Pushdown
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2535 Retest this please ---
[GitHub] carbondata issue #2535: [CARBONDATA-2606]Fix Complex array Pushdown
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2535 Retest this please ---
[GitHub] carbondata issue #2535: [CARBONDATA-2606]Fix Complex array Pushdown
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2535 Retest sdv please ---
[GitHub] carbondata issue #2535: [CARBONDATA-2606]Fix Complex array Pushdown
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2535 Retest this please ---
[GitHub] carbondata issue #2535: [CARBONDATA-2606]Fix Complex array Pushdown
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2535 Retest this please ---
[GitHub] carbondata issue #2535: [CARBONDATA-2606]Fix Complex array Pushdown
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2535 Retest this please ---
[GitHub] carbondata pull request #2535: [CARBONDATA-2606]Fix Complex array Pushdown
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2535 [CARBONDATA-2606]Fix Complex array Pushdown **What is PR for?** Check for if Complex Column contains ArrayType at n levels and add parent to projection if contains array. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata arrayfix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2535.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2535 commit e5b3d345d8054eaf1f326e70a2ab3453e9e1c153 Author: Indhumathi27 Date: 2018-07-21T10:46:21Z [CARBONDATA-2606]Fix Complex array Pushdown ---
[GitHub] carbondata issue #2532: [CARBONDATA-2759]Add Bad_Records_Options to STMPROPE...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2532 Retest this please ---
[GitHub] carbondata pull request #2532: [CARBONDATA-2759]Add Bad_Records_Options to S...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2532 [CARBONDATA-2759]Add Bad_Records_Options to STMPROPERTIES for Streaming Table Problem: Bad_Records_Options are not added for STMPROPERTIES for streaming table. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Testcase added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata streaming Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2532.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2532 commit 1b561c4ba7a570c5c6d1503d716c71cfe474a10b Author: Indhumathi27 Date: 2018-07-19T14:17:59Z [CARBONDATA-2759]Add Bad_Records_Options to STMPROPERTIES for Streaming Table ---
[GitHub] carbondata issue #2523: [CARBONDATA-2753] Fix Compatibility issue with Preag...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2523 Retest this please ---
[GitHub] carbondata pull request #2523: [CARBONDATA-2753] Fix Compatibility issue wit...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2523 [CARBONDATA-2753] Fix Compatibility issue with Preaggregate table Problem: User creates maintable and load data and create preaggregate datamap on maintable using old version and loads data into maintable using new version and dataload fails. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata preagg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2523.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2523 commit 7bfd652d3c1513206b0e5d719e68caccea6c2d47 Author: Indhumathi27 Date: 2018-07-18T15:17:06Z [CARBONDATA-2753] Fix Compatibility issue with Preaggregate table ---
[GitHub] carbondata issue #2501: [CARBONDATA-2738]Block Preaggregate, Compaction, Dic...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2501 Retest this please ---
[GitHub] carbondata issue #2502: [CARBONDATA-2738]Update documentation for Complex da...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2502 Retest this please ---
[GitHub] carbondata issue #2501: [CARBONDATA-2738]Block Preaggregate, Dictionary Excl...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2501 Retest this please ---
[GitHub] carbondata issue #2502: [CARBONDATA-2738]Update documentation for Complex da...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2502 Retest this please ---
[GitHub] carbondata issue #2489: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2489 Retest this please ---
[GitHub] carbondata pull request #2502: [CARBONDATA-2738]Update documentation for Com...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2502 [CARBONDATA-2738]Update documentation for Complex datatype For PR-2501, Update documentation for Complex datatype - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [x] Document update required? Added - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata complexdoc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2502.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2502 commit f36e97077f3852be4ac8851566cd5790709f7954 Author: Indhumathi27 Date: 2018-07-13T06:26:25Z [CARBONDATA-2738]Update documentation for Complex datatype ---
[GitHub] carbondata pull request #2501: [CARBONDATA-2738]Block Preaggregate, Dictiona...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2501 [CARBONDATA-2738]Block Preaggregate, Dictionary Exclude/Include for child columns for Complex datatype Block Preaggregate, Dictionary Exclude/Include for child columns for Complex datatype - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Test cases added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata orderby Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2501.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2501 commit c962b84bd88a3afe695245653d7bc3f14d89c3ce Author: Indhumathi27 Date: 2018-07-12T15:41:11Z [CARBONDATA-2738]Block features for complex data type ---
[GitHub] carbondata pull request #2489: [CARBONDATA-2606][Complex DataType Enhancemen...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2489 [CARBONDATA-2606][Complex DataType Enhancements]Fix Null result if First two Projection column have same parent and third column has different Parent Struct Problem: When multiple columns are there,then the first child elements is only going to make parent Object Array. For all other cases it should be null. For e.g. a : . here as 'a' is the parent column and b, c, d are child columns during traversal when we encounter the first element in list i.e. column 'b','a' will be completely filled. In case when column 'c' and 'd' encountered then only place null in the output. Hence, as Null is placed in the output, Select result is Null if First two Projection column have same parent and third column has different parent Struct column. Solution: Place null in the end of output - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Test cases added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata project2struct Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2489.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2489 commit be3008224f100d6e046987ab4c56792976d6c3a4 Author: Indhumathi27 Date: 2018-07-11T10:24:55Z [CARBONDATA-2606][Complex DataType Enhancements]Fix Null result when there are two Struct Column ---
[GitHub] carbondata issue #2439: [CARBONDATA-2606][Complex DataType Enhancements]Fixe...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2439 Retest this please ---
[GitHub] carbondata pull request #2442: [CARBONDATA-2684]Code Generator Error is thro...
Github user Indhumathi27 closed the pull request at: https://github.com/apache/carbondata/pull/2442 ---
[GitHub] carbondata issue #2439: [CARBONDATA-2606][Complex DataType Enhancements]Fixe...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2439 Retest sdv please ---
[GitHub] carbondata issue #2449: [CARBONDATA-2684] [PR-2442] Distinct count fails on ...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2449 Retest sdv please ---
[GitHub] carbondata issue #2439: [CARBONDATA-2606][Complex DataType Enhancements]Fixe...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2439 Retest this please ---
[GitHub] carbondata issue #2449: [CARBONDATA-2684] [PR-2442] Distinct count fails on ...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2449 Retest this please ---
[GitHub] carbondata issue #2449: [CARBONDATA-2684] [PR-2442] Distinct count fails on ...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2449 Retest sdv please ---
[GitHub] carbondata issue #2439: [CARBONDATA-2606][Complex DataType Enhancements]Fixe...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2439 Retest sdv please ---
[GitHub] carbondata issue #2439: [CARBONDATA-2606][Complex DataType Enhancements]Fixe...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2439 Retest this please ---
[GitHub] carbondata issue #2439: [CARBONDATA-2606][Complex DataType Enhancements]Fixe...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2439 Retest this please ---
[GitHub] carbondata pull request #2442: [CARBONDATA-2684]Code Generator Error is thro...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2442 [CARBONDATA-2684]Code Generator Error is thrown when Select filter contains more than one count of distinct of ComplexColumn with group by Clause This PR fixes Code Generator Error thrown when Select filter contains more than one count of distinct of ComplexColumn with group by Clause - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Testcase added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata complexarray Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2442.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2442 commit 492b60edbf8502c9f62f3f968926fa75f77504ed Author: Indhumathi27 Date: 2018-07-03T17:01:40Z Code Generator Error is thrown when Select filter contains more than one count of distinct of ComplexColumn with group by Clause ---
[GitHub] carbondata pull request #2439: [CARBONDATA-2606][Complex DataType Enhancemen...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2439 [CARBONDATA-2606][Complex DataType Enhancements]Fixed Projection Pushdown when Select filter contains Struct column **Problem:** If Select filter contains Struct Column which is not in Projection list, then only null value is stored for struct column given in filter and select query result is null. **Solution:** Pushdown Parent column of corresponding struct type if any struct column is present in Filter list. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Test Case Added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata filter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2439.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2439 commit 26df47ef07f6592919f0d9ccd5aa68703360de47 Author: Indhumathi27 Date: 2018-07-03T11:10:55Z Fixed Projection Pushdown when Select filter contains Struct column ---
[GitHub] carbondata issue #2421: [CARBONDATA-2606] [Complex DataType Enhancements]Fix...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2421 Retest this please ---
[GitHub] carbondata issue #2421: [CARBONDATA-2606] [Complex DataType Enhancements]Fix...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2421 Retest this please ---
[GitHub] carbondata issue #2421: [CARBONDATA-2606] [Complex DataType Enhancements]Fix...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2421 Retest sdv please ---
[GitHub] carbondata issue #2421: [CARBONDATA-2606] [Complex DataType Enhancements]Fix...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2421 Retest this please ---
[GitHub] carbondata issue #2421: [CARBONDATA-2606] [Complex DataType Enhancements]Fix...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2421 Retest this please ---
[GitHub] carbondata pull request #2421: [CARBONDATA-2606] [Complex DataType Enhanceme...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2421 [CARBONDATA-2606] [Complex DataType Enhancements]Fix for ComplexDataType Projection PushDown Fix for ComplexDataType Projection PushDown when Table Schema contains ColumnName in UpperCase - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata projectbug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2421.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2421 commit a043de47f6eee5e45e3c9ed84ebd53459609839f Author: Indhumathi27 Date: 2018-06-27T07:31:28Z Bug Fix For ComplexDataType Projection ---
[GitHub] carbondata issue #2396: [CARBONDATA-2606] [Complex DataType Enhancements] Pr...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2396 retest this please ---
[GitHub] carbondata issue #2396: [CARBONDATA-2606] [Complex DataType Enhancements] Pr...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2396 retest this please ---
[GitHub] carbondata pull request #2396: [CARBONDATA-2606] [Complex DataType Enhanceme...
Github user Indhumathi27 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2396#discussion_r197678648 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedResultCollector.java --- @@ -67,17 +69,38 @@ int noDictionaryColumnIndex; int complexTypeColumnIndex; + int noDictionaryComplexColumnIndex = 0; + int complexTypeComplexColumnIndex = 0; + boolean isDimensionExists; + private int[] surrogateResult; + private byte[][] noDictionaryKeys; + private byte[][] complexTypeKeyArray; + protected Map comlexDimensionInfoMap; + /** + * Field of this Map is the parent Column and associated child columns. + * Final Projection shuld be a merged list consist of only parents. + */ + public Map> mergedComplexDimensionColumns; + + /** + * Fields of this Map of Parent Ordinal with the List is the Child Column Dimension and + * the corresponding data buffer of that column. + */ + + public Map> mergedComplexDimensionDataMap; --- End diff -- It is already at class level and initialised only once and reused ---
[GitHub] carbondata pull request #2396: [CARBONDATA-2606] [Complex DataType Enhanceme...
Github user Indhumathi27 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2396#discussion_r197678288 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedResultCollector.java --- @@ -27,6 +28,7 @@ import org.apache.carbondata.core.keygenerator.directdictionary.DirectDictionaryKeyGeneratorFactory; import org.apache.carbondata.core.metadata.datatype.DataTypes; import org.apache.carbondata.core.metadata.encoder.Encoding; +import org.apache.carbondata.core.metadata.schema.table.column.CarbonDimension; --- End diff -- Dictionary will be handled with Adaptive encoding PR ---
[GitHub] carbondata pull request #2396: [CARBONDATA-2606] [Complex DataType Enhanceme...
Github user Indhumathi27 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2396#discussion_r197678273 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedResultCollector.java --- @@ -19,6 +19,7 @@ import java.nio.ByteBuffer; import java.util.ArrayList; import java.util.Arrays; +import java.util.HashMap; --- End diff -- VectorCollector is not required for now. It will be handled in next Complex data type enhancement PR ---
[GitHub] carbondata issue #2396: [CARBONDATA-2606] [Complex DataType Enhancements] Pr...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2396 retest this please ---
[GitHub] carbondata pull request #2396: [CARBONDATA-2606] [Complex DataType Enhanceme...
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2396#discussion_r197645303
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/complextypes/ArrayQueryType.java
---
@@ -97,4 +99,56 @@ public void
parseBlocksAndReturnComplexColumnByteArray(DimensionRawColumnChunk[]
return
DataTypeUtil.getDataTypeConverter().wrapWithGenericArrayData(data);
}
+ @Override public Object getDataBasedOnColumn(ByteBuffer dataBuffer,
CarbonDimension parent,
+ CarbonDimension child) {
+int dataLength;
+if (parent.getOrdinal() < child.getOrdinal()) {
+ dataLength = parent.getNumberOfChild();
+
+ if (dataLength == -1) {
+return null;
+ }
+ Object[] data = new Object[dataLength];
+ for (int i = 0; i < dataLength; i++) {
+data[i] = children
+.getDataBasedOnColumn(dataBuffer,
parent.getListOfChildDimensions().get(i), child);
+ }
+ return
DataTypeUtil.getDataTypeConverter().wrapWithGenericArrayData(data);
+} else if (parent.getOrdinal() > child.getOrdinal()) {
+ return null;
+} else {
+ // dataLength = dataBuffer.getInt();
+ return DataTypeUtil.getDataTypeConverter()
+ .wrapWithGenericArrayData(getDataBasedOnDataType(dataBuffer));
+}
+ }
+
+ @Override public Object getDataBasedOnColumnList(Map childBuffer,
--- End diff --
changed
---
[GitHub] carbondata pull request #2396: [CARBONDATA-2606] [Complex DataType Enhanceme...
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2396#discussion_r197645280
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDatasourceHadoopRelation.scala
---
@@ -67,14 +68,68 @@ case class CarbonDatasourceHadoopRelation(
override def schema: StructType =
tableSchema.getOrElse(carbonRelation.schema)
def buildScan(requiredColumns: Array[String],
+ projects: Seq[NamedExpression],
filters: Array[Filter],
partitions: Seq[PartitionSpec]): RDD[InternalRow] = {
val filterExpression: Option[Expression] = filters.flatMap { filter =>
CarbonFilters.createCarbonFilter(schema, filter)
}.reduceOption(new AndExpression(_, _))
+var parentColumns = new ListBuffer[String]
+// In case of Struct or StructofStruct Complex type, get the project
column for given
+// parent/child field and pushdown the corresponding project column.
In case of Array,
+// ArrayofStruct or StructofArray, pushdown parent column
+var reqColumns = projects.map {
+ case a@Alias(s: GetStructField, name) =>
+val arrayTypeExists = s.childSchema.map(x => x.dataType)
+ .filter(dataType => dataType.isInstanceOf[ArrayType])
+if (0 == arrayTypeExists.length) {
+ val columnName = s.toString().replaceAll("#[0-9]*", "")
+ parentColumns += columnName.split("\\.")(0)
+ columnName
+}
+else {
+ None
+}
+ case a@Alias(s: GetArrayItem, name) =>
+None
+ case other => other.name.replaceAll("#[0-9]*", "")
+}
+
+var reqCols = reqColumns.filterNot(none => none.equals(None)).map(col
=> col.toString)
+parentColumns = parentColumns.distinct
+reqCols = reqCols.distinct
+
+// if the parent column is there in the projection list then we can
filter out all the children
+// in that projection list
+val parentColumnOnProjectionList = reqCols.filter(col =>
parentColumns.contains(col))
--- End diff --
removed
---
