[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3858: [CARBONDATA-3919] Improve concurrent query performance

2020-08-09 Thread GitBox


ajantha-bhat commented on a change in pull request #3858:
URL: https://github.com/apache/carbondata/pull/3858#discussion_r467696850



##
File path: 
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
##
@@ -472,9 +471,6 @@ public IndexFilter getFilterPredicates(Configuration 
configuration) {
 QueryStatisticsRecorder recorder = 
CarbonTimeStatisticsFactory.createDriverRecorder();
 QueryStatistic statistic = new QueryStatistic();
 
-// get tokens for all the required FileSystem for table path

Review comment:
   Also removed and tested in user environment, no issues observed





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on pull request #3858: [CARBONDATA-3919] Improve concurrent query performance

2020-08-09 Thread GitBox


ajantha-bhat commented on pull request #3858:
URL: https://github.com/apache/carbondata/pull/3858#issuecomment-671171540


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Karan980 commented on pull request #3876: TestingCI

2020-08-09 Thread GitBox


Karan980 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-671170769


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Created] (CARBONDATA-3948) Bloom index fails to create when index server goes to fallback mode

2020-08-09 Thread Vikram Ahuja (Jira)
Vikram Ahuja created CARBONDATA-3948:


 Summary: Bloom index fails to create when index server goes to 
fallback mode
 Key: CARBONDATA-3948
 URL: https://issues.apache.org/jira/browse/CARBONDATA-3948
 Project: CarbonData
  Issue Type: Bug
  Components: spark-integration
Reporter: Vikram Ahuja


When index server goes to fallback mode and if create bloom index is triggered, 
create bloom index fails



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3775: [WIP] Bloom Index Server Fallback Issue

2020-08-09 Thread GitBox


vikramahuja1001 commented on a change in pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775#discussion_r467692421



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/index/CarbonCreateIndexCommand.scala
##
@@ -238,11 +238,22 @@ case class CarbonCreateIndexCommand(
   false)
 val enabledIndexInfo = IndexTableInfo.enableIndex(indexInfo, 
indexModel.indexName)
 
-// set index information in parent table
-val parentIndexMetadata = parentTable.getIndexMetadata
-parentIndexMetadata.updateIndexStatus(indexProviderName,
-  indexModel.indexName,
-  IndexStatus.ENABLED.name())
+// set index information in parent table. Create it if it is null.
+val parentIndexMetadata = if (
+  parentTable.getTableInfo.getFactTable.getTableProperties
+.get(parentTable.getCarbonTableIdentifier.getTableId) != null) 
{
+  val tempIndexMetaData = parentTable.getIndexMetadata
+  tempIndexMetaData.updateIndexStatus(indexProviderName,

Review comment:
   done, please check again!





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3775: [WIP] Bloom Index Server Fallback Issue

2020-08-09 Thread GitBox


vikramahuja1001 commented on a change in pull request #3775:
URL: https://github.com/apache/carbondata/pull/3775#discussion_r467689041



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/index/CarbonCreateIndexCommand.scala
##
@@ -238,11 +238,22 @@ case class CarbonCreateIndexCommand(
   false)
 val enabledIndexInfo = IndexTableInfo.enableIndex(indexInfo, 
indexModel.indexName)
 
-// set index information in parent table
-val parentIndexMetadata = parentTable.getIndexMetadata
-parentIndexMetadata.updateIndexStatus(indexProviderName,
-  indexModel.indexName,
-  IndexStatus.ENABLED.name())
+// set index information in parent table. Create it if it is null.
+val parentIndexMetadata = if (
+  parentTable.getTableInfo.getFactTable.getTableProperties
+.get(parentTable.getCarbonTableIdentifier.getTableId) != null) 
{

Review comment:
   @kevinjmh , in the case of spark-sql, the same jvm is used between 
driver and executor thus in fallback mode when cache is cleared in executor 
side, the serialized metadata information is also removed from the driver side.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] asfgit closed pull request #3883: [CARBONDATA-3940] CommitTask fails due to Rename IOException during L…

2020-08-09 Thread GitBox


asfgit closed pull request #3883:
URL: https://github.com/apache/carbondata/pull/3883


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] QiangCai commented on pull request #3883: [CARBONDATA-3940] CommitTask fails due to Rename IOException during L…

2020-08-09 Thread GitBox


QiangCai commented on pull request #3883:
URL: https://github.com/apache/carbondata/pull/3883#issuecomment-671131477


   LGTM



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto array columns read support

2020-08-09 Thread GitBox


CarbonDataQA1 commented on pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#issuecomment-671129567


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1930/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto array columns read support

2020-08-09 Thread GitBox


CarbonDataQA1 commented on pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#issuecomment-671129181


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3669/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akkio-97 commented on a change in pull request #3773: [CARBONDATA-3830]Presto array columns read support

2020-08-09 Thread GitBox


akkio-97 commented on a change in pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#discussion_r467639051



##
File path: 
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/compress/DirectCompressCodec.java
##
@@ -246,7 +239,29 @@ public void decodeAndFillVector(byte[] pageData, 
ColumnVectorInfo vectorInfo, Bi
   vector = ColumnarVectorWrapperDirectFactory
   .getDirectVectorWrapperFactory(vector, vectorInfo.invertedIndex, 
nullBits, deletedRows,
   true, false);
-  fillVector(pageData, vector, vectorDataType, pageDataType, pageSize, 
vectorInfo, nullBits);
+  Deque vectorStack = vectorInfo.getVectorStack();
+  // Only if vectorStack is null, it is initialized with the parent vector
+  if (vectorStack == null && vectorInfo.vector.getColumnVector() != null) {
+vectorStack = new ArrayDeque<>();
+// pushing the parent vector
+vectorStack.push((CarbonColumnVectorImpl) 
vectorInfo.vector.getColumnVector());
+vectorInfo.setVectorStack(vectorStack);
+  }
+  /*
+   * if top of vector stack is a complex vector then
+   * add their children into the stack and load them too.
+   * TODO: If there are multiple children push them into stack and load 
them iteratively
+   */
+  if (vectorStack != null && vectorStack.peek().isComplex()) {
+vectorStack.peek().setChildrenElements(pageData);

Review comment:
   done

##
File path: 
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/compress/DirectCompressCodec.java
##
@@ -246,7 +239,29 @@ public void decodeAndFillVector(byte[] pageData, 
ColumnVectorInfo vectorInfo, Bi
   vector = ColumnarVectorWrapperDirectFactory
   .getDirectVectorWrapperFactory(vector, vectorInfo.invertedIndex, 
nullBits, deletedRows,
   true, false);
-  fillVector(pageData, vector, vectorDataType, pageDataType, pageSize, 
vectorInfo, nullBits);
+  Deque vectorStack = vectorInfo.getVectorStack();
+  // Only if vectorStack is null, it is initialized with the parent vector
+  if (vectorStack == null && vectorInfo.vector.getColumnVector() != null) {
+vectorStack = new ArrayDeque<>();
+// pushing the parent vector
+vectorStack.push((CarbonColumnVectorImpl) 
vectorInfo.vector.getColumnVector());
+vectorInfo.setVectorStack(vectorStack);
+  }
+  /*
+   * if top of vector stack is a complex vector then
+   * add their children into the stack and load them too.
+   * TODO: If there are multiple children push them into stack and load 
them iteratively
+   */
+  if (vectorStack != null && vectorStack.peek().isComplex()) {
+vectorStack.peek().setChildrenElements(pageData);
+vectorStack.push(vectorStack.peek().getChildrenVector().get(0));
+vectorStack.peek().loadPage();
+return;
+  }
+
+  FillVector fill = new FillVector(pageData, vectorInfo, nullBits);
+  fill.basedOnType(vector, vectorDataType, pageSize, pageDataType);
+

Review comment:
   done

##
File path: 
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/FillVector.java
##
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.BitSet;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.DecimalConverterFactory;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.ColumnVectorInfo;
+import 
org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+import org.apache.carbondata.core.util.ByteUtil;
+
+public class FillVector {
+  private byte[] pageData;
+  private float floatFactor = 0;
+  private double factor = 0;
+  private ColumnVectorInfo vectorInfo;
+  private BitSet nullBits;
+
+  public FillVector(byte[] pageData, ColumnVectorInfo vectorInfo, 

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.

2020-08-09 Thread GitBox


CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-671090835


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1929/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.

2020-08-09 Thread GitBox


CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-671090661


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3668/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Created] (CARBONDATA-3947) Insert Into Select Operation is throwing exception for hive read/write operation in carbon.

2020-08-09 Thread Nihal kumar ojha (Jira)
Nihal kumar ojha created CARBONDATA-3947:


 Summary: Insert Into Select Operation is throwing exception for 
hive read/write operation in carbon.
 Key: CARBONDATA-3947
 URL: https://issues.apache.org/jira/browse/CARBONDATA-3947
 Project: CarbonData
  Issue Type: Bug
Reporter: Nihal kumar ojha


CREATE TABLE hive_carbon_table1(id INT, name STRING, scale DECIMAL, country 
STRING, salary DOUBLE) stored by 
'org.apache.carbondata.hive.CarbonStorageHandler';

INSERT into hive_carbon_table1 SELECT 1, 'RAM', '2.3', 'INDIA', 3500";
   
CREATE TABLE hive_carbon_table2(id INT, name STRING, scale DECIMAL, country 
STRING, salary DOUBLE) stored by 
'org.apache.carbondata.hive.CarbonStorageHandler';

INSERT into hive_carbon_table2 SELECT * FROM hive_carbon_table1";   -> Throwing 
exception as "CarbonData file is not present in the table location"




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [carbondata] asfgit closed pull request #3881: [CARBONDATA-3945] NPE While Data Loading

2020-08-09 Thread GitBox


asfgit closed pull request #3881:
URL: https://github.com/apache/carbondata/pull/3881


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-09 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-671034505


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1928/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-09 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-671034377


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3667/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] QiangCai commented on pull request #3881: [CARBONDATA-3945] NPE While Data Loading

2020-08-09 Thread GitBox


QiangCai commented on pull request #3881:
URL: https://github.com/apache/carbondata/pull/3881#issuecomment-671033903


   LGTM



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Karan980 commented on pull request #3876: TestingCI

2020-08-09 Thread GitBox


Karan980 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-671020621


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org