[GitHub] [carbondata] Indhumathi27 commented on pull request #3893: Added new property to set the value of executor LRU cache size to 70% of the total executor memory in IndexServer, if executor LRU c

2020-08-23 Thread GitBox


Indhumathi27 commented on pull request #3893:
URL: https://github.com/apache/carbondata/pull/3893#issuecomment-678918781


   @Karan980 Please create JIRA and update the PR description



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akashrn5 commented on a change in pull request #3862: [CARBONDATA-3933]Fix DDL/DML failures after table is created with column names having special characters like #,\,%

2020-08-23 Thread GitBox


akashrn5 commented on a change in pull request #3862:
URL: https://github.com/apache/carbondata/pull/3862#discussion_r475353691



##
File path: 
integration/spark/src/test/scala/org/apache/spark/carbondata/restructure/AlterTableRevertTestCase.scala
##
@@ -30,6 +32,13 @@ import 
org.apache.carbondata.spark.exception.ProcessMetaDataException
 class AlterTableRevertTestCase extends QueryTest with BeforeAndAfterAll {
 
   override def beforeAll() {
+new MockUp[MockClassForAlterRevertTests]() {

Review comment:
   yeah, i tried that, but since im using MockUp[T], it needs actual scala 
class name, not the object, as MockUp identifies class, so i followed this 
similarly by referring the `TableStatusBackupTest`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] QiangCai commented on a change in pull request #3862: [CARBONDATA-3933]Fix DDL/DML failures after table is created with column names having special characters like #,\,%

2020-08-23 Thread GitBox


QiangCai commented on a change in pull request #3862:
URL: https://github.com/apache/carbondata/pull/3862#discussion_r475347003



##
File path: 
integration/spark/src/test/scala/org/apache/spark/carbondata/restructure/AlterTableRevertTestCase.scala
##
@@ -30,6 +32,13 @@ import 
org.apache.carbondata.spark.exception.ProcessMetaDataException
 class AlterTableRevertTestCase extends QueryTest with BeforeAndAfterAll {
 
   override def beforeAll() {
+new MockUp[MockClassForAlterRevertTests]() {

Review comment:
   for scala, The class is CarbonSessionCatalogUtil$.  The method name is 
updateCatalogTableForAlter.  It is private, but not a static method.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #3896: [CARBONDATA-3955] Fix load failures due to daylight saving time changes

2020-08-23 Thread GitBox


ShreelekhyaG commented on a change in pull request #3896:
URL: https://github.com/apache/carbondata/pull/3896#discussion_r475346002



##
File path: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
##
@@ -1592,6 +1592,13 @@ private CarbonCommonConstants() {
 
   public static final String CARBON_LUCENE_INDEX_STOP_WORDS_DEFAULT = "false";
 
+  // Property to enable parsing the data with setLenient = true in load flow 
if it fails with

Review comment:
   Done





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Karan980 commented on pull request #3876: TestingCI

2020-08-23 Thread GitBox


Karan980 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-678905253


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akashrn5 commented on a change in pull request #3862: [CARBONDATA-3933]Fix DDL/DML failures after table is created with column names having special characters like #,\,%

2020-08-23 Thread GitBox


akashrn5 commented on a change in pull request #3862:
URL: https://github.com/apache/carbondata/pull/3862#discussion_r475336910



##
File path: 
integration/spark/src/test/scala/org/apache/spark/carbondata/restructure/AlterTableRevertTestCase.scala
##
@@ -30,6 +32,13 @@ import 
org.apache.carbondata.spark.exception.ProcessMetaDataException
 class AlterTableRevertTestCase extends QueryTest with BeforeAndAfterAll {
 
   override def beforeAll() {
+new MockUp[MockClassForAlterRevertTests]() {

Review comment:
   i checked for static methods, but all were defined in scala object not 
class, so i thought i can do this way.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akashrn5 commented on a change in pull request #3862: [CARBONDATA-3933]Fix DDL/DML failures after table is created with column names having special characters like #,\,%

2020-08-23 Thread GitBox


akashrn5 commented on a change in pull request #3862:
URL: https://github.com/apache/carbondata/pull/3862#discussion_r475336497



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonSessionCatalogUtil.scala
##
@@ -61,21 +61,6 @@ object CarbonSessionCatalogUtil {
   s"'dbName'='${ newTableIdentifier.database.get }', 'tablePath'='${ 
newTablePath }')")
   }
 
-  /**
-   * Below method will be used to update serde properties
-   * @param tableIdentifier table identifier
-   * @param schemaParts schema parts
-   * @param cols cols
-   */
-  def alterTable(tableIdentifier: TableIdentifier,
-  schemaParts: String,
-  cols: Option[Seq[ColumnSchema]],
-  sparkSession: SparkSession): Unit = {
-getClient(sparkSession)
-  .runSqlHive(s"ALTER TABLE 
`${tableIdentifier.database.get}`.`${tableIdentifier.table}` " +
-  s"SET TBLPROPERTIES($schemaParts)")

Review comment:
   yes, we are calling the API 
`org.apache.spark.sql.hive.CarbonSessionUtil#alterExternalCatalogForTableWithUpdatedSchema`
 which does for us. All the alter tests are passing if you see.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] QiangCai commented on a change in pull request #3862: [CARBONDATA-3933]Fix DDL/DML failures after table is created with column names having special characters like #,\,%

2020-08-23 Thread GitBox


QiangCai commented on a change in pull request #3862:
URL: https://github.com/apache/carbondata/pull/3862#discussion_r475301804



##
File path: 
integration/spark/src/test/scala/org/apache/spark/carbondata/restructure/AlterTableRevertTestCase.scala
##
@@ -30,6 +32,13 @@ import 
org.apache.carbondata.spark.exception.ProcessMetaDataException
 class AlterTableRevertTestCase extends QueryTest with BeforeAndAfterAll {
 
   override def beforeAll() {
+new MockUp[MockClassForAlterRevertTests]() {

Review comment:
   can we mock up the static method directly?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] QiangCai commented on a change in pull request #3862: [CARBONDATA-3933]Fix DDL/DML failures after table is created with column names having special characters like #,\,%

2020-08-23 Thread GitBox


QiangCai commented on a change in pull request #3862:
URL: https://github.com/apache/carbondata/pull/3862#discussion_r475298980



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonSessionCatalogUtil.scala
##
@@ -61,21 +61,6 @@ object CarbonSessionCatalogUtil {
   s"'dbName'='${ newTableIdentifier.database.get }', 'tablePath'='${ 
newTablePath }')")
   }
 
-  /**
-   * Below method will be used to update serde properties
-   * @param tableIdentifier table identifier
-   * @param schemaParts schema parts
-   * @param cols cols
-   */
-  def alterTable(tableIdentifier: TableIdentifier,
-  schemaParts: String,
-  cols: Option[Seq[ColumnSchema]],
-  sparkSession: SparkSession): Unit = {
-getClient(sparkSession)
-  .runSqlHive(s"ALTER TABLE 
`${tableIdentifier.database.get}`.`${tableIdentifier.table}` " +
-  s"SET TBLPROPERTIES($schemaParts)")

Review comment:
   if we do not change it, will spark/hive update it?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-23 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-678825751


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2101/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-23 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-678824999


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3842/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3862: [CARBONDATA-3933]Fix DDL/DML failures after table is created with column names having special characters like #,\,%

2020-08-23 Thread GitBox


CarbonDataQA1 commented on pull request #3862:
URL: https://github.com/apache/carbondata/pull/3862#issuecomment-678814227


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2100/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3862: [CARBONDATA-3933]Fix DDL/DML failures after table is created with column names having special characters like #,\,%

2020-08-23 Thread GitBox


CarbonDataQA1 commented on pull request #3862:
URL: https://github.com/apache/carbondata/pull/3862#issuecomment-678813779


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3841/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Karan980 commented on pull request #3876: TestingCI

2020-08-23 Thread GitBox


Karan980 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-678812610


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-23 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-678812332


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2099/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-23 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-678811731


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3840/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3896: [CARBONDATA-3955] Fix load failures due to daylight saving time changes

2020-08-23 Thread GitBox


CarbonDataQA1 commented on pull request #3896:
URL: https://github.com/apache/carbondata/pull/3896#issuecomment-678811108







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] kunal642 commented on a change in pull request #3834: [CARBONDATA-3865] Implementation of delete/update feature in carbondata SDK.

2020-08-23 Thread GitBox


kunal642 commented on a change in pull request #3834:
URL: https://github.com/apache/carbondata/pull/3834#discussion_r475244586



##
File path: sdk/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonIUD.java
##
@@ -0,0 +1,275 @@
+package org.apache.carbondata.sdk.file;
+
+import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.Field;
+import org.apache.carbondata.core.scan.expression.ColumnExpression;
+import org.apache.carbondata.core.scan.expression.Expression;
+import org.apache.carbondata.core.scan.expression.LiteralExpression;
+import 
org.apache.carbondata.core.scan.expression.conditional.EqualToExpression;
+import org.apache.carbondata.core.scan.expression.logical.AndExpression;
+import org.apache.carbondata.hadoop.api.CarbonTableOutputFormat;
+import org.apache.carbondata.hadoop.internal.ObjectArrayWritable;
+
+import java.io.File;
+import java.io.FilenameFilter;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.jcraft.jsch.IO;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.RecordWriter;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
+
+public class CarbonIUD {
+
+  private final Map> filterColumnToValueMapping;
+  private final Map> updateColumnToValueMapping;
+
+  private CarbonIUD() {
+filterColumnToValueMapping = new HashMap<>();
+updateColumnToValueMapping = new HashMap<>();
+  }
+
+  /**
+   * @return CarbonIUD object
+   */
+  public static CarbonIUD getInstance() {
+return new CarbonIUD();
+  }
+
+  /**
+   * @param path   is the segment path on which delete is performed
+   * @param column is the columeName on which records have to be deleted
+   * @param value  of column on which the records have to be deleted
+   * @return CarbonIUD object
+   * 
+   * for eg: DELETE WHERE column = value
+   */
+  public CarbonIUD delete(String path, String column, String value) {
+prepareDelete(path, column, value);
+return this;
+  }
+
+  /**
+   * @param path is the segment path on which delete is performed
+   * @param filterExpression is the expression to delete the records
+   * @throws IOException
+   * @throws InterruptedException
+   */
+  public void delete(String path, Expression filterExpression)
+  throws IOException, InterruptedException {
+CarbonReader reader = CarbonReader.builder(path, "_temp")
+.projection(new String[] { 
CarbonCommonConstants.CARBON_IMPLICIT_COLUMN_TUPLEID })
+.filter(filterExpression).build();
+
+RecordWriter deleteDeltaWriter =
+CarbonTableOutputFormat.getDeleteDeltaRecordWriter(path);
+ObjectArrayWritable writable = new ObjectArrayWritable();
+while (reader.hasNext()) {
+  Object[] row = (Object[]) reader.readNextRow();
+  writable.set(row);
+  deleteDeltaWriter.write(NullWritable.get(), writable);
+}
+deleteDeltaWriter.close(null);
+reader.close();
+  }
+
+  /**
+   * Calling this method will start the execution of delete process
+   * @throws IOException
+   * @throws InterruptedException
+   */
+  public void closeDelete() throws IOException, InterruptedException {
+for (Map.Entry> path : 
this.filterColumnToValueMapping.entrySet()) {
+  deleteExecution(path.getKey());
+}
+this.filterColumnToValueMapping.clear();
+  }
+
+  /**
+   * @param path  is the segment path on which update is performed
+   * @param columnis the columeName on which records have to be updated
+   * @param value of column on which the records have to be updated
+   * @param updColumn is the name of updatedColumn
+   * @param updValue  is the value of updatedColumn
+   * @return CarbonUID
+   * 
+   * for eg: UPDATE updColumn = updValue WHERE column = value
+   */
+  public CarbonIUD update(String path, String column, String value, String 
updColumn,
+  String updValue) {
+prepareUpdate(path, column, value, updColumn, updValue);
+return this;
+  }
+
+  /**
+   * @param pathis the segment path on which update is 
performed.
+   * @param filterExpressionis the expression object to update the 
records
+   * @param updatedColumnToValueMapping contains the mapping of updatedColumns 
to updatedValues
+   * @throws IOException
+   * @throws InterruptedException
+   * @throws InvalidLoadOptionException
+   */
+  public void update(String path, Expression filterExpression,
+  Map updatedColumnToValueMapping)
+  throws IOException, 

[GitHub] [carbondata] kunal642 commented on pull request #3862: [CARBONDATA-3933]Fix DDL/DML failures after table is created with column names having special characters like #,\,%

2020-08-23 Thread GitBox


kunal642 commented on pull request #3862:
URL: https://github.com/apache/carbondata/pull/3862#issuecomment-678801642


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Resolved] (CARBONDATA-3946) Support IndexServer with Presto Engine

2020-08-23 Thread Kunal Kapoor (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Kapoor resolved CARBONDATA-3946.
--
Fix Version/s: 2.1.0
   Resolution: Fixed

> Support IndexServer with Presto Engine
> --
>
> Key: CARBONDATA-3946
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3946
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Indhumathi Muthumurugesh
>Priority: Major
> Fix For: 2.1.0
>
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [carbondata] asfgit closed pull request #3885: [CARBONDATA-3946] Support IndexServer with Presto Engine

2020-08-23 Thread GitBox


asfgit closed pull request #3885:
URL: https://github.com/apache/carbondata/pull/3885


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] kunal642 commented on pull request #3885: [CARBONDATA-3946] Support IndexServer with Presto Engine

2020-08-23 Thread GitBox


kunal642 commented on pull request #3885:
URL: https://github.com/apache/carbondata/pull/3885#issuecomment-678801176


   LGTM



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Karan980 commented on pull request #3876: TestingCI

2020-08-23 Thread GitBox


Karan980 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-678799256


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3885: [CARBONDATA-3946] Support IndexServer with Presto Engine

2020-08-23 Thread GitBox


CarbonDataQA1 commented on pull request #3885:
URL: https://github.com/apache/carbondata/pull/3885#issuecomment-678798742


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2097/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3885: [CARBONDATA-3946] Support IndexServer with Presto Engine

2020-08-23 Thread GitBox


CarbonDataQA1 commented on pull request #3885:
URL: https://github.com/apache/carbondata/pull/3885#issuecomment-678798596


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3838/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] kunal642 commented on pull request #3885: [CARBONDATA-3946] Support IndexServer with Presto Engine

2020-08-23 Thread GitBox


kunal642 commented on pull request #3885:
URL: https://github.com/apache/carbondata/pull/3885#issuecomment-678782325


   please rebase



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org