[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


ajantha-bhat commented on a change in pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#discussion_r509008144



##
File path: 
integration/presto/src/test/prestodb/org/apache/carbondata/presto/server/PrestoTestUtil.scala
##
@@ -114,4 +114,60 @@ object PrestoTestUtil {
   }
 }
   }
+
+  // this method depends on prestodb jdbc PrestoArray class
+  def validateArrayOfPrimitiveTypeDataWithLocalDict(actualResult: 
List[Map[String, Any]],

Review comment:
   Add it for prestosql also and compile for both prestodb and prestosql 
and run the testcase locally once with both the profile





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-713320386


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2807/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


ajantha-bhat commented on a change in pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#discussion_r509004845



##
File path: 
core/src/main/java/org/apache/carbondata/core/scan/result/vector/impl/CarbonColumnVectorImpl.java
##
@@ -81,6 +82,8 @@
 
   private List childElementsForEachRow;
 
+  public DimensionRawColumnChunk rawColumnChunk;

Review comment:
   please make it private. and add getter and setter





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


ajantha-bhat commented on a change in pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#discussion_r509004084



##
File path: 
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/compress/DirectCompressCodec.java
##
@@ -325,6 +327,21 @@ private void fillPrimitiveType(byte[] pageData, 
CarbonColumnVector vector,
   int intSizeInBytes = DataTypes.INT.getSizeInBytes();
   int shortSizeInBytes = DataTypes.SHORT.getSizeInBytes();
   int lengthStoredInBytes;
+  // check if local dictionary is enabled for complex primitve type and 
call
+  // fillVector eventually
+  if (!vectorInfo.vectorStack.isEmpty()) {
+CarbonColumnVectorImpl tempVector =
+(CarbonColumnVectorImpl) 
(vectorInfo.vectorStack.peek().getColumnVector());
+if (tempVector.rawColumnChunk != null
+&& tempVector.rawColumnChunk.getLocalDictionary() != null) {
+  DimensionChunkStoreFactory.DimensionStoreType dimStoreType =
+  DimensionChunkStoreFactory.DimensionStoreType.LOCAL_DICT;
+  new VariableLengthDimensionColumnPage(pageData, new int[0], new 
int[0], pageSize,

Review comment:
   please move fill vector comment here, because 
VariableLengthDimensionColumnPage constructor is calling fill vector





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


ajantha-bhat commented on a change in pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#discussion_r509002285



##
File path: 
core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/LocalDictDimensionDataChunkStore.java
##
@@ -64,6 +66,14 @@ public void fillVector(int[] invertedIndex, int[] 
invertedIndexReverse, byte[] d
 int columnValueSize = dimensionDataChunkStore.getColumnValueSize();
 int rowsNum = dataLength / columnValueSize;
 CarbonColumnVector vector = vectorInfo.vector;
+if (vector.getType().isComplexType()) {
+  vector = vectorInfo.vectorStack.peek();

Review comment:
   directly call `getDirectVectorWrapperFactory`, it has 
vector.getType().isComplexType() inside only, other places like AdaptiveCodec 
uses the `getDirectVectorWrapperFactory`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


ajantha-bhat commented on a change in pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#discussion_r509001513



##
File path: 
core/src/main/java/org/apache/carbondata/core/datastore/chunk/reader/dimension/v3/DimensionChunkReaderV3.java
##
@@ -296,6 +297,12 @@ protected DimensionColumnPage 
decodeDimension(DimensionRawColumnChunk rawColumnP
 }
   }
   BitSet nullBitSet = QueryUtil.getNullBitSet(pageMetadata.presence, 
this.compressor);
+  // store rawColumnChunk for local dictionary
+  if (vectorInfo != null && !vectorInfo.vectorStack.isEmpty()) {

Review comment:
   please check if is local dictionary is present, then only store the 
rawColumnChunk





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Karan980 commented on a change in pull request #3979: [Carbondata-3954] Fix insertion from ORC table into carbon table when sort scope is global sort

2020-10-20 Thread GitBox


Karan980 commented on a change in pull request #3979:
URL: https://github.com/apache/carbondata/pull/3979#discussion_r509000503



##
File path: 
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/allqueries/InsertIntoCarbonTableTestCase.scala
##
@@ -67,6 +67,21 @@ class InsertIntoCarbonTableTestCase extends QueryTest with 
BeforeAndAfterAll {
 
   }
 
+  test("insert from orc-select columns with columns having null values and 
sort scope as global sort") {
+sql("drop table if exists TORCSource")
+sql("drop table if exists TCarbon")
+sql("create table TORCSource(name string,col array,fee int) STORED 
AS orc")
+sql("insert into TORCSource values('karan',null,2)")
+sql("create table TCarbon(name string, col array,fee int) STORED 
AS carbondata TBLPROPERTIES 
('SORT_COLUMNS'='name','TABLE_BLOCKSIZE'='128','TABLE_BLOCKLET_SIZE'='128','SORT_SCOPE'='global_SORT')")
+sql("insert overwrite table TCarbon select name,col,fee from TORCSource")
+val result = sql("show segments for table 
TCarbon").collect()(0).get(1).toString()
+if(!"Success".equalsIgnoreCase(result)) {

Review comment:
   Done

##
File path: 
integration/spark/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##
@@ -1011,38 +1011,46 @@ object CommonUtil {
   objectDataType: DataType): AnyRef = {
 objectDataType match {
   case _: ArrayType =>
-val arrayDataType = objectDataType.asInstanceOf[ArrayType]
-val arrayData = data.asInstanceOf[UnsafeArrayData]
-val size = arrayData.numElements()
-val childDataType = arrayDataType.elementType
-val arrayChildObjects = new Array[AnyRef](size)
-var i = 0
-while (i < size) {
-  arrayChildObjects(i) = 
convertSparkComplexTypeToCarbonObject(arrayData.get(i,
-childDataType), childDataType)
-  i = i + 1
+if (data == null) {

Review comment:
   Done

##
File path: 
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/allqueries/InsertIntoCarbonTableTestCase.scala
##
@@ -67,6 +67,21 @@ class InsertIntoCarbonTableTestCase extends QueryTest with 
BeforeAndAfterAll {
 
   }
 
+  test("insert from orc-select columns with columns having null values and 
sort scope as global sort") {
+sql("drop table if exists TORCSource")
+sql("drop table if exists TCarbon")
+sql("create table TORCSource(name string,col array,fee int) STORED 
AS orc")
+sql("insert into TORCSource values('karan',null,2)")
+sql("create table TCarbon(name string, col array,fee int) STORED 
AS carbondata TBLPROPERTIES 
('SORT_COLUMNS'='name','TABLE_BLOCKSIZE'='128','TABLE_BLOCKLET_SIZE'='128','SORT_SCOPE'='global_SORT')")
+sql("insert overwrite table TCarbon select name,col,fee from TORCSource")
+val result = sql("show segments for table 
TCarbon").collect()(0).get(1).toString()
+if(!"Success".equalsIgnoreCase(result)) {
+  assert(false)
+}
+sql("drop table if exists TORCSource")
+sql("drop table if exists TCarbon")
+  }
+

Review comment:
   Done





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


ajantha-bhat commented on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-713310868


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akkio-97 commented on pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


akkio-97 commented on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-713309055


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akkio-97 removed a comment on pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


akkio-97 removed a comment on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-713309055


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Karan980 commented on a change in pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK

2020-10-20 Thread GitBox


Karan980 commented on a change in pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#discussion_r508994112



##
File path: 
sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java
##
@@ -1260,8 +1260,8 @@ public int compare(Field o1, Field o2) {
   Assert.assertEquals((double) id / 2, row[4]);
   Assert.assertEquals(true, (boolean) row[5]);
   long day = 24L * 3600 * 1000;

Review comment:
   Done

##
File path: 
sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonIUDTest.java
##
@@ -72,6 +74,48 @@ public void testDelete() throws Exception {
 FileUtils.deleteDirectory(new File(path));
   }
 
+  @Test
+  public void testUpdateOnDateType() throws Exception {
+String path = "./testWriteFiles";
+CarbonProperties.getInstance()
+.addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+.addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT);
+FileUtils.deleteDirectory(new File(path));
+Field[] fields = new Field[3];
+fields[0] = new Field("intField", DataTypes.INT);
+fields[1] = new Field("dateField", DataTypes.DATE);
+fields[2] = new Field("timeField", DataTypes.TIMESTAMP);
+CarbonWriter writer = CarbonWriter.builder()
+.outputPath(path)
+.withCsvInput(new Schema(fields))
+.writtenBy("IUDTest")
+.build();
+for (int i = 0; i < 10; i++) {
+  String[] row2 = new String[]{
+  String.valueOf(i % 1),
+  "2019-03-02",
+  "2019-02-12 03:03:34",
+  };
+  writer.write(row2);
+}
+writer.close();
+CarbonIUD.getInstance().update(path, "intField", "0", "intField", 
"20").commit();
+CarbonReader reader =
+CarbonReader.builder(path).projection(new String[] { "intField", 
"dateField", "timeField" })
+.build();
+int i = 0;
+while (reader.hasNext()) {
+  Object[] row = (Object[]) reader.readNextRow();
+  assert ((int) row[0] != 0);

Review comment:
   Done

##
File path: 
sdk/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReader.java
##
@@ -109,7 +116,39 @@ public boolean hasNext() throws IOException, 
InterruptedException {
*/
   public T readNextRow() throws IOException, InterruptedException {
 validateReader();
-return currentReader.getCurrentValue();
+return formatDateAndTimeStamp((Object[]) currentReader.getCurrentValue());
+  }
+
+  public T formatDateAndTimeStamp(Object[] row) {
+List dimensions = ((AbstractRecordReader) 
currentReader)
+.getQueryModel().getProjectionDimensions();
+String carbonDateFormat = CarbonProperties.getInstance()
+.getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT);
+if (carbonDateFormat == null) {
+  carbonDateFormat = CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT;
+}
+SimpleDateFormat dateFormat = new SimpleDateFormat(carbonDateFormat);
+String carbonTimeStampFormat = CarbonProperties.getInstance()
+.getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT);
+if (carbonTimeStampFormat  == null) {
+  carbonTimeStampFormat  = 
CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT;
+}
+SimpleDateFormat timeStampFormat = new 
SimpleDateFormat(carbonTimeStampFormat);
+for (ProjectionDimension dimension : dimensions) {
+  ColumnSchema columnSchema = dimension.getDimension().getColumnSchema();
+  if (columnSchema == null) {
+continue;
+  }
+  DataType dataType = columnSchema.getDataType();
+  if (dataType == DataTypes.DATE) {
+row[dimension.getOrdinal()] = dateFormat
+.format(new Date(24L * 3600 * 1000 * 
(int)row[dimension.getOrdinal()]));

Review comment:
   Done





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filter

2020-10-20 Thread GitBox


ajantha-bhat commented on pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#issuecomment-713307212


   @kunal642 : PR is ready. Please check and merge.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3875:
URL: https://github.com/apache/carbondata/pull/3875#issuecomment-713305605


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4562/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3979: [Carbondata-3954] Fix insertion from ORC table into carbon table when sort scope is global sort

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3979:
URL: https://github.com/apache/carbondata/pull/3979#issuecomment-713303716


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4558/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Resolved] (CARBONDATA-4036) When the ` character is present in column name, the table creation fails

2020-10-20 Thread Kunal Kapoor (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Kapoor resolved CARBONDATA-4036.
--
Fix Version/s: 2.1.0
   Resolution: Fixed

> When the ` character is present in column name, the table creation fails
> 
>
> Key: CARBONDATA-4036
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4036
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Akash R Nilugal
>Assignee: Akash R Nilugal
>Priority: Minor
> Fix For: 2.1.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> When the ` character is present in column name, the table creation fails
> sql("create table special_char(`i#d` string, `nam(e` 

[GitHub] [carbondata] asfgit closed pull request #3983: [CARBONDATA-4036]Fix special char(`) issue in create table, when column name contains ` character

2020-10-20 Thread GitBox


asfgit closed pull request #3983:
URL: https://github.com/apache/carbondata/pull/3983


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-713302522


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4557/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3917: [CARBONDATA-3978] Clean Files Refactor and support for trash folder in carbondata

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3917:
URL: https://github.com/apache/carbondata/pull/3917#issuecomment-713302639


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4560/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3875:
URL: https://github.com/apache/carbondata/pull/3875#issuecomment-713302289


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2806/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#issuecomment-713301190


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4559/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] kunal642 commented on pull request #3983: [CARBONDATA-4036]Fix special char(`) issue in create table, when column name contains ` character

2020-10-20 Thread GitBox


kunal642 commented on pull request #3983:
URL: https://github.com/apache/carbondata/pull/3983#issuecomment-713299988


   LGTM



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] kunal642 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory

2020-10-20 Thread GitBox


kunal642 commented on pull request #3974:
URL: https://github.com/apache/carbondata/pull/3974#issuecomment-713299839


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filter

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#issuecomment-713298999


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2805/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-713298067


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2804/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-713276469


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4552/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-713272161


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4554/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-713265288


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2801/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#issuecomment-713265280


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2803/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#issuecomment-713264301


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4555/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filter

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#issuecomment-713264283


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4556/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] QiangCai commented on pull request #3972: [WIP]Launch same number of task as select query for insert into select and ctas cases when target table is of no_sort

2020-10-20 Thread GitBox


QiangCai commented on pull request #3972:
URL: https://github.com/apache/carbondata/pull/3972#issuecomment-713264026


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3977:
URL: https://github.com/apache/carbondata/pull/3977#issuecomment-713263471


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2802/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3977:
URL: https://github.com/apache/carbondata/pull/3977#issuecomment-713259905


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4553/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] shenjiayu17 commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-20 Thread GitBox


shenjiayu17 commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r508943092



##
File path: 
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
##
@@ -415,44 +415,66 @@ public boolean accept(CarbonFile pathName) {
   }
 
   /**
-   * Return all delta file for a block.
-   * @param segmentId
-   * @param blockName
-   * @return
+   * Get all delete delta files mapped to each block of the specified segment.
+   * First list all deletedelta files in the segment dir, then loop the files 
and find
+   * a map of blocks and .deletedelta files related to each block.
+   *
+   * @param seg the segment which is to find blocks
+   * @return a map of block and its file list
*/
-  public CarbonFile[] getDeleteDeltaFilesList(final Segment segmentId, final 
String blockName) {
-String segmentPath = CarbonTablePath.getSegmentPath(
-identifier.getTablePath(), segmentId.getSegmentNo());
-CarbonFile segDir =
-FileFactory.getCarbonFile(segmentPath);
+  public Map> getDeleteDeltaFilesList(final Segment 
seg) {
+
+Map blockDeltaStartAndEndTimestampMap = new HashMap<>();

Review comment:
   I modified this method as above approach. 
   The delete delta file name is generated from updatedetails, without listing 
files





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3917: [CARBONDATA-3978] Clean Files Refactor and support for trash folder in carbondata

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3917:
URL: https://github.com/apache/carbondata/pull/3917#issuecomment-713236605


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4551/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] shenjiayu17 commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-20 Thread GitBox


shenjiayu17 commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r508933755



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/HorizontalCompaction.scala
##
@@ -173,6 +176,9 @@ object HorizontalCompaction {
 
 val db = carbonTable.getDatabaseName
 val table = carbonTable.getTableName
+
+LOG.info(s"Horizontal Delete Compaction operation is getting valid 
segments for [$db.$table].")

Review comment:
   This log is modified, prints the deletedBlocksList and time taken to get 
it





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] shenjiayu17 commented on a change in pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-20 Thread GitBox


shenjiayu17 commented on a change in pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#discussion_r508933445



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/HorizontalCompaction.scala
##
@@ -125,6 +125,9 @@ object HorizontalCompaction {
   segLists: util.List[Segment]): Unit = {
 val db = carbonTable.getDatabaseName
 val table = carbonTable.getTableName
+
+LOG.info(s"Horizontal Update Compaction operation is getting valid 
segments for [$db.$table].")

Review comment:
   This log is modified, prints the validSegList and time taken to get it





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ydvpankaj99 commented on pull request #3979: [Carbondata-3954] Fix insertion from ORC table into carbon table when sort scope is global sort

2020-10-20 Thread GitBox


ydvpankaj99 commented on pull request #3979:
URL: https://github.com/apache/carbondata/pull/3979#issuecomment-713229757


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ydvpankaj99 commented on pull request #3917: [CARBONDATA-3978] Clean Files Refactor and support for trash folder in carbondata

2020-10-20 Thread GitBox


ydvpankaj99 commented on pull request #3917:
URL: https://github.com/apache/carbondata/pull/3917#issuecomment-713229548


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] QiangCai commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code

2020-10-20 Thread GitBox


QiangCai commented on pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#issuecomment-713229206


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ydvpankaj99 commented on pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.

2020-10-20 Thread GitBox


ydvpankaj99 commented on pull request #3875:
URL: https://github.com/apache/carbondata/pull/3875#issuecomment-713228858


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ydvpankaj99 commented on pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filter

2020-10-20 Thread GitBox


ydvpankaj99 commented on pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#issuecomment-713228564


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ydvpankaj99 commented on pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


ydvpankaj99 commented on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-713228390


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ydvpankaj99 commented on pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-20 Thread GitBox


ydvpankaj99 commented on pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#issuecomment-713228255


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…

2020-10-20 Thread GitBox


marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-713222723


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3917: [CARBONDATA-3978] Clean Files Refactor and support for trash folder in carbondata

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3917:
URL: https://github.com/apache/carbondata/pull/3917#issuecomment-713218644


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2800/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-713208327


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4544/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#issuecomment-713198827


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4547/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-713198832


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4549/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filter

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#issuecomment-713198829


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4548/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3977:
URL: https://github.com/apache/carbondata/pull/3977#issuecomment-713198831


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4545/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-713189481


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4543/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-713180217


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4542/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3988: [WIP] Improve the table status and segment file writing

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3988:
URL: https://github.com/apache/carbondata/pull/3988#issuecomment-713179538


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4550/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-713178626


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2789/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3988: [WIP] Improve the table status and segment file writing

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3988:
URL: https://github.com/apache/carbondata/pull/3988#issuecomment-713162507


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2799/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3989: [TEST] CI

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3989:
URL: https://github.com/apache/carbondata/pull/3989#issuecomment-713161614


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2787/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-713152841


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4546/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-713150750


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2795/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3987: [CARBONDATA-4039] Support Local dictionary for Presto complex datatypes

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-713149620


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2788/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3977:
URL: https://github.com/apache/carbondata/pull/3977#issuecomment-713149461


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2791/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-713148889


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2790/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3989: [TEST] CI

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3989:
URL: https://github.com/apache/carbondata/pull/3989#issuecomment-713148503


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4541/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#issuecomment-713143518


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2793/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filter

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#issuecomment-713136611


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2794/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-713095278


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2792/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] QiangCai commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code

2020-10-20 Thread GitBox


QiangCai commented on pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#issuecomment-712908989


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] marchpure commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…

2020-10-20 Thread GitBox


marchpure commented on pull request #3977:
URL: https://github.com/apache/carbondata/pull/3977#issuecomment-712888965


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Indhumathi27 opened a new pull request #3989: [TEST] CI

2020-10-20 Thread GitBox


Indhumathi27 opened a new pull request #3989:
URL: https://github.com/apache/carbondata/pull/3989


### Why is this PR needed?


### What changes were proposed in this PR?
   
   
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
   
### Is any new testcase added?
- No
- Yes
   
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ShreelekhyaG commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example

2020-10-20 Thread GitBox


ShreelekhyaG commented on pull request #3914:
URL: https://github.com/apache/carbondata/pull/3914#issuecomment-712858165


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filter

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#issuecomment-712853927


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4538/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3875:
URL: https://github.com/apache/carbondata/pull/3875#issuecomment-712853970


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4537/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3917: [CARBONDATA-3978] Clean Files Refactor and support for trash folder in carbondata

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3917:
URL: https://github.com/apache/carbondata/pull/3917#issuecomment-712853499


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2781/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Updated] (CARBONDATA-4039) Support Local dictionary for presto complex datatypes

2020-10-20 Thread Akshay (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akshay updated CARBONDATA-4039:
---
Description: 
Support Local dictionary for presto complex datatypes - 

Presto complex datatypes - array and struct only.

[https://github.com/apache/carbondata/pull/3987]

  was:
Support Local dictionary for presto complex datatypes - 

Local dictionary support for presto complex datatypes - array and struct only.

https://github.com/apache/carbondata/pull/3987


> Support Local dictionary for presto complex datatypes
> -
>
> Key: CARBONDATA-4039
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4039
> Project: CarbonData
>  Issue Type: New Feature
>  Components: core, presto-integration
>Reporter: Akshay
>Priority: Major
>
> Support Local dictionary for presto complex datatypes - 
> Presto complex datatypes - array and struct only.
> [https://github.com/apache/carbondata/pull/3987]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3917: [CARBONDATA-3978] Clean Files Refactor and support for trash folder in carbondata

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3917:
URL: https://github.com/apache/carbondata/pull/3917#issuecomment-712854071


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4535/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3979: [Carbondata-3954] Fix insertion from ORC table into carbon table when sort scope is global sort

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3979:
URL: https://github.com/apache/carbondata/pull/3979#issuecomment-712854016


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4536/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3875:
URL: https://github.com/apache/carbondata/pull/3875#issuecomment-712853401


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2783/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3979: [Carbondata-3954] Fix insertion from ORC table into carbon table when sort scope is global sort

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3979:
URL: https://github.com/apache/carbondata/pull/3979#issuecomment-712853433


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2782/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3987: [WIP ]local-dictionary

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-712853297


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2786/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#issuecomment-712853350


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2785/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Created] (CARBONDATA-4039) Support Local dictionary for presto complex datatypes

2020-10-20 Thread Akshay (Jira)
Akshay created CARBONDATA-4039:
--

 Summary: Support Local dictionary for presto complex datatypes
 Key: CARBONDATA-4039
 URL: https://issues.apache.org/jira/browse/CARBONDATA-4039
 Project: CarbonData
  Issue Type: Improvement
  Components: core, presto-integration
Reporter: Akshay


Support Local dictionary for presto complex datatypes - 

Local dictionary support for presto complex datatypes - array and struct only.

https://github.com/apache/carbondata/pull/3987



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-4039) Support Local dictionary for presto complex datatypes

2020-10-20 Thread Akshay (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akshay updated CARBONDATA-4039:
---
Issue Type: New Feature  (was: Improvement)

> Support Local dictionary for presto complex datatypes
> -
>
> Key: CARBONDATA-4039
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4039
> Project: CarbonData
>  Issue Type: New Feature
>  Components: core, presto-integration
>Reporter: Akshay
>Priority: Major
>
> Support Local dictionary for presto complex datatypes - 
> Local dictionary support for presto complex datatypes - array and struct only.
> https://github.com/apache/carbondata/pull/3987



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3987: [WIP ]local-dictionary

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3987:
URL: https://github.com/apache/carbondata/pull/3987#issuecomment-712851661


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4540/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Updated] (CARBONDATA-3934) Support insert into command for transactional support

2020-10-20 Thread Akash R Nilugal (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akash R Nilugal updated CARBONDATA-3934:

Attachment: Presto_write_flow.pdf

> Support insert into command for transactional support
> -
>
> Key: CARBONDATA-3934
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3934
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Akash R Nilugal
>Assignee: Akash R Nilugal
>Priority: Major
> Attachments: Presto_write_flow.pdf
>
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> Support insert into command for transactional support.
> Should support writing table status file, segment files, all the folder 
> structure similar to transactional carbon table.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3831) Support write carbon files with presto.

2020-10-20 Thread Akash R Nilugal (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akash R Nilugal updated CARBONDATA-3831:

Attachment: Presto_write_flow.pdf

> Support write carbon files with presto.
> ---
>
> Key: CARBONDATA-3831
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3831
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Akash R Nilugal
>Assignee: Akash R Nilugal
>Priority: Major
> Attachments: Presto_write_flow.pdf, carbon_presto_write_transactional 
> SUpport.pdf
>
>
> As we know the CarbonDataisan indexed columnar data format for fast analytics 
> on big data platforms. So we have already integrated with the query engines 
> like spark and even presto. Currently with presto we only support the 
> querying of carbondata files. But we don’t yet support the writing of 
> carbondata files
> through the presto engine.
>   Currentlypresto is integrated with carbondata for reading the 
> carbondata files via presto. For this, we should be having the store already 
> ready which may be written carbon in spark and the table
> should be hive metastore. So using carbondata connector we are able to read 
> the carbondata files. But we cannot create table or load the data to table in 
> presto. So it will somewhat hectic job to read the carbonfiles , by writing 
> first with other engine.
> So here i will be trying to support the transactional load support in presto 
> integration for carbon. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [carbondata] akashrn5 commented on a change in pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.

2020-10-20 Thread GitBox


akashrn5 commented on a change in pull request #3875:
URL: https://github.com/apache/carbondata/pull/3875#discussion_r508502780



##
File path: 
integration/presto/src/main/prestosql/org/apache/carbondata/presto/CarbondataModule.java
##
@@ -127,7 +127,8 @@ public void configure(Binder binder) {
 .in(Scopes.SINGLETON);
 binder.bind(HivePartitionManager.class).in(Scopes.SINGLETON);
 
binder.bind(LocationService.class).to(HiveLocationService.class).in(Scopes.SINGLETON);
-binder.bind(HiveMetadataFactory.class).in(Scopes.SINGLETON);
+
binder.bind(HiveLocationService.class).to(CarbonDataLocationService.class).in(Scopes.SINGLETON);

Review comment:
   added to jira





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3986: [CARBONDATA-4034] Improve the time-consuming of Horizontal Compaction for update

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3986:
URL: https://github.com/apache/carbondata/pull/3986#issuecomment-712840146


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4539/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filter

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#issuecomment-712837186


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2784/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#issuecomment-712826307


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4534/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#issuecomment-712824815


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2780/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akashrn5 commented on a change in pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.

2020-10-20 Thread GitBox


akashrn5 commented on a change in pull request #3875:
URL: https://github.com/apache/carbondata/pull/3875#discussion_r508461938



##
File path: 
integration/presto/src/main/prestosql/org/apache/carbondata/presto/CarbonDataFileWriter.java
##
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.presto;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.hadoop.api.CarbonTableOutputFormat;
+import org.apache.carbondata.hive.CarbonHiveSerDe;
+import org.apache.carbondata.hive.MapredCarbonOutputFormat;
+import org.apache.carbondata.presto.impl.CarbonTableConfig;
+
+import com.google.common.collect.ImmutableList;
+import io.prestosql.plugin.hive.HiveFileWriter;
+import io.prestosql.plugin.hive.HiveType;
+import io.prestosql.plugin.hive.HiveWriteUtils;
+import io.prestosql.spi.Page;
+import io.prestosql.spi.PrestoException;
+import io.prestosql.spi.block.Block;
+import io.prestosql.spi.type.Type;
+import io.prestosql.spi.type.TypeManager;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import 
org.apache.hadoop.hive.serde2.objectinspector.SettableStructObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.StructField;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.log4j.Logger;
+
+import static com.google.common.collect.ImmutableList.toImmutableList;
+import static io.prestosql.plugin.hive.HiveErrorCode.HIVE_WRITER_DATA_ERROR;
+import static java.util.Objects.requireNonNull;
+import static java.util.stream.Collectors.toList;
+import static org.apache.hadoop.hive.conf.HiveConf.ConfVars.COMPRESSRESULT;
+
+/**
+ * This class implements HiveFileWriter and it creates the carbonFileWriter to 
write the page data
+ * sent from presto.
+ */
+public class CarbonDataFileWriter implements HiveFileWriter {
+
+  private static final Logger LOG =
+  LogServiceFactory.getLogService(CarbonDataFileWriter.class.getName());
+
+  private final JobConf configuration;
+  private final Path outPutPath;
+  private final FileSinkOperator.RecordWriter recordWriter;
+  private final CarbonHiveSerDe serDe;
+  private final int fieldCount;
+  private final Object row;
+  private final SettableStructObjectInspector tableInspector;
+  private final List structFields;
+  private final HiveWriteUtils.FieldSetter[] setters;
+
+  private boolean isCommitDone;
+
+  public CarbonDataFileWriter(Path outPutPath, List inputColumnNames, 
Properties properties,
+  JobConf configuration, TypeManager typeManager) throws SerDeException {
+requireNonNull(outPutPath, "path is null");
+// take the outputPath same as location in compliance with the carbon 
store folder structure.
+this.outPutPath = new Path(properties.getProperty("location"));
+this.configuration = requireNonNull(configuration, "conf is null");
+List columnNames = Arrays
+.asList(properties.getProperty(IOConstants.COLUMNS, 
"").split(CarbonCommonConstants.COMMA));
+List fileColumnTypes =
+HiveType.toHiveTypes(properties.getProperty(IOConstants.COLUMNS_TYPES, 
"")).stream()
+.map(hiveType -> hiveType.getType(typeManager)).collect(toList());
+this.fieldCount = columnNames.size();
+this.serDe = new CarbonHiveSerDe();
+serDe.initialize(configuration, properties);
+this.tableInspector = (ArrayWritableObjectInspector) 
serDe.getObjectInspector();
+
+this.structFields =
+
ImmutableList.copyOf(inputColumnNames.stream().map(tableInspector::getStructFieldRef)
+   

[jira] [Created] (CARBONDATA-4038) Support metrics during presto write

2020-10-20 Thread Akash R Nilugal (Jira)
Akash R Nilugal created CARBONDATA-4038:
---

 Summary: Support metrics during presto write 
 Key: CARBONDATA-4038
 URL: https://issues.apache.org/jira/browse/CARBONDATA-4038
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Akash R Nilugal


Support metrics during presto write such as getSystemMemoryUsage () and 
getValidationCpuNanos()



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filt

2020-10-20 Thread GitBox


ajantha-bhat commented on a change in pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#discussion_r508460403



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/execution/strategy/CarbonLateDecodeStrategy.scala
##
@@ -984,6 +988,80 @@ private[sql] class CarbonLateDecodeStrategy extends 
SparkStrategy {
   null)(sparkSession)
 }
   }
+
+  object ExtractTakeOrderedAndProjectExec {
+
+def unapply(plan: LogicalPlan): Option[CarbonTakeOrderedAndProjectExec] = {
+  val allRelations = plan.collect { case logicalRelation: LogicalRelation 
=> logicalRelation }
+  // push down order by limit to carbon map task,
+  // only when there are only one CarbonDatasourceHadoopRelation
+  if (allRelations.size != 1 ||
+  allRelations.exists(x => 
!x.relation.isInstanceOf[CarbonDatasourceHadoopRelation])) {
+return None
+  }
+  //  check and Replace TakeOrderedAndProject with 
CarbonTakeOrderedAndProjectExec.
+  val relation = 
allRelations.head.relation.asInstanceOf[CarbonDatasourceHadoopRelation]
+  val sparkPlan = plan match {
+case ReturnAnswer(rootPlan) => rootPlan match {
+  case Limit(IntegerLiteral(limit), Sort(order, true, child)) =>
+TakeOrderedAndProjectExec(limit,
+  order,
+  child.output,
+  planLater(pushLimit(limit, child)))
+  case Limit(IntegerLiteral(limit), Project(projectList, Sort(order, 
true, child))) =>
+TakeOrderedAndProjectExec(limit, order, projectList, 
planLater(pushLimit(limit, child)))

Review comment:
   ok. Handled.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3932: [CARBONDATA-3994] Skip Order by for map task if it is a first sort column and use limit pushdown for array_contains filt

2020-10-20 Thread GitBox


ajantha-bhat commented on a change in pull request #3932:
URL: https://github.com/apache/carbondata/pull/3932#discussion_r508460792



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/execution/CarbonTakeOrderedAndProjectExec.scala
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.serializer.Serializer
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, NamedExpression, 
SortOrder, UnsafeProjection}
+import 
org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, 
SinglePartition}
+import org.apache.spark.sql.execution.exchange.ShuffleExchangeExec
+import org.apache.spark.util.Utils
+
+// To skip the order at map task
+case class CarbonTakeOrderedAndProjectExec(
+limit: Int,
+sortOrder: Seq[SortOrder],
+projectList: Seq[NamedExpression],
+child: SparkPlan,
+skipMapOrder: Boolean = false,
+readFromHead: Boolean = true) extends UnaryExecNode {

Review comment:
   Both are case class, so cannot extend this. 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] QiangCai commented on pull request #3979: [Carbondata-3954] Fix insertion from ORC table into carbon table when sort scope is global sort

2020-10-20 Thread GitBox


QiangCai commented on pull request #3979:
URL: https://github.com/apache/carbondata/pull/3979#issuecomment-712799969


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] QiangCai commented on pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.

2020-10-20 Thread GitBox


QiangCai commented on pull request #3875:
URL: https://github.com/apache/carbondata/pull/3875#issuecomment-712798753


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.

2020-10-20 Thread GitBox


ajantha-bhat commented on a change in pull request #3875:
URL: https://github.com/apache/carbondata/pull/3875#discussion_r508431395



##
File path: 
integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##
@@ -92,6 +95,14 @@ public void checkOutputSpecs(FileSystem fileSystem, JobConf 
jobConf) throws IOEx
 }
 String tablePath = 
FileFactory.getCarbonFile(carbonLoadModel.getTablePath()).getAbsolutePath();
 TaskAttemptID taskAttemptID = 
TaskAttemptID.forName(jc.get("mapred.task.id"));
+// taskAttemptID will be null when the insert job is fired from presto. 
Presto send the JobConf
+// and since presto does not use the MR framework for execution, the 
mapred.task.id will be
+// null, so prepare a new ID.
+if (taskAttemptID == null) {
+  SimpleDateFormat formatter = new SimpleDateFormat("MMddHHmm");
+  String jobTrackerId = formatter.format(new Date());
+  taskAttemptID = new TaskAttemptID(jobTrackerId, 0, TaskType.MAP, 0, 0);

Review comment:
   Also please check filenames while testing whether segment id and other 
info is proper in the file name created by presto.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.

2020-10-20 Thread GitBox


ajantha-bhat commented on a change in pull request #3875:
URL: https://github.com/apache/carbondata/pull/3875#discussion_r508430591



##
File path: 
integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##
@@ -92,6 +95,14 @@ public void checkOutputSpecs(FileSystem fileSystem, JobConf 
jobConf) throws IOEx
 }
 String tablePath = 
FileFactory.getCarbonFile(carbonLoadModel.getTablePath()).getAbsolutePath();
 TaskAttemptID taskAttemptID = 
TaskAttemptID.forName(jc.get("mapred.task.id"));
+// taskAttemptID will be null when the insert job is fired from presto. 
Presto send the JobConf
+// and since presto does not use the MR framework for execution, the 
mapred.task.id will be
+// null, so prepare a new ID.
+if (taskAttemptID == null) {
+  SimpleDateFormat formatter = new SimpleDateFormat("MMddHHmm");
+  String jobTrackerId = formatter.format(new Date());
+  taskAttemptID = new TaskAttemptID(jobTrackerId, 0, TaskType.MAP, 0, 0);

Review comment:
   ok, If this task number is used in file name, in case of 
non-transactional  concurrent write. two files can have same file name leading 
to many issues. so, I suggested UUID. you can check again.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3917: [CARBONDATA-3978] Clean Files Refactor and support for trash folder in carbondata

2020-10-20 Thread GitBox


CarbonDataQA1 commented on pull request #3917:
URL: https://github.com/apache/carbondata/pull/3917#issuecomment-712772523


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4533/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] QiangCai commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code

2020-10-20 Thread GitBox


QiangCai commented on pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#issuecomment-712772488


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




  1   2   >