[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3778: [CARBONDATA-3916] Support array complex type with SI

2020-08-19 Thread GitBox


ajantha-bhat commented on a change in pull request #3778:
URL: https://github.com/apache/carbondata/pull/3778#discussion_r473620842



##
File path: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
##
@@ -2400,6 +2400,23 @@ private CarbonCommonConstants() {
*/
   public static final String CARBON_SI_SEGMENT_MERGE_DEFAULT = "false";
 
+  /**
+   * Until the threshold for complex filter is reached, row id will be set to 
the bitset in
+   * implicit filter during secondary index pruning
+   */
+  public static final String SI_COMPLEX_FILTER_THRESHOLD = 
"carbon.si.complex.filter.threshold";
+
+  /**
+   * Maximum value for complex filter threshold
+   */
+  public static final String SI_COMPLEX_FILTER_THRESHOLD_DEFAULT = "32000";
+
+  /**
+   * Property to decide if position id till row level or not
+   */
+  public static final String IS_TUPLE_ID_TILL_ROW_FOR_SI_COMPLEX =

Review comment:
   cc: @kunal642 , @QiangCai , @ravipesala 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3778: [CARBONDATA-3916] Support array complex type with SI

2020-08-19 Thread GitBox


ajantha-bhat commented on a change in pull request #3778:
URL: https://github.com/apache/carbondata/pull/3778#discussion_r473601787



##
File path: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
##
@@ -2400,6 +2400,23 @@ private CarbonCommonConstants() {
*/
   public static final String CARBON_SI_SEGMENT_MERGE_DEFAULT = "false";
 
+  /**
+   * Until the threshold for complex filter is reached, row id will be set to 
the bitset in
+   * implicit filter during secondary index pruning
+   */
+  public static final String SI_COMPLEX_FILTER_THRESHOLD = 
"carbon.si.complex.filter.threshold";
+
+  /**
+   * Maximum value for complex filter threshold
+   */
+  public static final String SI_COMPLEX_FILTER_THRESHOLD_DEFAULT = "32000";
+
+  /**
+   * Property to decide if position id till row level or not
+   */
+  public static final String IS_TUPLE_ID_TILL_ROW_FOR_SI_COMPLEX =

Review comment:
   CC: @kunal642 , @QiangCai 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write

2020-08-19 Thread GitBox


ajantha-bhat commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-677162461


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Karan980 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


Karan980 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-677123138


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3778: [CARBONDATA-3916] Support array complex type with SI

2020-08-19 Thread GitBox


ajantha-bhat commented on a change in pull request #3778:
URL: https://github.com/apache/carbondata/pull/3778#discussion_r473601787



##
File path: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
##
@@ -2400,6 +2400,23 @@ private CarbonCommonConstants() {
*/
   public static final String CARBON_SI_SEGMENT_MERGE_DEFAULT = "false";
 
+  /**
+   * Until the threshold for complex filter is reached, row id will be set to 
the bitset in
+   * implicit filter during secondary index pruning
+   */
+  public static final String SI_COMPLEX_FILTER_THRESHOLD = 
"carbon.si.complex.filter.threshold";
+
+  /**
+   * Maximum value for complex filter threshold
+   */
+  public static final String SI_COMPLEX_FILTER_THRESHOLD_DEFAULT = "32000";
+
+  /**
+   * Property to decide if position id till row level or not
+   */
+  public static final String IS_TUPLE_ID_TILL_ROW_FOR_SI_COMPLEX =

Review comment:
   CC: @kunal642 , @QiangCai 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3778: [CARBONDATA-3916] Support array complex type with SI

2020-08-19 Thread GitBox


ajantha-bhat commented on a change in pull request #3778:
URL: https://github.com/apache/carbondata/pull/3778#discussion_r473600323



##
File path: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
##
@@ -2400,6 +2400,23 @@ private CarbonCommonConstants() {
*/
   public static final String CARBON_SI_SEGMENT_MERGE_DEFAULT = "false";
 
+  /**
+   * Until the threshold for complex filter is reached, row id will be set to 
the bitset in
+   * implicit filter during secondary index pruning
+   */
+  public static final String SI_COMPLEX_FILTER_THRESHOLD = 
"carbon.si.complex.filter.threshold";
+
+  /**
+   * Maximum value for complex filter threshold
+   */
+  public static final String SI_COMPLEX_FILTER_THRESHOLD_DEFAULT = "32000";
+
+  /**
+   * Property to decide if position id till row level or not
+   */
+  public static final String IS_TUPLE_ID_TILL_ROW_FOR_SI_COMPLEX =

Review comment:
   In the community, we concluded that no need of row level position 
reference. So, why this is required ?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3778: [CARBONDATA-3916] Support array complex type with SI

2020-08-19 Thread GitBox


ajantha-bhat commented on a change in pull request #3778:
URL: https://github.com/apache/carbondata/pull/3778#discussion_r473600574



##
File path: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
##
@@ -2400,6 +2400,23 @@ private CarbonCommonConstants() {
*/
   public static final String CARBON_SI_SEGMENT_MERGE_DEFAULT = "false";
 
+  /**
+   * Until the threshold for complex filter is reached, row id will be set to 
the bitset in
+   * implicit filter during secondary index pruning
+   */
+  public static final String SI_COMPLEX_FILTER_THRESHOLD = 
"carbon.si.complex.filter.threshold";
+
+  /**
+   * Maximum value for complex filter threshold
+   */
+  public static final String SI_COMPLEX_FILTER_THRESHOLD_DEFAULT = "32000";
+
+  /**
+   * Property to decide if position id till row level or not
+   */
+  public static final String IS_TUPLE_ID_TILL_ROW_FOR_SI_COMPLEX =

Review comment:
   I think above 2 properties also not needed if it is not row level 
position reference 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3778: [CARBONDATA-3916] Support array complex type with SI

2020-08-19 Thread GitBox


ajantha-bhat commented on a change in pull request #3778:
URL: https://github.com/apache/carbondata/pull/3778#discussion_r473599181



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/command/SICreationCommand.scala
##
@@ -443,10 +443,34 @@ private[sql] case class CarbonCreateSecondaryIndexCommand(
   databaseName: String, tableName: String, indexTableName: String,
   absoluteTableIdentifier: AbsoluteTableIdentifier): TableInfo = {
 var schemaOrdinal = -1
-var allColumns = indexModel.columnNames.map { indexCol =>
-  val colSchema = carbonTable.getDimensionByName(indexCol).getColumnSchema
+val complexDimensions = carbonTable.getAllDimensions.asScala
+  .filter(dim => dim.getDataType.isComplexType &&
+ indexModel.columnNames.asJava.contains(dim.getColName))
+if (complexDimensions.size > 1) {
+  throw new ErrorMessage("SI creation with more than one complex type is 
not supported yet");
+}
+var allColumns = List[ColumnSchema]()

Review comment:
   Consider the scenario where one SI table contains (complex, primitive1, 
primitive2), we need to maintain the same order. But now it will become 
primitive1,primitive2,complex1. which is wrong.
   
   So, suggest to keep the user-specified order.
   Can refer below code
   
   var allColumns = List[ColumnSchema]()
   indexModel.columnNames.foreach { indexCol =>
 val dimension = carbonTable.getDimensionByName(tableName, indexCol)
 val colSchema = dimension.getColumnSchema
 schemaOrdinal += 1
 allColumns = allColumns :+ cloneColumnSchema(colSchema, schemaOrdinal)
   }
   complexDimensions.foreach { complexDim =>
 if (complexDim.getNumberOfChild > 0) {
   if (complexDim.getListOfChildDimensions.asScala
 .exists(col => DataTypes.isArrayType(col.getDataType))) {
 throw new ErrorMessage("SI creation with nested array complex type is 
not supported yet");
   }
 }
   }





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-676813298


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2061/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-676804656


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3802/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Karan980 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


Karan980 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-676703965


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-676686821


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2060/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3896: [CARBONDATA-3955] Fix load failures due to daylight saving time changes

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3896:
URL: https://github.com/apache/carbondata/pull/3896#issuecomment-676683099


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2059/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3896: [CARBONDATA-3955] Fix load failures due to daylight saving time changes

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3896:
URL: https://github.com/apache/carbondata/pull/3896#issuecomment-676679976


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3800/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-676671661


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3801/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3865: [CARBONDATA-3928] Handled the Strings which length is greater than 32000 as a bad record.

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3865:
URL: https://github.com/apache/carbondata/pull/3865#issuecomment-676631987


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2058/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3865: [CARBONDATA-3928] Handled the Strings which length is greater than 32000 as a bad record.

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3865:
URL: https://github.com/apache/carbondata/pull/3865#issuecomment-676631397


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3799/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3873: [CARBONDATA-3956] Reindex command on SI table

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676589566







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Created] (CARBONDATA-3957) Adding 2 carbon properties to enable/disable and set a limit to number of segments repaired in SILoadEventListenerForFailedSegments

2020-08-19 Thread Vikram Ahuja (Jira)
Vikram Ahuja created CARBONDATA-3957:


 Summary: Adding 2 carbon properties to enable/disable and set a 
limit to number of segments repaired in SILoadEventListenerForFailedSegments
 Key: CARBONDATA-3957
 URL: https://issues.apache.org/jira/browse/CARBONDATA-3957
 Project: CarbonData
  Issue Type: Improvement
Reporter: Vikram Ahuja






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (CARBONDATA-3956) Implementing a new Reindex command to repair the missing SI Segments

2020-08-19 Thread Vikram Ahuja (Jira)
Vikram Ahuja created CARBONDATA-3956:


 Summary: Implementing a new Reindex command to repair the missing 
SI Segments
 Key: CARBONDATA-3956
 URL: https://issues.apache.org/jira/browse/CARBONDATA-3956
 Project: CarbonData
  Issue Type: New Feature
Reporter: Vikram Ahuja






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [carbondata] nihal0107 commented on pull request #3865: [CARBONDATA-3928] Handled the Strings which length is greater than 32000 as a bad record.

2020-08-19 Thread GitBox


nihal0107 commented on pull request #3865:
URL: https://github.com/apache/carbondata/pull/3865#issuecomment-676567910


   retest this please.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3865: [CARBONDATA-3928] Handled the Strings which length is greater than 32000 as a bad record.

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3865:
URL: https://github.com/apache/carbondata/pull/3865#issuecomment-676567218


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3797/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3865: [CARBONDATA-3928] Handled the Strings which length is greater than 32000 as a bad record.

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3865:
URL: https://github.com/apache/carbondata/pull/3865#issuecomment-676566449


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2055/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3887: [CARBONDATA-3830] Support Array and Struct of all primitive type reading from presto

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3887:
URL: https://github.com/apache/carbondata/pull/3887#issuecomment-676563702


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2052/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3887: [CARBONDATA-3830] Support Array and Struct of all primitive type reading from presto

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3887:
URL: https://github.com/apache/carbondata/pull/3887#issuecomment-676550943


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3795/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3856: [CARBONDATA-3929]Improve CDC performance

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3856:
URL: https://github.com/apache/carbondata/pull/3856#issuecomment-676550122


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2051/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3856: [CARBONDATA-3929]Improve CDC performance

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3856:
URL: https://github.com/apache/carbondata/pull/3856#issuecomment-676539409


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3793/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3873: [WIP] Reindex command on SI table

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676494128


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2054/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3873: [WIP] Reindex command on SI table

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676491522


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3796/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] vikramahuja1001 commented on pull request #3873: [WIP] Reindex command on SI table

2020-08-19 Thread GitBox


vikramahuja1001 commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676484171


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on pull request #3887: [CARBONDATA-3830] Support Array and Struct of all primitive type reading from presto

2020-08-19 Thread GitBox


ajantha-bhat commented on pull request #3887:
URL: https://github.com/apache/carbondata/pull/3887#issuecomment-676478366


   @QiangCai , @jackylk , @kumarvishal09 @ravipesala : please review. 
   Also please go through PR description and open points.
   
   some testcase validation need to improve, now manually verified. will add it 
by tomorrow. But can start the review.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3873: [WIP] Reindex command on SI table

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676474301


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3792/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-676474154


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3791/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3873: [WIP] Reindex command on SI table

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676473431


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2050/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-676471110


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2049/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] vikramahuja1001 commented on pull request #3873: [WIP] Reindex command on SI table

2020-08-19 Thread GitBox


vikramahuja1001 commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676462137


   okay



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] brijoobopanna commented on pull request #3873: [WIP] Reindex command on SI table

2020-08-19 Thread GitBox


brijoobopanna commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676461075


   @vikramahuja1001 please complete all implementations and once WIP is removed 
we can start the review



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] vikramahuja1001 commented on pull request #3873: [WIP] Reindex command on SI table

2020-08-19 Thread GitBox


vikramahuja1001 commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676458977


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Karan980 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


Karan980 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-676257863


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] vikramahuja1001 commented on pull request #3894: [WIP] Added property to enable disable SIforFailed segments and added prope…

2020-08-19 Thread GitBox


vikramahuja1001 commented on pull request #3894:
URL: https://github.com/apache/carbondata/pull/3894#issuecomment-676196999


   @kunal642 @VenuReddy2103 @akashrn5 , please review



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] vikramahuja1001 commented on pull request #3873: [WIP] Reindex command on SI table

2020-08-19 Thread GitBox


vikramahuja1001 commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676196273


   @VenuReddy2103 @akashrn5 @QiangCai , please review this PR



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3873: [WIP] Repair SI Command

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676194037


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2047/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3873: [WIP] Repair SI Command

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#issuecomment-676190060


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3789/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3887: [WIP] Refactor #3773 and support struct type

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3887:
URL: https://github.com/apache/carbondata/pull/3887#issuecomment-676189511


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2048/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3887: [WIP] Refactor #3773 and support struct type

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3887:
URL: https://github.com/apache/carbondata/pull/3887#issuecomment-676183932


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3790/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-676160206


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3787/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3873: [WIP] Repair SI Command

2020-08-19 Thread GitBox


vikramahuja1001 commented on a change in pull request #3873:
URL: https://github.com/apache/carbondata/pull/3873#discussion_r472947089



##
File path: core/src/main/java/org/apache/carbondata/core/util/SessionParams.java
##
@@ -155,6 +141,12 @@ private boolean validateKeyValue(String key, String value) 
throws InvalidConfigu
   case CARBON_PUSH_ROW_FILTERS_FOR_VECTOR:
   case CARBON_ENABLE_INDEX_SERVER:
   case CARBON_QUERY_STAGE_INPUT:
+  case CARBON_LOAD_SI_REPAIR:
+isValid = CarbonUtil.validateBoolean(value);

Review comment:
   removed





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-676141794


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2045/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3896: [CARBONDATA-3955] Fix load failures due to daylight saving time changes

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3896:
URL: https://github.com/apache/carbondata/pull/3896#issuecomment-676013906


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3786/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3894: [WIP] Added property to enable disable SIforFailed segments and added prope…

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3894:
URL: https://github.com/apache/carbondata/pull/3894#issuecomment-676010074


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2043/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3894: [WIP] Added property to enable disable SIforFailed segments and added prope…

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3894:
URL: https://github.com/apache/carbondata/pull/3894#issuecomment-676006295


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3785/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3896: [CARBONDATA-3955] Fix load failures due to daylight saving time changes

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3896:
URL: https://github.com/apache/carbondata/pull/3896#issuecomment-675990742


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2044/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Karan980 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


Karan980 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-675957376


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3885: [CARBONDATA-3946] Support IndexServer with Presto Engine

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3885:
URL: https://github.com/apache/carbondata/pull/3885#issuecomment-675946675


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2042/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3885: [CARBONDATA-3946] Support IndexServer with Presto Engine

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3885:
URL: https://github.com/apache/carbondata/pull/3885#issuecomment-675924673


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3784/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-675916096


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2039/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI

2020-08-19 Thread GitBox


CarbonDataQA1 commented on pull request #3876:
URL: https://github.com/apache/carbondata/pull/3876#issuecomment-675915053


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3781/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Created] (CARBONDATA-3955) Fix load failures due to daylight saving time changes

2020-08-19 Thread SHREELEKHYA GAMPA (Jira)
SHREELEKHYA GAMPA created CARBONDATA-3955:
-

 Summary: Fix load failures due to daylight saving time changes
 Key: CARBONDATA-3955
 URL: https://issues.apache.org/jira/browse/CARBONDATA-3955
 Project: CarbonData
  Issue Type: Bug
Reporter: SHREELEKHYA GAMPA


1) Fix load failures due to daylight saving time changes.
2) During load, date/timestamp year values with >4 digit should fail or be null 
according to bad records action property.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[GitHub] [carbondata] vikramahuja1001 commented on pull request #3894: [WIP] Added property to enable disable SIforFailed segments and added prope…

2020-08-19 Thread GitBox


vikramahuja1001 commented on pull request #3894:
URL: https://github.com/apache/carbondata/pull/3894#issuecomment-675885125


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] yutaoChina commented on pull request #3892: flink write carbon file to hdfs when file size is less than 1M,can't write

2020-08-19 Thread GitBox


yutaoChina commented on pull request #3892:
URL: https://github.com/apache/carbondata/pull/3892#issuecomment-675874179


   > 
   > 
   > @yutaoChina : Thanks for working on this.
   > a) please handle the compilation error
   > b) please create a jira issue and add it in the issue header
   this is jira issues
   https://issues.apache.org/jira/browse/CARBONDATA-3925



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Updated] (CARBONDATA-3925) flink-integration write carbon file to hdfs error

2020-08-19 Thread yutao (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yutao updated CARBONDATA-3925:
--
Description: 
in CarbonWriter.java code ,you can find this;
 public abstract class *{color:#FF}CarbonWriter{color}* extends 
ProxyFileWriter {
 private static final Logger LOGGER =
 
LogServiceFactory.getLogService({color:#FF}CarbonS3Writer{color}.class.getName());}
 always wo can find logfile print like ;
 2020-07-27 14:19:25,107 DEBUG org.apache.carbon.flink.CarbonS3Writer 
 this is puzzled;

2020-08-19 13:14:13,329 INFO 
org.apache.carbondata.hadoop.api.CarbonTableOutputFormat - Closed writer task 
attempt_f229b922-1f77-426f-a4bc-42e49aa53df7__m_1968253873_-1049302646
2020-08-19 13:14:13,329 DEBUG org.apache.carbon.flink.CarbonLocalWriter - 
Commit write. org.apache.carbon.flink.CarbonLocalWriter@41f5c4a9
2020-08-19 13:14:13,329 DEBUG org.apache.carbon.flink.CarbonS3Writer - Upload 
file[/home/hadoop/yutest/d963e9836ccb4318aa8fc953af983d07/part-0-a132f98547584dcabae6c43090626baf_batchno0-0-null-1597814047953.snappy.carbondata]
 to [hdfs://beh/user/dc_cbss/warehouse/yutest/tf_b_trade/stage_data] start.
2020-08-19 13:14:13,329 INFO org.apache.carbondata.core.util.CarbonUtil - 
Copying 
/home/hadoop/yutest/d963e9836ccb4318aa8fc953af983d07/part-0-a132f98547584dcabae6c43090626baf_batchno0-0-null-1597814047953.snappy.carbondata
 to hdfs://beh/user/dc_cbss/warehouse/yutest/tf_b_trade/stage_data, operation 
id 1597814053329
2020-08-19 13:14:13,331 DEBUG org.apache.carbondata.core.util.CarbonUtil 
-{color:#FF} The configured block size is 1 KB, the actual carbon file size 
is 277 KB, choose the max value 277 KB as the block size on HDFS{color}
2020-08-19 13:14:13,331 DEBUG org.apache.carbondata.core.util.CarbonUtil - HDFS 
file block size for file: 
hdfs://beh/user/dc_cbss/warehouse/yutest/tf_b_trade/stage_data/part-0-a132f98547584dcabae6c43090626baf_batchno0-0-null-1597814047953.snappy.carbondata
 is 284160 (bytes
2020-08-19 13:14:13,332 INFO 
org.apache.carbondata.processing.util.CarbonLoaderUtil - Deleted the local 
store location: 
/tmp/f97548ae6efc43d2ba269c9d35295bb9_attempt_f229b922-1f77-426f-a4bc-42e49aa53df7__m_1968253873_-1049302646
 : Time taken: 2
2020-08-19 13:14:13,358 ERROR org.apache.carbon.flink.CarbonS3Writer - Problem 
while copying file from local store to carbon store
org.apache.carbondata.core.datastore.exception.CarbonDataWriterException: 
Problem while copying file from local store to carbon store
 at 
org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2694)
 at 
org.apache.carbon.flink.CarbonWriter.uploadSegmentDataFiles(CarbonWriter.java:90)
 at org.apache.carbon.flink.CarbonLocalWriter.commit(CarbonLocalWriter.java:155)
 at org.apache.carbon.flink.CarbonLocalWriter.flush(CarbonLocalWriter.java:129)
 at 
org.apache.flink.streaming.api.functions.sink.filesystem.BulkPartWriter.closeForCommit(BulkPartWriter.java:61)
 at 
org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.closePartFile(Bucket.java:239)
 at 
org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.prepareBucketForCheckpointing(Bucket.java:280)
 at 
org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.onReceptionOfCheckpoint(Bucket.java:253)
 at 
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.snapshotActiveBuckets(Buckets.java:250)
 at 
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.snapshotState(Buckets.java:241)
 at 
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.snapshotState(StreamingFileSink.java:447)
 at 
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
 at 
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
 at 
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:90)
 at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:402)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1403)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1337)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:974)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$5(StreamTask.java:870)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:843)
 at