[carbondata] branch master updated: [CARBONDATA-3631] StringIndexOutOfBoundsException When Inserting Select From a Parquet Table with Empty array/map

[jackylk]

This is an automated email from the ASF dual-hosted git repository.

jackylk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git


The following commit(s) were added to refs/heads/master by this push:
 new 2ec9009  [CARBONDATA-3631] StringIndexOutOfBoundsException When 
Inserting Select From a Parquet Table with Empty array/map
2ec9009 is described below

commit 2ec90099595d6e5a7de4106d2f99b8725e4993ef
Author: h00424960 
AuthorDate: Mon Dec 30 07:40:13 2019 +0800

[CARBONDATA-3631] StringIndexOutOfBoundsException When Inserting Select 
From a Parquet Table with Empty array/map

Modification reason:
(1) StringIndexOutOfBoundsException When Inserting Select From a Parquet 
Table with Empty array/map.
(2) ArrayIndexOutOfBuoundsException When Inserting Select From a Parquet 
Table with a map with empty key and empty value
(3) Result is incorrect when Inserting Select From a Parquet Table with a 
Struct with Empty String, The result will be null while the correct result is 
"".

Modification content:
(1) When the input value is ARRAY(), return EMPTY_DATA_RETURN in the 
FieldConverter.scala, ArrayParserImpl handle it.
(2) When the input value is ARRAY(""), return EMPTY STRING ->"" in the 
FieldConverter.scala, ArrayParserImpl handle it.
(3) When the input value is MAP("",""), return EMPTY STRING ->"" in the 
FieldConverter.scala, MapParserImpl handle it.
(4) When the input value is MAP(), return EMPTY_DATA_RETURN ->"" in the 
FieldConverter.scala, MapParserImpl handle it.

This closes #3545
---
 .../core/constants/CarbonCommonConstants.java  |  9 +++
 .../primitiveTypes/ArrayDataTypeTestCase.scala | 61 ++
 .../primitiveTypes/MapDataTypeTestCase.scala   | 72 ++
 .../loading/parser/impl/ArrayParserImpl.java   |  8 ++-
 .../loading/parser/impl/MapParserImpl.java | 16 -
 .../streaming/parser/FieldConverter.scala  | 48 +--
 6 files changed, 191 insertions(+), 23 deletions(-)

diff --git 
a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
 
b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
index fa88027..9ad276c 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
@@ -294,6 +294,15 @@ public final class CarbonCommonConstants {
 
   public static final String CARBON_SKIP_EMPTY_LINE_DEFAULT = "false";
 
+
+  /**
+   *In the write processing, data needs to be converted to string, where the 
string forms of
+   * array("") and array() are EMPTY_STRING, causing confusion about the write 
results.
+   * In order to distinguish between array and array(), or map("") and map (), 
we need to
+   * identity array("") as the EMPTY_STRING, while array() is the 
SIZE_ZERO_DATA_RETURN
+   */
+  public static final String SIZE_ZERO_DATA_RETURN = 
"!LENGTH_ZERO_DATA_RETURN!";
+
   /**
* Currently the segment lock files are not deleted immediately when unlock,
* this value indicates the number of hours the segment lock files will be 
preserved.
diff --git 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/primitiveTypes/ArrayDataTypeTestCase.scala
 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/primitiveTypes/ArrayDataTypeTestCase.scala
new file mode 100644
index 000..fa5fe94
--- /dev/null
+++ 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/primitiveTypes/ArrayDataTypeTestCase.scala
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.integration.spark.testsuite.primitiveTypes
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+/**
+ * Test Class for filter query on Float datatypes
+ */
+class ArrayDataTypeTestCase extends QueryTest with BeforeAndAfterAll {
+
+  override def beforeAll {
+sql("DROP TABLE IF EXISTS 

Jenkins build is still unstable: carbondata-master-spark-2.1 » Apache CarbonData :: Store SDK #3942

[Apache Jenkins Server]

See 

Jenkins build is still unstable: carbondata-master-spark-2.1 » Apache CarbonData :: Spark Common Test #3942

[Apache Jenkins Server]

See 

Jenkins build is back to stable : carbondata-master-spark-2.1 » Apache CarbonData :: Processing #3942

[Apache Jenkins Server]

See 

Jenkins build is still unstable: carbondata-master-spark-2.2 » Apache CarbonData :: Store SDK #2163

[Apache Jenkins Server]

See 

Jenkins build is still unstable: carbondata-master-spark-2.2 » Apache CarbonData :: Spark Common Test #2163

[Apache Jenkins Server]

See 

Build failed in Jenkins: carbondata-master-spark-2.1 #3940

[Apache Jenkins Server]

See 


Changes:

[QiangCai] [CARBONATA-3605] Remove global dictionary feature


--
[...truncated 10.47 MB...]
2019-12-30 11:36:28 AUDIT audit:93 - {"time":"December 30, 2019 3:36:28 AM 
PST","username":"jenkins","opName":"CREATE 
DATAMAP","opId":"10892901309885161","opStatus":"SUCCESS","opTime":"216 
ms","table":"partition_mv.partitionone","extraInfo":{"provider":"mv","dmName":"dm1"}}
- test partition at last column *** FAILED ***
  Results do not match for query:
  == Parsed Logical Plan ==
  'Aggregate ['c], ['c, unresolvedalias('sum('b), None)]
  +- 'UnresolvedRelation `partitionone`
  
  == Analyzed Logical Plan ==
  c: int, sum(b): bigint
  Aggregate [partitionone_c#49375], [partitionone_c#49375 AS c#49380, 
sum(sum_b#49376L) AS sum(b)#49381L]
  +- SubqueryAlias dm1_table
 +- Relation[partitionone_c#49375,sum_b#49376L] 
CarbonDatasourceHadoopRelation
  
  == Optimized Logical Plan ==
  CarbonDictionaryCatalystDecoder [partition_mv_dm1_table], 
ExcludeProfile(ArrayBuffer()), CarbonAliasDecoderRelation(), true
  +- Aggregate [partitionone_c#49375], [partitionone_c#49375 AS c#49380, 
sum(sum_b#49376L) AS sum(b)#49381L]
 +- Relation[partitionone_c#49375,sum_b#49376L] 
CarbonDatasourceHadoopRelation
  
  == Physical Plan ==
  *HashAggregate(keys=[partitionone_c#49375], 
functions=[sum(sum_b#49376L)], output=[c#49380, sum(b)#49381L])
  +- Exchange hashpartitioning(partitionone_c#49375, 200)
 +- *HashAggregate(keys=[partitionone_c#49375], 
functions=[partial_sum(sum_b#49376L)], output=[partitionone_c#49375, 
sum#49398L])
+- *FileScan carbondata 
partition_mv.dm1_table[partitionone_c#49375,sum_b#49376L] ReadSchema: 
struct
  == Results ==
  !== Correct Answer - 1 ==   == Spark Answer - 0 ==
  ![3,2] (QueryTest.scala:90)
2019-12-30 11:36:28 AUDIT audit:72 - {"time":"December 30, 2019 3:36:28 AM 
PST","username":"jenkins","opName":"DROP 
TABLE","opId":"10892901956993791","opStatus":"START"}
2019-12-30 11:36:28 AUDIT audit:93 - {"time":"December 30, 2019 3:36:28 AM 
PST","username":"jenkins","opName":"DROP 
TABLE","opId":"10892901956993791","opStatus":"SUCCESS","opTime":"219 
ms","table":"partition_mv.partitionone","extraInfo":{}}
2019-12-30 11:36:28 AUDIT audit:72 - {"time":"December 30, 2019 3:36:28 AM 
PST","username":"jenkins","opName":"CREATE 
TABLE","opId":"10892902178188578","opStatus":"START"}
2019-12-30 11:36:28 AUDIT audit:93 - {"time":"December 30, 2019 3:36:28 AM 
PST","username":"jenkins","opName":"CREATE 
TABLE","opId":"10892902178188578","opStatus":"SUCCESS","opTime":"54 
ms","table":"partition_mv.partitionone","extraInfo":{"bad_record_path":"","local_dictionary_enable":"true","external":"false","sort_columns":"","comment":""}}
2019-12-30 11:36:28 AUDIT audit:72 - {"time":"December 30, 2019 3:36:28 AM 
PST","username":"jenkins","opName":"INSERT 
INTO","opId":"10892902327167460","opStatus":"START"}
2019-12-30 11:36:29 AUDIT audit:93 - {"time":"December 30, 2019 3:36:29 AM 
PST","username":"jenkins","opName":"INSERT 
INTO","opId":"10892902327167460","opStatus":"SUCCESS","opTime":"364 
ms","table":"partition_mv.partitionone","extraInfo":{}}
2019-12-30 11:36:29 AUDIT audit:72 - {"time":"December 30, 2019 3:36:29 AM 
PST","username":"jenkins","opName":"DROP 
DATAMAP","opId":"10892902693828831","opStatus":"START"}
2019-12-30 11:36:29 AUDIT audit:93 - {"time":"December 30, 2019 3:36:29 AM 
PST","username":"jenkins","opName":"DROP 
DATAMAP","opId":"10892902693828831","opStatus":"SUCCESS","opTime":"4 
ms","table":"NA","extraInfo":{"dmName":"dm1"}}
2019-12-30 11:36:29 AUDIT audit:72 - {"time":"December 30, 2019 3:36:29 AM 
PST","username":"jenkins","opName":"CREATE 
DATAMAP","opId":"10892902701996036","opStatus":"START"}
2019-12-30 11:36:29 AUDIT audit:72 - {"time":"December 30, 2019 3:36:29 AM 
PST","username":"jenkins","opName":"CREATE 
TABLE","opId":"10892902722999072","opStatus":"START"}
2019-12-30 11:36:29 AUDIT audit:93 - {"time":"December 30, 2019 3:36:29 AM 
PST","username":"jenkins","opName":"CREATE 
TABLE","opId":"10892902722999072","opStatus":"SUCCESS","opTime":"50 
ms","table":"partition_mv.dm1_table","extraInfo":{"local_dictionary_threshold":"1","bad_record_path":"","table_blocksize":"1024","local_dictionary_enable":"true","flat_folder":"false","external":"false","parent_tables":"partitionone","sort_columns":"","comment":"","_internal.deferred.rebuild":"false","carbon.column.compressor":"snappy","datamap_name":"dm1"}}
2019-12-30 11:36:29 AUDIT audit:93 - {"time":"December 30, 2019 3:36:29 AM 
PST","username":"jenkins","opName":"CREATE 
DATAMAP","opId":"10892902701996036","opStatus":"SUCCESS","opTime":"225 

Jenkins build is still unstable: carbondata-master-spark-2.1 » Apache CarbonData :: Store SDK #3940

[Apache Jenkins Server]

See 

Jenkins build is unstable: carbondata-master-spark-2.1 » Apache CarbonData :: Spark Common Test #3940

[Apache Jenkins Server]

See 

Build failed in Jenkins: carbondata-master-spark-2.1 #3941

[Apache Jenkins Server]

See 


Changes:

[QiangCai] [CARBONATA-3605] Remove global dictionary feature


--
[...truncated 10.46 MB...]
  Results do not match for query:
  == Parsed Logical Plan ==
  'Aggregate ['c], ['c, unresolvedalias('sum('b), None)]
  +- 'UnresolvedRelation `partitionone`
  
  == Analyzed Logical Plan ==
  c: int, sum(b): bigint
  Aggregate [partitionone_c#49375], [partitionone_c#49375 AS c#49380, 
sum(sum_b#49376L) AS sum(b)#49381L]
  +- SubqueryAlias dm1_table
 +- Relation[partitionone_c#49375,sum_b#49376L] 
CarbonDatasourceHadoopRelation
  
  == Optimized Logical Plan ==
  CarbonDictionaryCatalystDecoder [partition_mv_dm1_table], 
ExcludeProfile(ArrayBuffer()), CarbonAliasDecoderRelation(), true
  +- Aggregate [partitionone_c#49375], [partitionone_c#49375 AS c#49380, 
sum(sum_b#49376L) AS sum(b)#49381L]
 +- Relation[partitionone_c#49375,sum_b#49376L] 
CarbonDatasourceHadoopRelation
  
  == Physical Plan ==
  *HashAggregate(keys=[partitionone_c#49375], 
functions=[sum(sum_b#49376L)], output=[c#49380, sum(b)#49381L])
  +- Exchange hashpartitioning(partitionone_c#49375, 200)
 +- *HashAggregate(keys=[partitionone_c#49375], 
functions=[partial_sum(sum_b#49376L)], output=[partitionone_c#49375, 
sum#49398L])
+- *FileScan carbondata 
partition_mv.dm1_table[partitionone_c#49375,sum_b#49376L] ReadSchema: 
struct
  == Results ==
  !== Correct Answer - 1 ==   == Spark Answer - 0 ==
  ![3,2] (QueryTest.scala:90)
2019-12-30 11:36:00 AUDIT audit:72 - {"time":"December 30, 2019 3:36:00 AM 
PST","username":"jenkins","opName":"DROP 
TABLE","opId":"10892874100188015","opStatus":"START"}
2019-12-30 11:36:00 AUDIT audit:93 - {"time":"December 30, 2019 3:36:00 AM 
PST","username":"jenkins","opName":"DROP 
TABLE","opId":"10892874100188015","opStatus":"SUCCESS","opTime":"234 
ms","table":"partition_mv.partitionone","extraInfo":{}}
2019-12-30 11:36:00 AUDIT audit:72 - {"time":"December 30, 2019 3:36:00 AM 
PST","username":"jenkins","opName":"CREATE 
TABLE","opId":"10892874337763735","opStatus":"START"}
2019-12-30 11:36:00 AUDIT audit:93 - {"time":"December 30, 2019 3:36:00 AM 
PST","username":"jenkins","opName":"CREATE 
TABLE","opId":"10892874337763735","opStatus":"SUCCESS","opTime":"40 
ms","table":"partition_mv.partitionone","extraInfo":{"bad_record_path":"","local_dictionary_enable":"true","external":"false","sort_columns":"","comment":""}}
2019-12-30 11:36:01 AUDIT audit:72 - {"time":"December 30, 2019 3:36:01 AM 
PST","username":"jenkins","opName":"INSERT 
INTO","opId":"10892874468616719","opStatus":"START"}
2019-12-30 11:36:01 AUDIT audit:93 - {"time":"December 30, 2019 3:36:01 AM 
PST","username":"jenkins","opName":"INSERT 
INTO","opId":"10892874468616719","opStatus":"SUCCESS","opTime":"339 
ms","table":"partition_mv.partitionone","extraInfo":{}}
2019-12-30 11:36:01 AUDIT audit:72 - {"time":"December 30, 2019 3:36:01 AM 
PST","username":"jenkins","opName":"DROP 
DATAMAP","opId":"10892874809473547","opStatus":"START"}
2019-12-30 11:36:01 AUDIT audit:93 - {"time":"December 30, 2019 3:36:01 AM 
PST","username":"jenkins","opName":"DROP 
DATAMAP","opId":"10892874809473547","opStatus":"SUCCESS","opTime":"5 
ms","table":"NA","extraInfo":{"dmName":"dm1"}}
2019-12-30 11:36:01 AUDIT audit:72 - {"time":"December 30, 2019 3:36:01 AM 
PST","username":"jenkins","opName":"CREATE 
DATAMAP","opId":"10892874817755331","opStatus":"START"}
2019-12-30 11:36:01 AUDIT audit:72 - {"time":"December 30, 2019 3:36:01 AM 
PST","username":"jenkins","opName":"CREATE 
TABLE","opId":"10892874839638505","opStatus":"START"}
2019-12-30 11:36:01 AUDIT audit:93 - {"time":"December 30, 2019 3:36:01 AM 
PST","username":"jenkins","opName":"CREATE 
TABLE","opId":"10892874839638505","opStatus":"SUCCESS","opTime":"51 
ms","table":"partition_mv.dm1_table","extraInfo":{"local_dictionary_threshold":"1","bad_record_path":"","table_blocksize":"1024","local_dictionary_enable":"true","flat_folder":"false","external":"false","parent_tables":"partitionone","sort_columns":"","comment":"","_internal.deferred.rebuild":"false","carbon.column.compressor":"snappy","datamap_name":"dm1"}}
2019-12-30 11:36:01 AUDIT audit:93 - {"time":"December 30, 2019 3:36:01 AM 
PST","username":"jenkins","opName":"CREATE 
DATAMAP","opId":"10892874817755331","opStatus":"SUCCESS","opTime":"243 
ms","table":"partition_mv.partitionone","extraInfo":{"provider":"mv","dmName":"dm1"}}
- test partition on timeseries column *** FAILED ***
  0 did not equal 1 (TestPartitionWithMV.scala:695)
2019-12-30 11:36:02 AUDIT audit:72 - {"time":"December 30, 2019 3:36:02 AM 
PST","username":"jenkins","opName":"DROP 

Jenkins build is still unstable: carbondata-master-spark-2.1 » Apache CarbonData :: Store SDK #3941

[Apache Jenkins Server]

See 

Jenkins build is still unstable: carbondata-master-spark-2.1 » Apache CarbonData :: Spark Common Test #3941

[Apache Jenkins Server]

See 

Build failed in Jenkins: carbondata-master-spark-2.1 » Apache CarbonData :: Materialized View Core #3941

[Apache Jenkins Server]

See 


Changes:

[QiangCai] [CARBONATA-3605] Remove global dictionary feature


--
[...truncated 1.14 MB...]
2019-12-30 11:35:49 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:49 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:49 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:49 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:50 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:50 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:50 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:50 ERROR GlobalSortHelper$:45 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:50 AUDIT audit:93 - {"time":"December 30, 2019 3:35:50 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892862735201245","opStatus":"SUCCESS","opTime":"1022 
ms","table":"partition_mv.partitionallcompaction","extraInfo":{}}
2019-12-30 11:35:50 AUDIT audit:72 - {"time":"December 30, 2019 3:35:50 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892863759961330","opStatus":"START"}
2019-12-30 11:35:50 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:50 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:50 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:50 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:50 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:51 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:51 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:51 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:51 ERROR GlobalSortHelper$:45 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:51 AUDIT audit:93 - {"time":"December 30, 2019 3:35:51 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892863759961330","opStatus":"SUCCESS","opTime":"949 
ms","table":"partition_mv.partitionallcompaction","extraInfo":{}}
2019-12-30 11:35:51 AUDIT audit:72 - {"time":"December 30, 2019 3:35:51 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892864712355148","opStatus":"START"}
2019-12-30 11:35:51 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:51 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:51 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:51 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:51 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:51 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:52 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:52 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:52 ERROR GlobalSortHelper$:45 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:52 AUDIT audit:93 - {"time":"December 30, 2019 3:35:52 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892864712355148","opStatus":"SUCCESS","opTime":"944 
ms","table":"partition_mv.partitionallcompaction","extraInfo":{}}
2019-12-30 11:35:52 AUDIT audit:72 - {"time":"December 30, 2019 3:35:52 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892865659643777","opStatus":"START"}
2019-12-30 11:35:52 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:52 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:52 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:35:52 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table 

Jenkins build became unstable: carbondata-master-spark-2.1 » Apache CarbonData :: Processing #3941

[Apache Jenkins Server]

See 

Build failed in Jenkins: carbondata-master-spark-2.1 » Apache CarbonData :: Materialized View Core #3940

[Apache Jenkins Server]

See 


Changes:

[QiangCai] [CARBONATA-3605] Remove global dictionary feature


--
[...truncated 1.15 MB...]
2019-12-30 11:36:17 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:17 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:17 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:17 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:17 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:17 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:17 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:17 ERROR GlobalSortHelper$:45 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:17 AUDIT audit:93 - {"time":"December 30, 2019 3:36:17 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892890389529349","opStatus":"SUCCESS","opTime":"928 
ms","table":"partition_mv.partitionallcompaction","extraInfo":{}}
2019-12-30 11:36:17 AUDIT audit:72 - {"time":"December 30, 2019 3:36:17 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892891320819696","opStatus":"START"}
2019-12-30 11:36:18 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:18 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:18 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:18 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:18 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:18 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:18 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:18 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:18 ERROR GlobalSortHelper$:45 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:18 AUDIT audit:93 - {"time":"December 30, 2019 3:36:18 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892891320819696","opStatus":"SUCCESS","opTime":"975 
ms","table":"partition_mv.partitionallcompaction","extraInfo":{}}
2019-12-30 11:36:18 AUDIT audit:72 - {"time":"December 30, 2019 3:36:18 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892892299049520","opStatus":"START"}
2019-12-30 11:36:19 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:19 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:19 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:19 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:19 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:19 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:19 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:19 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:19 ERROR GlobalSortHelper$:45 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:19 AUDIT audit:93 - {"time":"December 30, 2019 3:36:19 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892892299049520","opStatus":"SUCCESS","opTime":"1019 
ms","table":"partition_mv.partitionallcompaction","extraInfo":{}}
2019-12-30 11:36:19 AUDIT audit:72 - {"time":"December 30, 2019 3:36:19 AM 
PST","username":"jenkins","opName":"LOAD DATA 
OVERWRITE","opId":"10892893320922064","opStatus":"START"}
2019-12-30 11:36:20 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:20 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:20 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table partitionallcompaction
2019-12-30 11:36:20 ERROR DataLoadExecutor:55 - Data Load is partially success 
for table 

Jenkins build is still unstable: carbondata-master-spark-2.2 » Apache CarbonData :: Store SDK #2162

[Apache Jenkins Server]

See 

Jenkins build is unstable: carbondata-master-spark-2.2 » Apache CarbonData :: Spark Common Test #2162

[Apache Jenkins Server]

See 

Jenkins build is still unstable: carbondata-master-spark-2.2 » Apache CarbonData :: Store SDK #2161

[Apache Jenkins Server]

See 

Jenkins build is still unstable: carbondata-master-spark-2.1 » Apache CarbonData :: Store SDK #3939

[Apache Jenkins Server]

See 

Jenkins build is still unstable: carbondata-master-spark-2.2 » Apache CarbonData :: Store SDK #2160

[Apache Jenkins Server]

See 

Jenkins build is still unstable: carbondata-master-spark-2.2 » Apache CarbonData :: Spark Common Test #2160

[Apache Jenkins Server]

See 

[carbondata] branch master updated: [DOC][FAQ] add faq for how to deal with slow task

[jackylk]

This is an automated email from the ASF dual-hosted git repository.

jackylk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git


The following commit(s) were added to refs/heads/master by this push:
 new 1ded13e  [DOC][FAQ] add faq for how to deal with slow task
1ded13e is described below

commit 1ded13efa0a00f9b04b0714292aedc738b2f2d8d
Author: litao 
AuthorDate: Wed Dec 18 20:25:23 2019 +0800

[DOC][FAQ] add faq for how to deal with slow task

This closes #3514
---
 docs/faq.md | 24 
 1 file changed, 24 insertions(+)

diff --git a/docs/faq.md b/docs/faq.md
index 9ba7082..16cdfa5 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -29,6 +29,7 @@
 * [Why all executors are showing success in Spark UI even after Dataload 
command failed at Driver 
side?](#why-all-executors-are-showing-success-in-spark-ui-even-after-dataload-command-failed-at-driver-side)
 * [Why different time zone result for select query output when query SDK 
writer 
output?](#why-different-time-zone-result-for-select-query-output-when-query-sdk-writer-output)
 * [How to check LRU cache memory 
footprint?](#how-to-check-lru-cache-memory-footprint)
+* [How to deal with the trailing task in 
query?](#How-to-deal-with-the-trailing-task-in-query)
 
 # TroubleShooting
 
@@ -227,6 +228,29 @@ This property will enable the DEBUG log for the 
CarbonLRUCache and UnsafeMemoryM
 **Note:** If  `Removed entry from InMemory LRU cache` are frequently observed 
in logs, you may have to increase the configured LRU size.
 
 To observe the LRU cache from heap dump, check the heap used by CarbonLRUCache 
class.
+
+## How to deal with the trailing task in query?
+
+When tuning query performance, user may found that a few tasks slow down the 
overall query progress.  To improve performance in such case, user can set 
spark.locality.wait and spark.speculation=true to enable speculation in spark, 
which will launch multiple task and get the result the one of the task which is 
finished first. Besides, user can also consider following configurations to 
further improve performance in this case.
+
+**Example:**
+
+```
+spark.locality.wait = 500
+spark.speculation = true
+spark.speculation.quantile = 0.75
+spark.speculation.multiplier = 5
+spark.blacklist.enabled = false
+```
+
+**Note:** 
+
+spark.locality control data locality the value of 500 is used to shorten the 
waiting time of spark. 
+
+spark.speculation is a group of configuration, that can monitor trailing tasks 
and start new tasks when conditions are met.
+
+spark.blacklist.enabled, avoid reduction of available executors due to 
blacklist mechanism.
+
 ## Getting tablestatus.lock issues When loading data
 
   **Symptom**