[
https://issues.apache.org/jira/browse/CARBONDATA-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146837#comment-16146837
]
Zhichao Zhang commented on CARBONDATA-1421:
--------------------------------------------
Hi Pallavi Singh, I run below code and it worked with pr-1266,
My config:
carbon.enable.auto.load.merge=true
carbon.compaction.level.threshold=2,4
My code:
{code:java}
import spark.implicits._
val df1 = spark.sparkContext.parallelize(0 to 50)
.map(x => ("a", x.toString(), (x % 2).toString(), x, x.toLong, x * 2))
.toDF("stringField1", "stringField2", "stringField3", "intField",
"longField", "int2Field")
val df2 = spark.sparkContext.parallelize(51 to 100)
.map(x => ("b", x.toString(), (x % 2).toString(), x, x.toLong, x * 2))
.toDF("stringField1", "stringField2", "stringField3", "intField",
"longField", "int2Field")
val df3 = df1.union(df2)
spark.sql("DROP TABLE IF EXISTS study_carbondata ").show()
spark.sql("""
| CREATE TABLE IF NOT EXISTS study_carbondata (
| stringField1 string,
| stringField2 string,
| stringField3 string,
| intField int,
| longField bigint,
| int2Field int
| )
| STORED BY 'carbondata'
| TBLPROPERTIES(
| 'TABLE_BLOCKSIZE'='8'
| )
""".stripMargin)
df3.write
.format("carbondata")
.option("tableName", "study_carbondata")
.option("compress", "true") // just valid when tempCSV is true
.option("tempCSV", "false")
.option("single_pass", "true")
.option("sort_scope", "LOCAL_SORT") //GLOBAL_SORT LOCAL_SORT
.mode(SaveMode.Append)
.save()
df3.write
.format("carbondata")
.option("tableName", "study_carbondata")
.option("compress", "true") // just valid when tempCSV is true
.option("tempCSV", "false")
.option("single_pass", "true")
.option("sort_scope", "LOCAL_SORT") //GLOBAL_SORT LOCAL_SORT
.mode(SaveMode.Append)
.save()
{code}
> Auto Compaction Failing in CarbonData Loading
> ---------------------------------------------
>
> Key: CARBONDATA-1421
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1421
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
> Affects Versions: 1.2.0
> Reporter: Pallavi Singh
> Fix For: 1.2.0
>
>
> I ran the create query followed by multiple load queries and the
> auto-compaction is failing.
> 0: jdbc:hive2://localhost:10000> LOAD DATA inpath
> 'hdfs://localhost:54310/data/4000_UniqData.csv' INTO table uniqdata
> options('DELIMITER'=',', 'FILEHEADER'='CUST_ID, CUST_NAME,
> ACTIVE_EMUI_VERSION, DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2,
> DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1, Double_COLUMN2,
> INTEGER_COLUMN1');
> +---------+--+
> | Result |
> +---------+--+
> +---------+--+
> No rows selected (1.183 seconds)
> 0: jdbc:hive2://localhost:10000> LOAD DATA inpath
> 'hdfs://localhost:54310/data/5000_UniqData.csv' INTO table uniqdata
> options('DELIMITER'=',', 'FILEHEADER'='CUST_ID, CUST_NAME,
> ACTIVE_EMUI_VERSION, DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2,
> DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1, Double_COLUMN2,
> INTEGER_COLUMN1');
> Error: java.lang.Exception: Dataload is success. Auto-Compaction has failed.
> Please check logs. (state=,code=0)
> 0: jdbc:hive2://localhost:10000> LOAD DATA inpath
> 'hdfs://localhost:54310/data/7000_UniqData.csv' INTO table uniqdata
> options('DELIMITER'=',', 'FILEHEADER'='CUST_ID, CUST_NAME,
> ACTIVE_EMUI_VERSION, DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2,
> DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1, Double_COLUMN2,
> INTEGER_COLUMN1');
> Error: java.lang.Exception: Dataload is success. Auto-Compaction has failed.
> Please check logs. (state=,code=0)
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000>
> 0: jdbc:hive2://localhost:10000> show segments for table uniqdata;
> +--------------------+----------+--------------------------+--------------------------+--+
> | SegmentSequenceId | Status | Load Start Time | Load End
> Time |
> +--------------------+----------+--------------------------+--------------------------+--+
> | 4 | Success | 2017-08-29 10:37:13.053 | 2017-08-29
> 10:37:13.888 |
> | 3 | Success | 2017-08-29 10:36:57.851 | 2017-08-29
> 10:36:59.08 |
> | 2 | Success | 2017-08-29 10:36:49.439 | 2017-08-29
> 10:36:50.373 |
> | 1 | Success | 2017-08-29 10:36:37.365 | 2017-08-29
> 10:36:38.768 |
> | 0 | Success | 2017-08-29 10:36:21.011 | 2017-08-29
> 10:36:26.1 |
> +--------------------+----------+--------------------------+--------------------------+--+
> 5 rows selected (0.099 seconds)
> 0: jdbc:hive2://localhost:10000> LOAD DATA inpath
> 'hdfs://localhost:54310/data/7000_UniqData.csv' INTO table uniqdata
> options('DELIMITER'=',', 'FILEHEADER'='CUST_ID, CUST_NAME,
> ACTIVE_EMUI_VERSION, DOB, DOJ, BIGINT_COLUMN1, BIGINT_COLUMN2,
> DECIMAL_COLUMN1, DECIMAL_COLUMN2, Double_COLUMN1, Double_COLUMN2,
> INTEGER_COLUMN1');
> Error: java.lang.Exception: Dataload is success. Auto-Compaction has failed.
> Please check logs. (state=,code=0)
> 0: jdbc:hive2://localhost:10000> show segments for table uniqdata;
> +--------------------+----------+--------------------------+--------------------------+--+
> | SegmentSequenceId | Status | Load Start Time | Load End
> Time |
> +--------------------+----------+--------------------------+--------------------------+--+
> | 5 | Success | 2017-08-29 10:38:15.727 | 2017-08-29
> 10:38:16.548 |
> | 4 | Success | 2017-08-29 10:37:13.053 | 2017-08-29
> 10:37:13.888 |
> | 3 | Success | 2017-08-29 10:36:57.851 | 2017-08-29
> 10:36:59.08 |
> | 2 | Success | 2017-08-29 10:36:49.439 | 2017-08-29
> 10:36:50.373 |
> | 1 | Success | 2017-08-29 10:36:37.365 | 2017-08-29
> 10:36:38.768 |
> | 0 | Success | 2017-08-29 10:36:21.011 | 2017-08-29
> 10:36:26.1 |
> +--------------------+----------+--------------------------+--------------------------+--+
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
