GitHub user xubo245 opened a pull request:

    https://github.com/apache/carbondata/pull/1963

    [WIP][CARBONDATA-2153]  Failed to update table status for pre-aggregate 
table when maintable load twice

    
    
     // TODO: to be fixed
      test("test whether all segments are loaded into pre-aggregate table: 
error in auto merge and input segment") {
        CarbonProperties.getInstance()
          .addProperty(CarbonCommonConstants.ENABLE_AUTO_LOAD_MERGE, "true")
        sql("reset")
        sql("DROP TABLE IF EXISTS main_table")
        sql(
          """
            | CREATE TABLE main_table(
            |     id INT,
            |     name STRING,
            |     city STRING,
            |     age INT)
            | STORED BY 'org.apache.carbondata.format'
          """.stripMargin)
        sql(s"INSERT INTO main_table VALUES(1, 'xyz', 'bengaluru', 26)")
        sql(s"INSERT INTO main_table VALUES(1, 'xyz', 'bengaluru', 26)")
    
        sql(
          s"""
             | CREATE DATAMAP preagg_sum
             | ON TABLE main_table
             | USING 'preaggregate'
             | AS SELECT id, SUM(age)
             | FROM main_table
             | GROUP BY id
           """.stripMargin)
    
    
        sql(s"INSERT INTO main_table VALUES(1, 'xyz', 'bengaluru', 26)")
        sql("show segments for table main_table_preagg_sum").show()
        sql(s"LOAD DATA LOCAL INPATH '$testData' INTO TABLE main_table")
        sql(s"LOAD DATA LOCAL INPATH '$testData' INTO TABLE main_table")
    
        sql("show segments for table main_table_preagg_sum").show()
        checkExistence(sql("show segments for table main_table_preagg_sum"), 
true, "Compacted")
        CarbonProperties.getInstance()
          .addProperty(CarbonCommonConstants.ENABLE_AUTO_LOAD_MERGE, "false")
      }
    ```
    committing data maps
        java.lang.RuntimeException: Failed to update table status for 
pre-aggregate table
                at scala.sys.package$.error(package.scala:27)
                at 
org.apache.spark.sql.execution.command.preaaggregate.CommitPreAggregateListener$.onEvent(PreAggregateListeners.scala:317)
                at 
org.apache.carbondata.events.OperationListenerBus.fireEvent(OperationListenerBus.java:117)
                at 
org.apache.carbondata.spark.rdd.CarbonTableCompactor.triggerCompaction(CarbonTableCompactor.scala:265)
                at 
org.apache.carbondata.spark.rdd.CarbonTableCompactor.scanSegmentsAndSubmitJob(CarbonTableCompactor.scala:123)
                at 
org.apache.carbondata.spark.rdd.CarbonTableCompactor.executeCompaction(CarbonTableCompactor.scala:74)
                at 
org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$$anon$2.run(CarbonDataRDDFactory.scala:172)
                at 
org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.startCompactionThreads(CarbonDataRDDFactory.scala:260)
                at 
org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.handleSegmentMerging(CarbonDataRDDFactory.scala:780)
                at 
org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.loadCarbonData(CarbonDataRDDFactory.scala:568)
                at 
org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand.loadData(CarbonLoadDataCommand.scala:485)
                at 
org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand.processData(CarbonLoadDataCommand.scala:261)
                at 
org.apache.spark.sql.execution.command.AtomicRunnableCommand.run(package.scala:92)
                at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
                at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
                at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:67)
                at org.apache.spark.sql.Dataset.<init>(Dataset.scala:183)
                at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:68)
                at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:632)
                at 
org.apache.spark.sql.test.Spark2TestQueryExecutor.sql(Spark2TestQueryExecutor.scala:35)
                at 
org.apache.spark.sql.test.util.QueryTest.sql(QueryTest.scala:113)
                at 
org.apache.carbondata.integration.spark.testsuite.preaggregate.TestPreAggregateLoad$$anonfun$35.apply$mcV$sp(TestPreAggregateLoad.scala:1196)
    ```
    
    
    and auto compaction fail:
    
    ```
    ScalaTestFailureLocation: 
org.apache.spark.sql.test.util.QueryTest$$anonfun$checkExistence$1 at 
(QueryTest.scala:59)
    org.scalatest.exceptions.TestFailedException: "3Success2018-02-09 
02:10:17.4422018-02-09 02:10:18.146NACOLUMNAR_V32Success2018-02-09 
02:10:15.8312018-02-09 02:10:16.756NACOLUMNAR_V31Success2018-02-09 
02:10:14.1842018-02-09 02:10:15.175NACOLUMNAR_V30Success2018-02-09 
02:10:11.0172018-02-09 02:10:13.773NACOLUMNAR_V3" did not contain "Compacted" 
Failed for [SegmentSequenceId: string, Status: string ... 4 more fields] 
(Compacted doesn't exist in result)
        at 
org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)
    
    ```
    Be sure to do all of the following checklist to help us incorporate 
    your contribution quickly and easily:
    
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
    
     - [ ] Testing done
            Please provide details on 
            - Whether new unit test cases have been added or why no new tests 
are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance 
test report.
            - Any additional information to help reviewers in testing this 
change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xubo245/carbondata 
CARBONDATA2153_FailedUpdateTable

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1963.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1963
    
----
commit 5481032c48fcd3c2b12ac3741fe553cf046efde1
Author: xubo245 <601450868@...>
Date:   2018-02-09T10:09:42Z

    [CARBONDATA-2153]  Failed to update table status for pre-aggregate table 
when maintable load twice

----


---

Reply via email to