[GitHub] carbondata issue #1962: [CARBONDATA-2149]Fix complex type data displaying er...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1962 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3693/ ---
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1967 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3692/ ---
[GitHub] carbondata issue #1966: [HOTFIX]Upgrade pom version from 1.3.0 to 1.4.0 in p...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1966 Again, thanks for your contribution. Can you please close this PR, look forward to seeing your other PRs. ---
[GitHub] carbondata pull request #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Ro...
Github user zzcclp commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1959#discussion_r167423204 --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala --- @@ -775,6 +776,8 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll { .option("bad_records_action", badRecordAction) .option("dbName", tableIdentifier.database.get) .option("tableName", tableIdentifier.table) +.option(CarbonStreamParser.CARBON_STREAM_PARSER, + "org.apache.carbondata.streaming.parser.CSVStreamParserImp") --- End diff -- After PR#1962 is merged, I will modify it , otherwise there is error when write complex type data. ---
[GitHub] carbondata issue #1962: [CARBONDATA-2149]Fix complex type data displaying er...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1962 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2453/ ---
[GitHub] carbondata issue #1772: [CARBONDATA-1995] Unify all writer steps and make te...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1772 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3689/ ---
[GitHub] carbondata issue #1458: [CARBONDATA-1663] Decouple spark and core modules
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1458 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3691/ ---
[jira] [Updated] (CARBONDATA-2157) It failed when running the pre-aggregate code again after stopping the running pre-aggregate code
[ https://issues.apache.org/jira/browse/CARBONDATA-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-2157: Description: It failed when running the pre-aggregate code again after stopping the running pre-aggregate code like debug failed, step: 1.create breakpoint 2. run to break point 3.stop 4.run again => fail was: It failed when running the pre-aggregate code again after stopping the running pre-aggregate code like debug failed > It failed when running the pre-aggregate code again after stopping the > running pre-aggregate code > - > > Key: CARBONDATA-2157 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2157 > Project: CarbonData > Issue Type: Bug > Components: core, spark-integration >Affects Versions: 1.3.0 >Reporter: xubo245 >Priority: Major > > It failed when running the pre-aggregate code again after stopping the > running pre-aggregate code > like debug failed, step: > 1.create breakpoint > 2. run to break point > 3.stop > 4.run again => fail -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #1808: [CARBONDATA-2023][DataLoad] Add size base block allo...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/1808 retest sdv please ---
[GitHub] carbondata issue #1792: [CARBONDATA-2018][DataLoad] Optimization in reading/...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/1792 retest sdv please ---
[GitHub] carbondata issue #1458: [CARBONDATA-1663] Decouple spark and core modules
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1458 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2451/ ---
[jira] [Created] (CARBONDATA-2158) Pre-aggregate table don't support 'AS SELECT * FROM MAINTABLE'
xubo245 created CARBONDATA-2158: --- Summary: Pre-aggregate table don't support 'AS SELECT * FROM MAINTABLE' Key: CARBONDATA-2158 URL: https://issues.apache.org/jira/browse/CARBONDATA-2158 Project: CarbonData Issue Type: Bug Components: spark-integration Affects Versions: 1.3.0 Reporter: xubo245 Pre-aggregate table don't support 'AS SELECT * FROM MAINTABLE' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1967 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2452/ ---
[jira] [Closed] (CARBONDATA-2085) It's different between load twice and create datamap with load again after load data and create datamap
[ https://issues.apache.org/jira/browse/CARBONDATA-2085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 closed CARBONDATA-2085. --- Resolution: Fixed > It's different between load twice and create datamap with load again after > load data and create datamap > --- > > Key: CARBONDATA-2085 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2085 > Project: CarbonData > Issue Type: Bug > Components: core, spark-integration >Affects Versions: 1.3.0 >Reporter: xubo245 >Priority: Major > Fix For: 1.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > It's different between two test case > test case 1: load twice and create datamap , and then query > test case 2:load once , create datamap and load again, and then query > {code:java} > + test("load data into mainTable after create timeseries datamap on table > 1") { > +sql("drop table if exists mainTable") > +sql( > + """ > +| CREATE TABLE mainTable( > +| mytime timestamp, > +| name string, > +| age int) > +| STORED BY 'org.apache.carbondata.format' > + """.stripMargin) > + > +sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into > table mainTable") > + > +sql( > + """ > +| create datamap agg0 on table mainTable > +| using 'preaggregate' > +| DMPROPERTIES ( > +| 'timeseries.eventTime'='mytime', > +| > 'timeseries.hierarchy'='second=1,minute=1,hour=1,day=1,month=1,year=1') > +| as select mytime, sum(age) > +| from mainTable > +| group by mytime""".stripMargin) > + > +sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into > table mainTable") > +val df = sql( > + """ > +| select > +| timeseries(mytime,'minute') as minuteLevel, > +| sum(age) as sum > +| from mainTable > +| where timeseries(mytime,'minute')>='2016-02-23 01:01:00' > +| group by > +| timeseries(mytime,'minute') > +| order by > +| timeseries(mytime,'minute') > + """.stripMargin) > + > +// only for test, it need remove before merge > +df.show() > +sql("select * from maintable_agg0_minute").show(100) > + > +checkAnswer(df, > + Seq(Row(Timestamp.valueOf("2016-02-23 01:01:00"), 120), > +Row(Timestamp.valueOf("2016-02-23 01:02:00"), 280))) > + > + } > + > + test("load data into mainTable after create timeseries datamap on table > 2") { > +sql("drop table if exists mainTable") > +sql( > + """ > +| CREATE TABLE mainTable( > +| mytime timestamp, > +| name string, > +| age int) > +| STORED BY 'org.apache.carbondata.format' > + """.stripMargin) > + > +sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into > table mainTable") > +sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into > table mainTable") > +sql( > + """ > +| create datamap agg0 on table mainTable > +| using 'preaggregate' > +| DMPROPERTIES ( > +| 'timeseries.eventTime'='mytime', > +| > 'timeseries.hierarchy'='second=1,minute=1,hour=1,day=1,month=1,year=1') > +| as select mytime, sum(age) > +| from mainTable > +| group by mytime""".stripMargin) > + > + > +val df = sql( > + """ > +| select > +| timeseries(mytime,'minute') as minuteLevel, > +| sum(age) as sum > +| from mainTable > +| where timeseries(mytime,'minute')>='2016-02-23 01:01:00' > +| group by > +| timeseries(mytime,'minute') > +| order by > +| timeseries(mytime,'minute') > + """.stripMargin) > + > +// only for test, it need remove before merge > +df.show() > +sql("select * from maintable_agg0_minute").show(100) > + > + > +checkAnswer(df, > + Seq(Row(Timestamp.valueOf("2016-02-23 01:01:00"), 120), > +Row(Timestamp.valueOf("2016-02-23 01:02:00"), 280))) > + } > + > {code} > test case 1 and 2 should success , but test case 1 fail -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #1772: [CARBONDATA-1995] Unify all writer steps and make te...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1772 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2449/ ---
[GitHub] carbondata issue #1390: [CARBONDATA-1360] fix for update table failures for ...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/1390 @ashwini-krishnakumar I met the same problem and raised a pr to fix this issue, please help to review [PR#1962](https://github.com/apache/carbondata/pull/1962) ---
[GitHub] carbondata issue #1571: [CARBONDATA-1811] Use StructType as schema when crea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1571 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3690/ ---
[GitHub] carbondata issue #1792: [CARBONDATA-2018][DataLoad] Optimization in reading/...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1792 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3688/ ---
[GitHub] carbondata issue #1571: [CARBONDATA-1811] Use StructType as schema when crea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1571 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2450/ ---
[GitHub] carbondata issue #1792: [CARBONDATA-2018][DataLoad] Optimization in reading/...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1792 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2448/ ---
[GitHub] carbondata issue #1798: [CARBONDATA-1995][CARBONDATA-1996] Support file leve...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1798 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3687/ ---
[GitHub] carbondata issue #1808: [CARBONDATA-2023][DataLoad] Add size base block allo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1808 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3686/ ---
[GitHub] carbondata issue #1808: [CARBONDATA-2023][DataLoad] Add size base block allo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1808 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2446/ ---
[GitHub] carbondata issue #1798: [CARBONDATA-1995][CARBONDATA-1996] Support file leve...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1798 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2447/ ---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1825 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3685/ ---
[GitHub] carbondata issue #1953: [CARBONDATA-2091][DataLoad] Support specifying sort ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1953 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3684/ ---
[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1825 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2445/ ---
[GitHub] carbondata issue #1953: [CARBONDATA-2091][DataLoad] Support specifying sort ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1953 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2444/ ---
[jira] [Updated] (CARBONDATA-2118) It throw error when select a1 after creating datamap uses like "as select a as a1,sum(b) from tablename"
[ https://issues.apache.org/jira/browse/CARBONDATA-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-2118: Summary: It throw error when select a1 after creating datamap uses like "as select a as a1,sum(b) from tablename" (was: It should throw error when create datamap uses like "as select a as a1,sum(b) from tablename") > It throw error when select a1 after creating datamap uses like "as select a > as a1,sum(b) from tablename" > > > Key: CARBONDATA-2118 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2118 > Project: CarbonData > Issue Type: Bug > Components: core, spark-integration >Affects Versions: 1.3.0 >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Fix For: 1.4.0 > > > After create datamap using > {code:java} > create datamap preagg2 on table PreAggMain using 'preaggregate' as select a > as a1,sum(b) from PreAggMain group by a > {code} > it will error when select a1: > {code:java} > sql("select a1 from PreAggMain_preagg2").show() > {code} > It should throw error when create datamap. > code: > {code:java} > test("test pre agg create table 2") { > sql("create datamap preagg2 on table PreAggMain using > 'preaggregate' as select a as a1,sum(b) from PreAggMain group by a") > checkExistence(sql("DESCRIBE FORMATTED PreAggMain_preagg2"), true, > "preaggmain_a") > checkExistence(sql("DESCRIBE FORMATTED PreAggMain_preagg2"), true, > "preaggmain_b_sum") > checkExistence(sql("DESCRIBE FORMATTED PreAggMain_preagg2"), false, > "preaggmain_a1") > sql("DESCRIBE FORMATTED PreAggMain_preagg2").show() > sql("select * from PreAggMain_preagg2").show() > sql("select a1 from PreAggMain_preagg2").show() > > sql("DESCRIBE FORMATTED PreAggMain").show() > sql("select * from PreAggMain").show() > sql("select a as a1,sum(b) from PreAggMain group by a").show() > > sql("drop datamap preagg2 on table PreAggMain") > } > {code} > error: > {code:java} > +++ > |preaggmain_a|preaggmain_b_sum| > +++ > +++ > > > cannot resolve '`a1`' given input columns: [preaggmain_a, > preaggmain_b_sum]; line 1 pos 7; > 'Project ['a1] > +- SubqueryAlias preaggmain_preagg2 > +- Relation[preaggmain_a#101,preaggmain_b_sum#102] > CarbonDatasourceHadoopRelation [ Database name :default, Table name > :preaggmain_preagg2, Schema > :Some(StructType(StructField(preaggmain_a,StringType,true), > StructField(preaggmain_b_sum,DoubleType,true))) ] > > org.apache.spark.sql.AnalysisException: cannot resolve '`a1`' given > input columns: [preaggmain_a, preaggmain_b_sum]; line 1 pos 7; > 'Project ['a1] > +- SubqueryAlias preaggmain_preagg2 > +- Relation[preaggmain_a#101,preaggmain_b_sum#102] > CarbonDatasourceHadoopRelation [ Database name :default, Table name > :preaggmain_preagg2, Schema > :Some(StructType(StructField(preaggmain_a,StringType,true), > StructField(preaggmain_b_sum,DoubleType,true))) ] > > at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:88) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:85) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2144) There are some improper place in pre-aggregate documentation
[ https://issues.apache.org/jira/browse/CARBONDATA-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-2144: Summary: There are some improper place in pre-aggregate documentation (was: Optimize pre-aggregate documentation) > There are some improper place in pre-aggregate documentation > > > Key: CARBONDATA-2144 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2144 > Project: CarbonData > Issue Type: Improvement > Components: docs >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > > Optimize pre-aggregate documentation: > * add blank space > * upper case > like: > Carbondata supports pre aggregating of data so that OLAP kind of queries can > fetch data much faster.Aggregate tables are created as datamaps so that the > handling is as efficient as other indexing support.Users can create as many > aggregate tables they require as datamaps to improve their query > performance,provided the storage requirements and loading speeds are > acceptable. > For main table called sales which is defined as > CREATE TABLE sales ( > order_time timestamp, > user_id string, > sex string, > country string, > quantity int, > price bigint) > STORED BY 'carbondata') > need to -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Ro...
Github user zzcclp commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1959#discussion_r167420058 --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala --- @@ -775,6 +776,8 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll { .option("bad_records_action", badRecordAction) .option("dbName", tableIdentifier.database.get) .option("tableName", tableIdentifier.table) +.option(CarbonStreamParser.CARBON_STREAM_PARSER, + "org.apache.carbondata.streaming.parser.CSVStreamParserImp") --- End diff -- OK, I will modify it later. ---
[jira] [Updated] (CARBONDATA-2153) Failed to update table status for pre-aggregate table when maintain insert twice and auto merge open
[ https://issues.apache.org/jira/browse/CARBONDATA-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-2153: Summary: Failed to update table status for pre-aggregate table when maintain insert twice and auto merge open (was: Failed to update table status for pre-aggregate table when maintain insert twice ) > Failed to update table status for pre-aggregate table when maintain insert > twice and auto merge open > - > > Key: CARBONDATA-2153 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2153 > Project: CarbonData > Issue Type: Improvement > Components: core, spark-integration >Affects Versions: 1.3.0 >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Fix For: 1.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Failed to update table status for pre-aggregate table when maintain insert > twice -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2153) Failed to update table status for pre-aggregate table when maintain insert twice
[ https://issues.apache.org/jira/browse/CARBONDATA-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-2153: Description: Failed to update table status for pre-aggregate table when maintain insert twice was:Failed to update table status for pre-aggregate table when maintain insert twice > Failed to update table status for pre-aggregate table when maintain insert > twice > > > Key: CARBONDATA-2153 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2153 > Project: CarbonData > Issue Type: Improvement > Components: core, spark-integration >Affects Versions: 1.3.0 >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Fix For: 1.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Failed to update table status for pre-aggregate table when maintain insert > twice -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2153) Failed to update table status for pre-aggregate table when maintain insert twice
[ https://issues.apache.org/jira/browse/CARBONDATA-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-2153: Summary: Failed to update table status for pre-aggregate table when maintain insert twice (was: Failed to update table status for pre-aggregate table when maintain insert twice) > Failed to update table status for pre-aggregate table when maintain insert > twice > - > > Key: CARBONDATA-2153 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2153 > Project: CarbonData > Issue Type: Improvement > Components: core, spark-integration >Affects Versions: 1.3.0 >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Fix For: 1.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Failed to update table status for pre-aggregate table when maintain insert > twice -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #1965: [DUMMY] Run SDV test suite
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1965 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3683/ ---
[GitHub] carbondata issue #1965: [DUMMY] Run SDV test suite
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1965 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2443/ ---
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1967 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3682/ ---
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1967 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2442/ ---
[GitHub] carbondata issue #1968: [CARBONDATA-2156] Add interface annotation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1968 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3681/ ---
[GitHub] carbondata issue #1949: [HOTFIX][CARBONDATA2144] Optimize preaggregate table...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1949 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3680/ ---
[GitHub] carbondata issue #1968: [CARBONDATA-2156] Add interface annotation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1968 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2441/ ---
[GitHub] carbondata pull request #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user QiangCai commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1967#discussion_r167419025 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/Schema.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.sdk.file; + +import java.io.IOException; + +import org.apache.carbondata.common.annotations.InterfaceAudience; +import org.apache.carbondata.common.annotations.InterfaceStability; + +import com.google.gson.GsonBuilder; +import com.google.gson.TypeAdapter; +import com.google.gson.stream.JsonReader; +import com.google.gson.stream.JsonWriter; + +/** + * A schema used to write and read data files + */ +@InterfaceAudience.User +@InterfaceStability.Unstable +public class Schema { + + private Field[] fields; + + private Schema(Field[] fields) { --- End diff -- Suggest public this interface to the user. Because it is easy to build this field list ---
[GitHub] carbondata issue #1949: [HOTFIX][CARBONDATA2144] Optimize preaggregate table...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1949 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2440/ ---
[jira] [Created] (CARBONDATA-2157) It failed when running the pre-aggregate code again after stopping the running pre-aggregate code
xubo245 created CARBONDATA-2157: --- Summary: It failed when running the pre-aggregate code again after stopping the running pre-aggregate code Key: CARBONDATA-2157 URL: https://issues.apache.org/jira/browse/CARBONDATA-2157 Project: CarbonData Issue Type: Bug Components: core, spark-integration Affects Versions: 1.3.0 Reporter: xubo245 It failed when running the pre-aggregate code again after stopping the running pre-aggregate code like debug failed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #1965: [DUMMY] Run SDV test suite
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1965 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3678/ ---
[GitHub] carbondata issue #1792: [CARBONDATA-2018][DataLoad] Optimization in reading/...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1792 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2438/ ---
[GitHub] carbondata issue #1713: [CARBONDATA-1899] Optimize CarbonData concurrency te...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1713 @jackylk Please review it. ---
[GitHub] carbondata issue #1966: [HOTFIX]Upgrade pom version from 1.3.0 to 1.4.0 in p...
Github user gangken commented on the issue: https://github.com/apache/carbondata/pull/1966 ACK ---
[GitHub] carbondata issue #1968: [CARBONDATA-2156] Add interface annotation
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1968 Merged into carbonstore branch ---
[GitHub] carbondata pull request #1968: [CARBONDATA-2156] Add interface annotation
Github user jackylk closed the pull request at: https://github.com/apache/carbondata/pull/1968 ---
[GitHub] carbondata issue #1968: [CARBONDATA-2156] Add interface annotation
Github user QiangCai commented on the issue: https://github.com/apache/carbondata/pull/1968 LGTM ---
[GitHub] carbondata issue #1792: [CARBONDATA-2018][DataLoad] Optimization in reading/...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1792 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3677/ ---
[GitHub] carbondata issue #1965: [DUMMY] Run SDV test suite
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1965 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2437/ ---
[GitHub] carbondata issue #1713: [CARBONDATA-1899] Optimize CarbonData concurrency te...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1713 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3676/ ---
[GitHub] carbondata issue #1713: [CARBONDATA-1899] Optimize CarbonData concurrency te...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1713 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2436/ ---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1856 @jackylk Please review it ---
[GitHub] carbondata issue #1949: [HOTFIX][CARBONDATA2144] Optimize preaggregate table...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1949 @jackylk @chenliang613 @jackylk Please review it asap. ---
[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1857 @jackylk Please review it ---
[GitHub] carbondata pull request #1955: [HOTFIX] Fix documentation errors.
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1955#discussion_r167416957 --- Diff: docs/data-management-on-carbondata.md --- @@ -983,8 +975,8 @@ roll-up for the queries on these hierarchies. USING "timeseries" DMPROPERTIES ( 'event_timeâ=âorder_timeâ, - 'day_granualrityâ=â1â, --- End diff -- I has fixed it in: https://github.com/apache/carbondata/pull/1949 ãããã ---
[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1857 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3674/ ---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1856 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3675/ ---
[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1857 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2434/ ---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1856 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2435/ ---
[GitHub] carbondata issue #1965: [DUMMY] Run SDV test suite
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/1965 retest this please ---
[GitHub] carbondata pull request #1792: [CARBONDATA-2018][DataLoad] Optimization in r...
GitHub user xuchuanyin reopened a pull request: https://github.com/apache/carbondata/pull/1792 [CARBONDATA-2018][DataLoad] Optimization in reading/writing for sort temp row Pick up the no-sort fields in the row and pack them as bytes array and skip parsing them during merge sort to reduce CPU consumption I've tested it in my cluster and seen about 8% performance gained (74MB/s/Node -> 81MB/s/Node) in data loading. Please note that global_sort will not gain benefit from this feature since there are no sort temp file in that procedure. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [x] Any interfaces changed? `Some internal used interface has been changed` - [x] Any backward compatibility impacted? `No` - [x] Document update required? `No` - [x] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? `No` - How it is tested? Please attach test report. `Tested in 3-node cluster with real business data` - Is it a performance related change? Please attach the performance test report. `Yes, I've tested it in my cluster and seen about 8% performance gained (74MB/s/Node -> 81MB/s/Node) in data loading.` - Any additional information to help reviewers in testing this change. `No` - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. `Unrelated` You can merge this pull request into a Git repository by running: $ git pull https://github.com/xuchuanyin/carbondata opt_sort_temp_serializeation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1792.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1792 commit de71999872761008365efc5bb943c77219479d14 Author: xuchuanyinDate: 2018-02-08T06:35:14Z Optimization in reading/writing for sort temp row Pick up the no-sort fields in the row and pack them as byte array and skip parsing them during merge sort to reduce CPU consumption. ---
[jira] [Closed] (CARBONDATA-2114) carbondata issues on sparksql
[ https://issues.apache.org/jira/browse/CARBONDATA-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] legend closed CARBONDATA-2114. -- Resolution: Fixed > carbondata issues on sparksql > - > > Key: CARBONDATA-2114 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2114 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: legend >Priority: Major > > Hi all: > We are testing carbondata for our project. There are some problems on > hdfs 2.6, spark 2.1 carbondata 1.3. > 1.no multiple levels partitions, we need three levels partions. > 2.spark need import carbondata jar, differ from parquet. > 3.low stability, insert failure frequently. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Ro...
Github user QiangCai commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1959#discussion_r167416315 --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala --- @@ -775,6 +776,8 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll { .option("bad_records_action", badRecordAction) .option("dbName", tableIdentifier.database.get) .option("tableName", tableIdentifier.table) +.option(CarbonStreamParser.CARBON_STREAM_PARSER, + "org.apache.carbondata.streaming.parser.CSVStreamParserImp") --- End diff -- Better to use default parser, Can you modify it, like CarbonStructuredStreamingExample.scala ---
[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1929 @sraghunandan @QiangCai @jackylk @sraghunandan @kumarvishal09 Please review it. ---
[GitHub] carbondata issue #1930: [CARBONDATA-2130] Find some spelling error in Carbon...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1930 @chenliang613 @QiangCai @jackylk Pease review it. ---
[GitHub] carbondata issue #1867: [CARBONDATA-2055][Streaming][WIP]Support integrating...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/1867 retest sdv please ---
[GitHub] carbondata issue #1939: [CARBONDATA-2139] Optimize CTAS documentation and te...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1939 @chenliang613 @jackylk @ravipesala @sraghunandan Please review it ---
[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/1959 retest sdv please ---
[GitHub] carbondata issue #1713: [CARBONDATA-1899] Optimize CarbonData concurrency te...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1713 retest this please ---
[GitHub] carbondata issue #1713: [CARBONDATA-1899] Optimize CarbonData concurrency te...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1713 retest sdv please ---
[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1857 retest sdv please ---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1856 retest sdv please ---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1856 retest this please ---
[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1857 retest this please ---
[GitHub] carbondata issue #1968: [CARBONDATA-2156] Add interface annotation
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1968 retest sdv please ---
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1967 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3507/ ---
[GitHub] carbondata issue #1390: [CARBONDATA-1360] fix for update table failures for ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1390 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3506/ ---
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1967 retest sdv please ---
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1967 retest this please ---
[GitHub] carbondata issue #1968: [CARBONDATA-2156] Add interface annotation
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1968 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3505/ ---
[GitHub] carbondata issue #1664: [CARBONDATA-1890][Improvement]-Changes for Struct Op...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1664 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3504/ ---
[GitHub] carbondata issue #1390: [CARBONDATA-1360] fix for update table failures for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1390 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2433/ ---
[GitHub] carbondata issue #1968: [CARBONDATA-2156] Add interface annotation
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1968 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3503/ ---
[GitHub] carbondata issue #1966: [HOTFIX]Upgrade pom version from 1.3.0 to 1.4.0 in p...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1966 @gangken yes, you are right, pr 1961 missed change this pom version. Thanks for your contribution. but pr #1955 already fixed it. ---
[GitHub] carbondata issue #1968: [CARBONDATA-2156] Add interface annotation
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1968 LGTM ---
[GitHub] carbondata issue #1390: [CARBONDATA-1360] fix for update table failures for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1390 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3673/ ---
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1967 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3502/ ---
[GitHub] carbondata issue #1390: [CARBONDATA-1360] fix for update table failures for ...
Github user ashwini-krishnakumar commented on the issue: https://github.com/apache/carbondata/pull/1390 retest this please ---
[GitHub] carbondata issue #1968: [CARBONDATA-2156] Add interface annotation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1968 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2431/ ---
[GitHub] carbondata issue #1968: [CARBONDATA-2156] Add interface annotation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1968 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3671/ ---
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1967 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3670/ ---
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1967 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2432/ ---
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1967 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3501/ ---
[GitHub] carbondata issue #1664: [CARBONDATA-1890][Improvement]-Changes for Struct Op...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1664 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3668/ ---
[GitHub] carbondata issue #1967: [CARBONDATA-1997] Add CarbonWriter SDK API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1967 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3667/ ---
[GitHub] carbondata issue #1664: [CARBONDATA-1890][Improvement]-Changes for Struct Op...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1664 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2428/ ---