[GitHub] carbondata issue #2125: [CARBONDATA-2299]Support showing all segment informa...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2125 @ravipesala @jackylk please help to review, thanks. ---
[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/2139 Currently, there are two kinds of partition: 1. Standard hive partition: same as Apache Hive and Spark 2.CarbonData partition : Hash,List,Range partition So , In this PR, please update all info with "hive partition". ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301][SDK] CarbonStore interface and two...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2127 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3627/ ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301][SDK] CarbonStore interface and two...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2127 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4318/ ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301][SDK] CarbonStore interface and two...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2127 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4849/ ---
[GitHub] carbondata pull request #2140: [CARBONDATA-2315] success and failure log fix...
Github user manishgupta88 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2140#discussion_r179661112 --- Diff: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala --- @@ -493,18 +493,15 @@ object CarbonDataRDDFactory { } // as no record loaded in new segment, new segment should be deleted val newEntryLoadStatus = - if (!carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable.isChildDataMap && +if (!carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable.isChildDataMap && !CarbonLoaderUtil.isValidSegment(carbonLoadModel, carbonLoadModel.getSegmentId.toInt)) { -LOGGER.audit(s"Data load is failed for " + - s"${ carbonLoadModel.getDatabaseName }.${ carbonLoadModel.getTableName }" + - " as there is no data to load") -LOGGER.warn("Cannot write load metadata file as data load failed") + LOGGER.warn("Cannot write load metadata file as there is no data to load") -SegmentStatus.MARKED_FOR_DELETE - } else { -loadStatus - } + SegmentStatus.MARKED_FOR_DELETE +} else { + loadStatus +} --- End diff -- correct the formatting of code ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2131 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4301/ ---
[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2109 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4303/ ---
[GitHub] carbondata pull request #2109: [CARBONDATA-2294] Partition preaggregate supp...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2109 ---
[GitHub] carbondata issue #2066: [CARBONDATA-2257] Added SDV test cases for Partition...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2066 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3610/ ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2134 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3613/ ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/2134 retest this please ---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET Param...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2065 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3619/ ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2131 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3609/ ---
[GitHub] carbondata pull request #2065: [CARBONDATA-2256] Adding sdv Testcases for SE...
Github user kunal642 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2065#discussion_r179413817 --- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/SetParameterTestCase.scala --- @@ -0,0 +1,267 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.carbondata.cluster.sdv.generated + +import org.apache.spark.sql.Row +import org.apache.spark.sql.common.util._ +import org.scalatest.BeforeAndAfterAll + +import org.apache.carbondata.core.datastore.filesystem.{CarbonFile, CarbonFileFilter} +import org.apache.carbondata.core.datastore.impl.FileFactory + +/** + * Test Class for SetParameterTestcase to verify all scenarios + */ + +class SetParameterTestCase extends QueryTest with BeforeAndAfterAll { + + override def beforeAll(): Unit = { +cleanAllTable() + } + + private def cleanAllTable(): Unit = { +sql("drop table if exists carbon_table") +sql("drop table if exists emptyColumnValues") +sql("drop table if exists carbon_table_bad_record_logger") +sql("drop table if exists carbon_table_single_pass") +sql("drop table if exists carbon_table_disable_bad_record_logger") +sql("drop table if exists carbon_table_load") +sql("RESET") + } + + override def afterAll(): Unit = { +cleanAllTable() + } + + test("TC_001-test SET property for Bad Record Logger Enable=FALSE") { +sql("drop table if exists carbon_table_disable_bad_record_logger") +sql("SET carbon.options.bad.records.logger.enable=false") +sql( + "create table carbon_table_disable_bad_record_logger(empno int, empname String, designation String, " + + "doj Timestamp," + + "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," + + "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," + + "utilization int,salary int) STORED BY 'org.apache.carbondata.format'") +sql( + s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table + |carbon_table_disable_bad_record_logger options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary', + |'BAD_RECORDS_ACTION'='REDIRECT','BAD_RECORD_PATH'='$resourcesPath')""" +.stripMargin) +assert(getLogFileCount("default", "carbon_table_disable_bad_record_logger", "0") == 0) + } + + test("TC_002-test SET property for Bad Record Logger Enable=TRUE") { +sql("drop table if exists carbon_table_bad_record_logger") +sql("SET carbon.options.bad.records.logger.enable=true") +sql( + "create table carbon_table_bad_record_logger(empno int, empname String, designation String," + + " doj Timestamp," + + "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," + + "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," + + "utilization int,salary int) STORED BY 'org.apache.carbondata.format'") +sql( + s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table + |carbon_table_bad_record_logger options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary', + |'BAD_RECORDS_ACTION'='REDIRECT','BAD_RECORD_PATH'='$resourcesPath')""" +.stripMargin) +assert(getLogFileCount("default", "carbon_table_bad_record_logger", "0") >= 1) + } + + test("TC_003-test SET property for Bad Record Action=FORCE") { +sql("drop table if exists carbon_table") +sql("SET carbon.options.bad.records.action=force") +sql( + "create table
[GitHub] carbondata pull request #2134: [WIP][CARBONDATA-2310] Refactored code to imp...
Github user dhatchayani commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2134#discussion_r179416485 --- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/UnsafeMemoryDMStore.java --- @@ -52,11 +62,23 @@ private final long taskId = ThreadLocalTaskInfo.getCarbonTaskInfo().getTaskId(); - public UnsafeMemoryDMStore(CarbonRowSchema[] schema) throws MemoryException { + private boolean addRowToUnsafe = true; --- End diff -- This variable is to decide whether to write row by row to unsafe or to collect all rows and then write all to unsafe. ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2134 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3618/ ---
[GitHub] carbondata issue #2140: [CARBONDATA-2315] success and failure log fixed when...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2140 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3617/ ---
[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2109 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3608/ ---
[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2109 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4831/ ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2131 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3611/ ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2131 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4837/ ---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2113 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4838/ ---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET Param...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2065 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4842/ ---
[jira] [Resolved] (CARBONDATA-2294) Support preaggregate table creation on partition tables
[ https://issues.apache.org/jira/browse/CARBONDATA-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manish Gupta resolved CARBONDATA-2294. -- Resolution: Fixed Fix Version/s: 1.4.0 > Support preaggregate table creation on partition tables > --- > > Key: CARBONDATA-2294 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2294 > Project: CarbonData > Issue Type: Sub-task >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Major > Fix For: 1.4.0 > > Time Spent: 17h 20m > Remaining Estimate: 0h > > # Allow creation of aggregate table on partition and non-partition columnsĀ > # Support to load data into aggregate table after insertion into parent > table is complete. > # Support to compact aggregate tables if parent table is compacted. > # If parent table partition is dropped then drop theĀ same partition in child > tables also. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2065: [CARBONDATA-2256] Adding sdv Testcases for SE...
Github user kunal642 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2065#discussion_r179413870 --- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/SetParameterTestCase.scala --- @@ -0,0 +1,267 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.carbondata.cluster.sdv.generated + +import org.apache.spark.sql.Row +import org.apache.spark.sql.common.util._ +import org.scalatest.BeforeAndAfterAll + +import org.apache.carbondata.core.datastore.filesystem.{CarbonFile, CarbonFileFilter} +import org.apache.carbondata.core.datastore.impl.FileFactory + +/** + * Test Class for SetParameterTestcase to verify all scenarios + */ + +class SetParameterTestCase extends QueryTest with BeforeAndAfterAll { + + override def beforeAll(): Unit = { +cleanAllTable() + } + + private def cleanAllTable(): Unit = { +sql("drop table if exists carbon_table") +sql("drop table if exists emptyColumnValues") +sql("drop table if exists carbon_table_bad_record_logger") +sql("drop table if exists carbon_table_single_pass") +sql("drop table if exists carbon_table_disable_bad_record_logger") +sql("drop table if exists carbon_table_load") +sql("RESET") + } + + override def afterAll(): Unit = { +cleanAllTable() + } + + test("TC_001-test SET property for Bad Record Logger Enable=FALSE") { +sql("drop table if exists carbon_table_disable_bad_record_logger") +sql("SET carbon.options.bad.records.logger.enable=false") +sql( + "create table carbon_table_disable_bad_record_logger(empno int, empname String, designation String, " + + "doj Timestamp," + + "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," + + "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," + + "utilization int,salary int) STORED BY 'org.apache.carbondata.format'") +sql( + s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table + |carbon_table_disable_bad_record_logger options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary', + |'BAD_RECORDS_ACTION'='REDIRECT','BAD_RECORD_PATH'='$resourcesPath')""" +.stripMargin) +assert(getLogFileCount("default", "carbon_table_disable_bad_record_logger", "0") == 0) + } + + test("TC_002-test SET property for Bad Record Logger Enable=TRUE") { +sql("drop table if exists carbon_table_bad_record_logger") +sql("SET carbon.options.bad.records.logger.enable=true") +sql( + "create table carbon_table_bad_record_logger(empno int, empname String, designation String," + + " doj Timestamp," + + "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," + + "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," + + "utilization int,salary int) STORED BY 'org.apache.carbondata.format'") +sql( + s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table + |carbon_table_bad_record_logger options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary', + |'BAD_RECORDS_ACTION'='REDIRECT','BAD_RECORD_PATH'='$resourcesPath')""" +.stripMargin) +assert(getLogFileCount("default", "carbon_table_bad_record_logger", "0") >= 1) + } + + test("TC_003-test SET property for Bad Record Action=FORCE") { +sql("drop table if exists carbon_table") +sql("SET carbon.options.bad.records.action=force") +sql( + "create table
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2131 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3614/ ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2131 retest this please ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2131 retest this please ---
[GitHub] carbondata issue #2066: [CARBONDATA-2257] Added SDV test cases for Partition...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2066 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4833/ ---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2113 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3615/ ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2134 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4840/ ---
[GitHub] carbondata issue #1940: [CARBONDATA-2140 ] Refactoring code to improve perfo...
Github user anubhav100 commented on the issue: https://github.com/apache/carbondata/pull/1940 In this PR we have removed CarbonDataRecordset, CarbonDataRecordSetProvider, and CarbonDataRecordCursor classes because now our column vectors are filled in CarbonDataPageSource, so there is no need to iterate on the record using recordcursor ---
[GitHub] carbondata issue #2140: [CARBONDATA-2315] success and failure log fixed when...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2140 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4841/ ---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET Param...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2065 LGTM ---
[jira] [Created] (CARBONDATA-2315) DataLoad is showing success and failure message in log,when no data is loaded into table during LOAD
Indhumathi Muthumurugesh created CARBONDATA-2315: Summary: DataLoad is showing success and failure message in log,when no data is loaded into table during LOAD Key: CARBONDATA-2315 URL: https://issues.apache.org/jira/browse/CARBONDATA-2315 Project: CarbonData Issue Type: Improvement Reporter: Indhumathi Muthumurugesh Assignee: Indhumathi Muthumurugesh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2131 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4832/ ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2131 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4304/ ---
[GitHub] carbondata pull request #2140: [CARBONDATA-2315] success and failure log fix...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2140 [CARBONDATA-2315] success and failure log fixed when no data is loaded into table Success and failure log fixed when no data is loaded into table - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata load1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2140.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2140 commit 309dda3922bd3c0755e479fe878c48bb1e3e1add Author: Indhumathi27Date: 2018-04-05T10:39:22Z [CARBONDATA-2315] success and failure log fixed when no data is loaded into table Please enter the commit message for your changes. Lines starting ---
[GitHub] carbondata issue #2066: [CARBONDATA-2257] Added SDV test cases for Partition...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2066 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4305/ ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2134 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4307/ ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2131 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4306/ ---
[GitHub] carbondata pull request #2131: [CARBONDATA-2313] Support unmanaged carbon ta...
Github user ajantha-bhat commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2131#discussion_r179360469 --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java --- @@ -2910,10 +2914,16 @@ private static long getMaxOfBlockAndFileSize(long blockSize, long fileSize) { * @return */ public static String getBlockId(AbsoluteTableIdentifier identifier, String filePath, - String segmentId) { + String segmentId, boolean isUnmangedTable) { String blockId; String blockName = filePath.substring(filePath.lastIndexOf("/") + 1, filePath.length()); String tablePath = identifier.getTablePath(); + --- End diff -- OK. Hanlded. ---
[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2109 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4302/ ---
[GitHub] carbondata pull request #2137: [CARBONDATA-2314] Removed block for Streaming...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2137 ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2131 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4830/ ---
[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2109 retest sdv please ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2131 retest this please ---
[GitHub] carbondata pull request #2131: [CARBONDATA-2313] Support unmanaged carbon ta...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2131#discussion_r179361367 --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/TableStatusReadCommittedScope.java --- @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.carbondata.core.readcommitter; + +import java.io.IOException; +import java.util.Map; + +import org.apache.carbondata.common.annotations.InterfaceAudience; +import org.apache.carbondata.common.annotations.InterfaceStability; +import org.apache.carbondata.core.datamap.Segment; +import org.apache.carbondata.core.indexstore.blockletindex.SegmentIndexFileStore; +import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier; +import org.apache.carbondata.core.metadata.SegmentFileStore; +import org.apache.carbondata.core.statusmanager.LoadMetadataDetails; +import org.apache.carbondata.core.statusmanager.SegmentStatusManager; +import org.apache.carbondata.core.util.path.CarbonTablePath; + +/** + * ReadCommittedScope for the managed carbon table + */ +@InterfaceAudience.Internal +@InterfaceStability.Stable +public class TableStatusReadCommittedScope implements ReadCommittedScope { + private LoadMetadataDetails[] loadMetadataDetails; + private AbsoluteTableIdentifier identifier; + + public TableStatusReadCommittedScope(AbsoluteTableIdentifier identifier) { +this.identifier = identifier; +takeCarbonIndexFileSnapShot(); + } + + @Override public LoadMetadataDetails[] getSegmentList() throws IOException { +try { + this.loadMetadataDetails = SegmentStatusManager + .readTableStatusFile(CarbonTablePath.getTableStatusFilePath(identifier.getTablePath())); + return this.loadMetadataDetails; +} catch (IOException ex) { + this.loadMetadataDetails = new LoadMetadataDetails[0]; --- End diff -- should throw exception here, it is not valid behaviour to return empty list ---
[GitHub] carbondata issue #2137: [CARBONDATA-2314] Removed block for Streaming with P...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2137 LGTM ---
[GitHub] carbondata pull request #2131: [CARBONDATA-2313] Support unmanaged carbon ta...
Github user ajantha-bhat commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2131#discussion_r179360518 --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java --- @@ -128,12 +134,18 @@ protected CarbonTable getOrCreateCarbonTable(Configuration configuration) throws @Override public List getSplits(JobContext job) throws IOException { AbsoluteTableIdentifier identifier = getAbsoluteTableIdentifier(job.getConfiguration()); -LoadMetadataDetails[] loadMetadataDetails = SegmentStatusManager - .readTableStatusFile(CarbonTablePath.getTableStatusFilePath(identifier.getTablePath())); + CarbonTable carbonTable = getOrCreateCarbonTable(job.getConfiguration()); if (null == carbonTable) { throw new IOException("Missing/Corrupt schema file for table."); } +this.readCommittedScope = getReadCommitted(job, identifier); --- End diff -- done ---
[jira] [Resolved] (CARBONDATA-2314) Data mismatch in Pre-Aggregate table after Streaming load due to threadset issue
[ https://issues.apache.org/jira/browse/CARBONDATA-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2314. -- Resolution: Fixed Fix Version/s: 1.4.0 > Data mismatch in Pre-Aggregate table after Streaming load due to threadset > issue > > > Key: CARBONDATA-2314 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2314 > Project: CarbonData > Issue Type: Bug >Reporter: Praveen M P >Assignee: Praveen M P >Priority: Major > Fix For: 1.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Wrong data insertion into Pre-Aggregate table on Streaming table due to > ThreadSet issue -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2131 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3607/ ---
[jira] [Assigned] (CARBONDATA-2279) presto handle empty carbondata file
[ https://issues.apache.org/jira/browse/CARBONDATA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geetika Gupta reassigned CARBONDATA-2279: - Assignee: Geetika Gupta > presto handle empty carbondata file > > > Key: CARBONDATA-2279 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2279 > Project: CarbonData > Issue Type: Bug > Components: core >Affects Versions: 1.3.0 >Reporter: zhangwei >Assignee: Geetika Gupta >Priority: Major > Fix For: 1.3.0 > > Attachments: outofindex.patch > > > Caused by: java.lang.RuntimeException: Error creating Splits from > CarbonTableInputFormat > at > org.apache.carbondata.presto.impl.CarbonTableReader.getInputSplits2(CarbonTableReader.java:433) > at > org.apache.carbondata.presto.CarbondataSplitManager.getSplits(CarbondataSplitManager.java:75) > ... 22 more > Caused by: java.io.IOException: Problem in loading segment blocks. > at > org.apache.carbondata.core.indexstore.BlockletDataMapIndexStore.getAll(BlockletDataMapIndexStore.java:153) > at > org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapFactory.getDataMaps(BlockletDataMapFactory.java:76) > at > org.apache.carbondata.core.datamap.TableDataMap.prune(TableDataMap.java:95) > at > org.apache.carbondata.hadoop.api.CarbonTableInputFormat.getDataBlocksOfSegment(CarbonTableInputFormat.java:748) > at > org.apache.carbondata.hadoop.api.CarbonTableInputFormat.getSplits(CarbonTableInputFormat.java:679) > at > org.apache.carbondata.hadoop.api.CarbonTableInputFormat.getSplits(CarbonTableInputFormat.java:439) > at > org.apache.carbondata.presto.impl.CarbonTableReader.getInputSplits2(CarbonTableReader.java:417) > ... 23 more > Caused by: java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.getLocations(AbstractDFSCarbonFile.java:514) > at > org.apache.carbondata.core.indexstore.BlockletDataMapIndexStore.getAll(BlockletDataMapIndexStore.java:142) > ... 29 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2066: [CARBONDATA-2257] Added SDV test cases for Partition...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2066 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4843/ ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user gvramana commented on the issue: https://github.com/apache/carbondata/pull/2131 LGTM ---
[GitHub] carbondata pull request #2102: [CARBONDATA-2277] fix for filter of default v...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2102 ---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET Param...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2065 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4312/ ---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2113 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4308/ ---
[GitHub] carbondata issue #2140: [CARBONDATA-2315] success and failure log fixed when...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2140 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4311/ ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2131 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4844/ ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2134 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4310/ ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2134 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4309/ ---
[jira] [Resolved] (CARBONDATA-2256) Adding sdv Testcases for SET_Parameter_Dynamically_Feature
[ https://issues.apache.org/jira/browse/CARBONDATA-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2256. -- Resolution: Fixed Fix Version/s: 1.4.0 > Adding sdv Testcases for SET_Parameter_Dynamically_Feature > -- > > Key: CARBONDATA-2256 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2256 > Project: CarbonData > Issue Type: Improvement >Reporter: Indhumathi Muthumurugesh >Assignee: Indhumathi Muthumurugesh >Priority: Minor > Fix For: 1.4.0 > > Time Spent: 10h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2131: [CARBONDATA-2313] Support unmanaged carbon ta...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2131 ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2134 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3622/ ---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2113 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3624/ ---
[GitHub] carbondata issue #2066: [CARBONDATA-2257] Added SDV test cases for Partition...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2066 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3621/ ---
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2131 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3620/ ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2134 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4845/ ---
[GitHub] carbondata issue #2102: [CARBONDATA-2277] fix for filter of default values o...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2102 LGTM ---
[jira] [Updated] (CARBONDATA-2313) Support Reading unmanaged carbon table
[ https://issues.apache.org/jira/browse/CARBONDATA-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G updated CARBONDATA-2313: - Issue Type: Improvement (was: Bug) > Support Reading unmanaged carbon table > -- > > Key: CARBONDATA-2313 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2313 > Project: CarbonData > Issue Type: Improvement >Reporter: Ajantha Bhat >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2066: [CARBONDATA-2257] Added SDV test cases for Partition...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2066 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4847/ ---
[jira] [Updated] (CARBONDATA-2313) Support Reading unmanaged carbon table
[ https://issues.apache.org/jira/browse/CARBONDATA-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G updated CARBONDATA-2313: - Issue Type: New Feature (was: Improvement) > Support Reading unmanaged carbon table > -- > > Key: CARBONDATA-2313 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2313 > Project: CarbonData > Issue Type: New Feature >Reporter: Ajantha Bhat >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2131 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4313/ ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2134 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4848/ ---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2113 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4846/ ---
[GitHub] carbondata pull request #2123: [CARBONDATA-2297] Support SEARCH_MODE for bas...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2123 ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2134 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3626/ ---
[GitHub] carbondata issue #2066: [CARBONDATA-2257] Added SDV test cases for Partition...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2066 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3625/ ---
[GitHub] carbondata issue #2066: [CARBONDATA-2257] Added SDV test cases for Partition...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2066 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4314/ ---
[GitHub] carbondata issue #2123: [CARBONDATA-2297] Support SEARCH_MODE for basic filt...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2123 LGTM ---
[jira] [Resolved] (CARBONDATA-2297) Support SEARCH_MODE for basic filter query
[ https://issues.apache.org/jira/browse/CARBONDATA-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-2297. -- Resolution: Fixed Fix Version/s: 1.4.0 > Support SEARCH_MODE for basic filter query > -- > > Key: CARBONDATA-2297 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2297 > Project: CarbonData > Issue Type: New Feature >Reporter: jiangmanhua >Assignee: jiangmanhua >Priority: Major > Fix For: 1.4.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently, tasks could be jammed subjected to available core number if query > generates many tasks. For basic filter query(with limited result rows), we > could add a new strategy to assign only one task with multiple splits each > node and scan all the splits in parallel(get the result immediately and > instead of using iterator) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2113 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4315/ ---
[GitHub] carbondata issue #2066: [CARBONDATA-2257] Added SDV test cases for Partition...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2066 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4316/ ---
[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2134 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4317/ ---