[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2056 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4239/ ---
[GitHub] carbondata issue #2128: [CARBONDATA-2303] [WIP] If dataload is failed for pa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2128 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4733/ ---
[GitHub] carbondata issue #2128: [CARBONDATA-2303] [WIP] If dataload is failed for pa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2128 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3506/ ---
[jira] [Created] (CARBONDATA-2303) If dataload is failed for parition table then cleanup is not working.
Rahul Kumar created CARBONDATA-2303: --- Summary: If dataload is failed for parition table then cleanup is not working. Key: CARBONDATA-2303 URL: https://issues.apache.org/jira/browse/CARBONDATA-2303 Project: CarbonData Issue Type: Bug Reporter: Rahul Kumar Test Step : 1. create table 2. load data (make sure data load is failed either manually or other) 3. clean files for table *Expected Output*: after clean files data from HDFS should be delete for segments which is Marked for delete. *Actual Output:*Alter cleanup ,data are not deleted from HDFS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2129: [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock f...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2129 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4238/ ---
[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2056 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4237/ ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2130 @ravipesala @jackylk please review, thanks ---
[GitHub] carbondata issue #2128: [WIP] partition table clean files fixed
Github user rahulforallp commented on the issue: https://github.com/apache/carbondata/pull/2128 retest this please ---
[GitHub] carbondata issue #2126: [CARBONDATA-2300] Add ENABLE_UNSAFE_IN_QUERY_EXECUTI...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2126 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4236/ ---
[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2056 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4732/ ---
[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2056 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3505/ ---
[GitHub] carbondata issue #2083: [CARBONDATA-2269]Support Query On PreAggregate table...
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/2083 retest sdv please ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2099 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4731/ ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2099 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3503/ ---
[GitHub] carbondata issue #2126: [CARBONDATA-2300] Add ENABLE_UNSAFE_IN_QUERY_EXECUTI...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2126 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4727/ ---
[GitHub] carbondata issue #2126: [CARBONDATA-2300] Add ENABLE_UNSAFE_IN_QUERY_EXECUTI...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2126 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3500/ ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2130 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4235/ ---
[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2056 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4730/ ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301][SDK] CarbonStore interface and que...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2127 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3502/ ---
[GitHub] carbondata pull request #2126: [CARBONDATA-2300] Add ENABLE_UNSAFE_IN_QUERY_...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2126#discussion_r178482124 --- Diff: integration/presto/src/main/java/org/apache/carbondata/presto/impl/CarbonTableReader.java --- @@ -372,6 +372,11 @@ public TBase create() { CarbonProperties.getInstance().addProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB, config.getUnsafeMemoryInMb()); } +if(config.getEnableUnsafeInQueryExecution() != null) { + CarbonProperties.getInstance() --- End diff -- change the style to like ``` CarbonProperties.getInstance().addProperty( CarbonCommonConstants.ENABLE_UNSAFE_IN_QUERY_EXECUTION, config.getEnableUnsafeInQueryExecution()); ``` ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301][SDK] CarbonStore interface and que...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2127 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4729/ ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2099 retest this please ---
[GitHub] carbondata issue #2129: [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock f...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2129 retest sdv please ---
[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2056 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4728/ ---
[GitHub] carbondata issue #2083: [CARBONDATA-2269]Support Query On PreAggregate table...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2083 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4726/ ---
[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2056 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3501/ ---
[GitHub] carbondata issue #2083: [CARBONDATA-2269]Support Query On PreAggregate table...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2083 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3499/ ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2130 retest sdv please ---
[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1853 @sraghunandan @ravipesala @QiangCai @kumarvishal09 @jackylk please review it ---
[GitHub] carbondata issue #1990: [CARBONDATA-2195] Add new test case for partition fe...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1990 @anubhav100 How to handle this PR? ---
[GitHub] carbondata issue #2073: [CARBONDATA-2260] CarbonThriftServer should support ...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2073 @jackylk please review it. ---
[GitHub] carbondata issue #2083: [CARBONDATA-2269]Support Query On PreAggregate table...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2083 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4234/ ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2099 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3498/ ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2099 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4725/ ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2130 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4233/ ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2130 retest sdv please ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2099 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4232/ ---
[jira] [Updated] (CARBONDATA-2302) Fix some bugs when separate visible and invisible segments info into two files
[ https://issues.apache.org/jira/browse/CARBONDATA-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhichao Zhang updated CARBONDATA-2302: --- Description: There are some bugs when separate visible and invisible segments info into two files: # It will not delete physical data of history segments after separating # Generate duplicated segment id. was: There are some bugs where separate visible and invisible segments info into two files: # It will not delete physical data of history segments after separating # Generate duplicated segment id. > Fix some bugs when separate visible and invisible segments info into two files > -- > > Key: CARBONDATA-2302 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2302 > Project: CarbonData > Issue Type: Bug > Components: core, data-load >Affects Versions: 1.4.0 >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Major > Fix For: 1.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > There are some bugs when separate visible and invisible segments info into > two files: > # It will not delete physical data of history segments after separating > # Generate duplicated segment id. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2130 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4231/ ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301] CarbonStore interface and query AP...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2127 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4724/ ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301] CarbonStore interface and query AP...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2127 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3497/ ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301] CarbonStore interface and query AP...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2127 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4230/ ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2130 retest sdv please ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2130 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4229/ ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2130 retest sdv please ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2099 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3496/ ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2099 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4723/ ---
[GitHub] carbondata issue #2129: [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2129 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3495/ ---
[GitHub] carbondata issue #2129: [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2129 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4722/ ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2099 retest this please ---
[GitHub] carbondata pull request #2099: [CARBONDATA-2276][SDK] Support API to read sc...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2099#discussion_r178471774 --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java --- @@ -2392,6 +2376,12 @@ static DataType thriftDataTyopeToWrapperDataType( tableInfo.setDataMapSchemas(null); return tableInfo; } else { + TBaseCreator createTBase = new ThriftReader.TBaseCreator() { --- End diff -- I think you can use `readSchemaFile` method here. ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2130 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4228/ ---
[GitHub] carbondata issue #2129: [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock f...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2129 retest this please ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2130 retest sdv please ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2130 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4720/ ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2130 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3493/ ---
[GitHub] carbondata issue #2130: [CARBONDATA-2302]Fix some bugs when separate visible...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2130 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4227/ ---
[GitHub] carbondata issue #2129: [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2129 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4721/ ---
[GitHub] carbondata issue #2128: [WIP] partition table clean files fixed
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2128 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4226/ ---
[GitHub] carbondata issue #2129: [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2129 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3494/ ---
[GitHub] carbondata issue #2128: [WIP] partition table clean files fixed
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2128 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4225/ ---
[GitHub] carbondata pull request #2130: [CARBONDATA-2302]Fix some bugs when separate ...
GitHub user zzcclp opened a pull request: https://github.com/apache/carbondata/pull/2130 [CARBONDATA-2302]Fix some bugs when separate visible and invisible segments info into two files There are some bugs where separate visible and invisible segments info into two files: 1. It will not delete physical data of history segments after separating 2. Generate duplicated segment id Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? No - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zzcclp/carbondata CARBONDATA-2302 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2130.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2130 commit 5aae4310110fb69f0def02654bb49a96fe606bf3 Author: Zhang Zhichao <441586683@...> Date: 2018-04-01T17:09:52Z [CARBONDATA-2302]Fix some bugs when separate visible and invisible segments info into two files There are some bugs where separate visible and invisible segments info into two files: 1. It will not delete physical data of history segments after separating 2. Generate duplicated segment id ---
[GitHub] carbondata issue #2129: [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock f...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2129 retest this please ---
[GitHub] carbondata issue #2129: [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock f...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2129 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4224/ ---
[GitHub] carbondata issue #2128: [WIP] partition table clean files fixed
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2128 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4223/ ---
[GitHub] carbondata issue #2128: [WIP] partition table clean files fixed
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2128 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4719/ ---
[GitHub] carbondata issue #2129: [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2129 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3490/ ---
[GitHub] carbondata issue #2128: [WIP] partition table clean files fixed
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2128 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3492/ ---
[GitHub] carbondata issue #2128: [WIP] partition table clean files fixed
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2128 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4222/ ---
[GitHub] carbondata pull request #2129: [CARBONDATA-2298][BACKPORT-1.3]Delete segment...
GitHub user zzcclp opened a pull request: https://github.com/apache/carbondata/pull/2129 [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock files before update metadata If there are some COMPACTED segments and their last modified time is within one hour, the segment lock files deletion operation will not be executed. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? No - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zzcclp/carbondata CARBONDATA-2298_backport_1.3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2129.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2129 commit 74dc33e7bba6c34c368b7df3a9aee17e891a5635 Author: Zhang Zhichao <441586683@...> Date: 2018-04-01T16:18:06Z [CARBONDATA-2298][BACKPORT-1.3]Delete segment lock files before update metadata If there are some COMPACTED segments and their last modified time is within one hour, the segment lock files deletion operation will not be executed. ---
[jira] [Created] (CARBONDATA-2302) Fix some bugs when separate visible and invisible segments info into two files
Zhichao Zhang created CARBONDATA-2302: -- Summary: Fix some bugs when separate visible and invisible segments info into two files Key: CARBONDATA-2302 URL: https://issues.apache.org/jira/browse/CARBONDATA-2302 Project: CarbonData Issue Type: Bug Components: core, data-load Affects Versions: 1.4.0 Reporter: Zhichao Zhang Assignee: Zhichao Zhang Fix For: 1.4.0 There are some bugs where separate visible and invisible segments info into two files: # It will not delete physical data of history segments after separating # Generate duplicated segment id. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2128: [WIP] partition table clean files fixed
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2128 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3489/ ---
[GitHub] carbondata issue #1713: [CARBONDATA-1899] Optimize CarbonData concurrency te...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1713 @jackylk CI pass, please check it. ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301] CarbonStore interface and query AP...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2127 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4221/ ---
[GitHub] carbondata issue #2128: [WIP] partition table clean files fixed
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2128 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4220/ ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301] CarbonStore interface and query AP...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2127 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4219/ ---
[GitHub] carbondata issue #2128: [WIP] partition table clean files fixed
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2128 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4713/ ---
[GitHub] carbondata issue #2128: [WIP] partition table clean files fixed
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2128 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3486/ ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2099 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4218/ ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301] CarbonStore interface and query AP...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2127 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3487/ ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2099 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4711/ ---
[GitHub] carbondata issue #2099: [CARBONDATA-2276][SDK] Support API to read schema in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2099 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3484/ ---
[GitHub] carbondata pull request #2125: [CARBONDATA-2299]Support showing all segment ...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2125#discussion_r178457078 --- Diff: integration/spark2/src/test/scala/org/apache/spark/util/CarbonCommandSuite.scala --- @@ -190,4 +190,21 @@ class CarbonCommandSuite extends Spark2QueryTest with BeforeAndAfterAll { dropTable(tableName) } + test("show history segments") { +val tableName = "test_tablestatus_history" +sql(s"drop table if exists ${tableName}") +sql(s"create table ${tableName} (name String, age int) stored by 'carbondata' " + + "TBLPROPERTIES('AUTO_LOAD_MERGE'='true','COMPACTION_LEVEL_THRESHOLD'='2,2')") +val carbonTable = CarbonMetadata.getInstance().getCarbonTable("default", tableName) +sql(s"insert into ${tableName} select 'abc1',1") +sql(s"insert into ${tableName} select 'abc2',2") +sql(s"insert into ${tableName} select 'abc3',3") +assert(sql(s"show segments for table ${tableName}").collect().length == 4) +assert(sql(s"show history segments for table ${tableName}").collect().length == 4) +sql(s"clean files for table ${tableName}") +assert(sql(s"show segments for table ${tableName}").collect().length == 2) +assert(sql(s"show history segments for table ${tableName}").collect().length == 4) --- End diff -- can you assert the segment name in the collect result ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301] CarbonStore interface and query AP...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2127 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4712/ ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301] CarbonStore interface and query AP...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2127 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3485/ ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301] CarbonStore interface and query AP...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2127 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4217/ ---
[GitHub] carbondata issue #1713: [CARBONDATA-1899] Optimize CarbonData concurrency te...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1713 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4216/ ---
[GitHub] carbondata issue #1713: [CARBONDATA-1899] Optimize CarbonData concurrency te...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1713 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4709/ ---
[GitHub] carbondata issue #1713: [CARBONDATA-1899] Optimize CarbonData concurrency te...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1713 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3482/ ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301] CarbonStore interface and query AP...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2127 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4710/ ---
[GitHub] carbondata pull request #2128: [WIP] partition table clean files fixed
GitHub user rahulforallp opened a pull request: https://github.com/apache/carbondata/pull/2128 [WIP] partition table clean files fixed Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rahulforallp/incubator-carbondata part_tab_cleanFile Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2128.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2128 commit 8044edb5afa858fa72ae7b2d0d1cf0685cf92597 Author: rahulforallpDate: 2018-04-01T12:08:51Z partition table clean files fixed ---
[GitHub] carbondata issue #2127: [CARBONDATA-2301] CarbonStore interface and query AP...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2127 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3483/ ---
[GitHub] carbondata pull request #2127: [CARBONDATA-2301] CarbonStore interface and q...
GitHub user jackylk opened a pull request: https://github.com/apache/carbondata/pull/2127 [CARBONDATA-2301] CarbonStore interface and query API User should be able to query carbondata using CarbonStore interface. 1. Get API: It can be used for filter query. It accepts projection column names and filter expression, and returns matched rows. 2. SQL API: it accepts SQL statement and return query result set. This PR is on top of #2099 - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jackylk/incubator-carbondata carbonstore-query Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2127.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2127 commit 0909d4b205962c6fd4a859eb7b55c7806ae5c370 Author: Jacky LiDate: 2018-03-24T14:38:20Z support read schema commit 67e150902e4c0ec75eb07225306eeecd14afbef1 Author: Jacky Li Date: 2018-03-24T16:05:37Z fix style: commit 062616e2ba2431d33cf0374306020e01b3f863d3 Author: Jacky Li Date: 2018-03-24T16:06:20Z fix style commit 25be162f51070791da4a35c7171fbb1df5cb1739 Author: Jacky Li Date: 2018-04-01T08:30:22Z support CarbonStore API ---
[jira] [Created] (CARBONDATA-2301) Support query interface in CarbonStore
Jacky Li created CARBONDATA-2301: Summary: Support query interface in CarbonStore Key: CARBONDATA-2301 URL: https://issues.apache.org/jira/browse/CARBONDATA-2301 Project: CarbonData Issue Type: Sub-task Reporter: Jacky Li Assignee: Jacky Li Fix For: 1.4.0 User should be able to query carbondata using CarbonStore API. 1. Get API: It can be used for filter query. It accepts projection column names and filter expression, and returns matched rows. 2. SQL API: it accepts SQL statement and return query result set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-1601) Add carbon store module
[ https://issues.apache.org/jira/browse/CARBONDATA-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-1601. -- Resolution: Fixed Assignee: Jacky Li > Add carbon store module > --- > > Key: CARBONDATA-1601 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1601 > Project: CarbonData > Issue Type: Sub-task >Reporter: Jacky Li >Assignee: Jacky Li >Priority: Major > Fix For: 1.4.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Add carbondata-store module and move corresponding code from integration > module -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #1713: [CARBONDATA-1899] Optimize CarbonData concurr...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1713#discussion_r178455504 --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/benchmark/ConcurrentQueryBenchmark.scala --- @@ -0,0 +1,575 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.benchmark + +import java.io.File +import java.text.SimpleDateFormat +import java.util +import java.util.Date +import java.util.concurrent.{Callable, Executors, Future, TimeUnit} + +import scala.util.Random + +import org.apache.spark.sql.{DataFrame, Row, SaveMode, SparkSession} +import org.apache.spark.sql.types._ + +import org.apache.carbondata.core.constants.{CarbonCommonConstants, CarbonVersionConstants} +import org.apache.carbondata.core.util.{CarbonProperties, CarbonUtil} + +// scalastyle:off println +/** + * Test concurrent query performance of CarbonData + * + * This benchmark will print out some information: + * 1.Environment information + * 2.Parameters information + * 3.concurrent query performance result using parquet format + * 4.concurrent query performance result using CarbonData format + * + * This benchmark default run in local model, + * user can change 'runInLocal' to false if want to run in cluster, + * user can change variables like: + * + * spark-submit \ +--class org.apache.carbondata.benchmark.ConcurrentQueryBenchmark \ --- End diff -- ok ---
[GitHub] carbondata pull request #1713: [CARBONDATA-1899] Optimize CarbonData concurr...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1713#discussion_r178455501 --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/benchmark/ConcurrentQueryBenchmark.scala --- @@ -0,0 +1,575 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.benchmark + +import java.io.File +import java.text.SimpleDateFormat +import java.util +import java.util.Date +import java.util.concurrent.{Callable, Executors, Future, TimeUnit} + +import scala.util.Random + +import org.apache.spark.sql.{DataFrame, Row, SaveMode, SparkSession} +import org.apache.spark.sql.types._ + +import org.apache.carbondata.core.constants.{CarbonCommonConstants, CarbonVersionConstants} +import org.apache.carbondata.core.util.{CarbonProperties, CarbonUtil} + +// scalastyle:off println +/** + * Test concurrent query performance of CarbonData + * + * This benchmark will print out some information: + * 1.Environment information + * 2.Parameters information + * 3.concurrent query performance result using parquet format + * 4.concurrent query performance result using CarbonData format + * + * This benchmark default run in local model, + * user can change 'runInLocal' to false if want to run in cluster, + * user can change variables like: + * + * spark-submit \ +--class org.apache.carbondata.benchmark.ConcurrentQueryBenchmark \ +--master yarn \ +--deploy-mode client \ +--driver-memory 16g \ +--executor-cores 4g \ +--executor-memory 24g \ +--num-executors 3 \ +concurrencyTest.jar \ +totalNum threadNum taskNum resultIsEmpty runInLocal generateFile deleteFile + * details in initParameters method of this benchmark + */ +object ConcurrentQueryBenchmark { + + // generate number of data + var totalNum = 1 * 10 * 1000 + // the number of thread pool + var threadNum = 16 + // task number of spark sql query + var taskNum = 100 + // whether is result empty, if true then result is empty + var resultIsEmpty = true + // the store path of task details + var path: String = "/tmp/carbondata" + // whether run in local or cluster + var runInLocal = true + // whether generate new file + var generateFile = true + // whether delete file + var deleteFile = true + + val cardinalityId = 100 * 1000 * 1000 + val cardinalityCity = 6 + + def parquetTableName: String = "Num" + totalNum + "_" + "comparetest_parquet" + + def orcTableName: String = "Num" + totalNum + "_" + "comparetest_orc" + + def carbonTableName(version: String): String = +"Num" + totalNum + "_" + s"comparetest_carbonV$version" + + --- End diff -- ok ---
[GitHub] carbondata pull request #1713: [CARBONDATA-1899] Optimize CarbonData concurr...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1713#discussion_r178455498 --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/benchmark/GenerateData.scala --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.benchmark + +import org.apache.spark.sql.{DataFrame, Row, SparkSession} +import org.apache.spark.sql.types._ + +object GenerateData { --- End diff -- ok ---
[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1853 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3481/ ---
[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1853 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4708/ ---