[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583719152 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/180/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583723271 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1883/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583727207 Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/181/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583727305 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1884/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583729531 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/182/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583735058 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1885/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (CARBONDATA-3683) Support compress offheap data directly in the columnpage in IndexStorageCodec
Xingjun Hao created CARBONDATA-3683: --- Summary: Support compress offheap data directly in the columnpage in IndexStorageCodec Key: CARBONDATA-3683 URL: https://issues.apache.org/jira/browse/CARBONDATA-3683 Project: CarbonData Issue Type: Sub-task Reporter: Xingjun Hao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3684) Remove MDK in write path
Jacky Li created CARBONDATA-3684: Summary: Remove MDK in write path Key: CARBONDATA-3684 URL: https://issues.apache.org/jira/browse/CARBONDATA-3684 Project: CarbonData Issue Type: Improvement Reporter: Jacky Li Since only DATE is dictionary now, MDK is always 4 bytes. We can simplify many places in write path to save memory -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] jackylk closed pull request #3596: [CARBONDATA-3673] Remove unused declarations
jackylk closed pull request #3596: [CARBONDATA-3673] Remove unused declarations URL: https://github.com/apache/carbondata/pull/3596 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path
CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path URL: https://github.com/apache/carbondata/pull/3598#issuecomment-583757935 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/183/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3606: [CARBONDATA-3681] Change default compressor to zstd
CarbonDataQA1 commented on issue #3606: [CARBONDATA-3681] Change default compressor to zstd URL: https://github.com/apache/carbondata/pull/3606#issuecomment-583758880 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/184/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path
CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path URL: https://github.com/apache/carbondata/pull/3598#issuecomment-583762940 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1886/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3606: [CARBONDATA-3681] Change default compressor to zstd
CarbonDataQA1 commented on issue #3606: [CARBONDATA-3681] Change default compressor to zstd URL: https://github.com/apache/carbondata/pull/3606#issuecomment-583764317 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1887/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path
CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path URL: https://github.com/apache/carbondata/pull/3598#issuecomment-583769450 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/186/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path
CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path URL: https://github.com/apache/carbondata/pull/3598#issuecomment-583774424 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1889/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] marchpure opened a new pull request #3607: [CARBONDATA-3670] Support compress offheap columnpage directly, avoding a copy of data from offhead to heap when compressed.
marchpure opened a new pull request #3607: [CARBONDATA-3670] Support compress offheap columnpage directly, avoding a copy of data from offhead to heap when compressed. URL: https://github.com/apache/carbondata/pull/3607 ### Why is this PR needed? When loading, the columnpages are stored on the offheap by default, compression is needed to save storage cost. But, in the compression, the data must be copied from the offheap to the heap before compressed, leads to heavier GC overhead compared with compress offhead data directly. Overall, this pr aims to support compress offheap data in columnpage directly, avoding a copy of data from offhead to heap when compressed. ### What changes were proposed in this PR? 1. Support compress direct bytebuffer in the SNAPPY/ZSTD/GZIP compressor Add Interface compressByte(ByteBuffer) in the Compressor/SnappyCompressor/ZstdCompressor/GzipCompressor.java 2. Support compress offheap data directly in the columnpage if the dataype is primitive 2.1 Add Interface getPage in columnpage to get data as directbytebuffer 2.2 The compress() in the Columnpage.java is changed. If the datatype is primitve and the page is unsafe, compress the directbytebuffer returned by getPage() directly. 3. Support compress offheap data directly in the columnpage in IndexStorageCodec 3.1 For String/Varchar, the RLE and InvertIndex needs to get the columnpage as 2-dimension bytearray, in which each bytearray presents a row, We add a interface getByteBufferArray() in the Columnpage, to replace the 2-dimension bytearray. Then, InvertIndex and RLE can work on the directbytebuffer directly. 3.2 If there are no need to build RLE and InvertIndex, getByteBufferArray() return the flatten data as directbytebuffer, which can be compressed directly. ### Does this PR introduce any user interface change? - No ### Is any new testcase added? - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3607: [CARBONDATA-3670] Support compress offheap data in columnpage directly, avoding a copy of data from offhead to heap before compressed.
CarbonDataQA1 commented on issue #3607: [CARBONDATA-3670] Support compress offheap data in columnpage directly, avoding a copy of data from offhead to heap before compressed. URL: https://github.com/apache/carbondata/pull/3607#issuecomment-583794215 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/188/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3607: [CARBONDATA-3670] Support compress offheap data in columnpage directly, avoding a copy of data from offhead to heap before compressed.
CarbonDataQA1 commented on issue #3607: [CARBONDATA-3670] Support compress offheap data in columnpage directly, avoding a copy of data from offhead to heap before compressed. URL: https://github.com/apache/carbondata/pull/3607#issuecomment-583797146 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1890/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path
CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path URL: https://github.com/apache/carbondata/pull/3598#issuecomment-583808736 Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/189/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path
CarbonDataQA1 commented on issue #3598: [CARBONDATA-3684] Remove MDK and cardinality in write path URL: https://github.com/apache/carbondata/pull/3598#issuecomment-583812575 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1891/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
