[GitHub] carbondata issue #2735: [CARBONDATA-2950]alter add column of hive table fail...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2735 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/362/ ---
[GitHub] carbondata pull request #2735: [CARBONDATA-2950]alter add column of hive tab...
GitHub user akashrn5 opened a pull request: https://github.com/apache/carbondata/pull/2735 [CARBONDATA-2950]alter add column of hive table from carbon for spark versions above 2.1 ### Problem: spark does not support add columns in spark-2.1, but it is supported in 2.2 and above when add column is fired for hive table in carbon session, for spark -version above 2.1, it throws error as unsupported operation on hive table ### Solution: when alter add columns for hive is fired for spark-2.2 and above, it should not throw any exception and it should pass Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [x] Any interfaces changed? NA - [x] Any backward compatibility impacted? NA - [x] Document update required? NA - [x] Testing done Test case is added Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/akashrn5/incubator-carbondata alter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2735.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2735 commit 05f53caa6d1a389eb6dce61359e800f81731966a Author: akashrn5 Date: 2018-09-19T14:21:39Z alter add column of hive table from carbon for spark versions above 2.1 ---
[jira] [Created] (CARBONDATA-2950) Alter table add columns fails for hive table in carbon session for spark version above 2.1
Akash R Nilugal created CARBONDATA-2950: --- Summary: Alter table add columns fails for hive table in carbon session for spark version above 2.1 Key: CARBONDATA-2950 URL: https://issues.apache.org/jira/browse/CARBONDATA-2950 Project: CarbonData Issue Type: Bug Reporter: Akash R Nilugal Assignee: Akash R Nilugal spark does not support add columns in spark-2.1, but it is supported in 2.2 and above when add column is fired for hive table in carbon session, for spark -version above 2.1, it throws error as unsupported operation on hive table -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2728: WIP: optimize compressor
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2728 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/538/ ---
[GitHub] carbondata issue #2728: WIP: optimize compressor
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2728 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8608/ ---
[GitHub] carbondata issue #2728: WIP: optimize compressor
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2728 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/361/ ---
[jira] [Created] (CARBONDATA-2949) Add Java example for CarbonSession
Jacky Li created CARBONDATA-2949: Summary: Add Java example for CarbonSession Key: CARBONDATA-2949 URL: https://issues.apache.org/jira/browse/CARBONDATA-2949 Project: CarbonData Issue Type: New Feature Reporter: Jacky Li User should be able to write Java application to use CarbonSession -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2732: [WIP] lz4 as column compressor in final store
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2732 better to provide performance test result for this ---
[GitHub] carbondata pull request #2732: [WIP] lz4 as column compressor in final store
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2732#discussion_r219024705
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/compression/Lz4Compressor.java
---
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.compression;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.DoubleBuffer;
+import java.nio.FloatBuffer;
+import java.nio.IntBuffer;
+import java.nio.LongBuffer;
+import java.nio.ShortBuffer;
+import java.util.Arrays;
+
+import org.apache.carbondata.core.util.ByteUtil;
+
+import net.jpountz.lz4.LZ4Compressor;
+import net.jpountz.lz4.LZ4Factory;
+import net.jpountz.lz4.LZ4FastDecompressor;
+
+
+public class Lz4Compressor implements Compressor {
+
+ private LZ4Compressor compressor;
+ private LZ4FastDecompressor decompressor;
+
+ public Lz4Compressor() {
+LZ4Factory factory = LZ4Factory.fastestInstance();
+compressor = factory.fastCompressor();
+decompressor = factory.fastDecompressor();
+ }
+
+ @Override
+ public String getName() {
+return "lz4";
+ }
+
+ @Override
+ public byte[] compressByte(byte[] unCompInput) {
+// get max compressed length
--- End diff --
If this code is copied from LZ4, better to make a declaration
---
[GitHub] carbondata issue #2732: [WIP] lz4 as column compressor in final store
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2732 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/537/ ---
[GitHub] carbondata issue #2732: [WIP] lz4 as column compressor in final store
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2732 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8607/ ---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2665 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/536/ ---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2665 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8606/ ---
[GitHub] carbondata issue #2732: [WIP] lz4 as column compressor in final store
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2732 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/360/ ---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2665 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/359/ ---
[GitHub] carbondata issue #2732: [WIP] lz4 as column compressor in final store
Github user kevinjmh commented on the issue: https://github.com/apache/carbondata/pull/2732 retest this please ---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user kevinjmh commented on the issue: https://github.com/apache/carbondata/pull/2665 retest this please ---
[GitHub] carbondata issue #2728: WIP: optimize compressor
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2728 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/535/ ---
[GitHub] carbondata issue #2728: WIP: optimize compressor
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2728 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8605/ ---
[GitHub] carbondata issue #2726: [CARBONDATA-2948] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/534/ ---
[GitHub] carbondata issue #2726: [CARBONDATA-2948] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8604/ ---
[GitHub] carbondata issue #2728: WIP: optimize compressor
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2728 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/358/ ---
[GitHub] carbondata issue #2726: [CARBONDATA-2948] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/357/ ---
[GitHub] carbondata issue #2726: [CARBONDATA-2948] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8603/ ---
[GitHub] carbondata issue #2726: [CARBONDATA-2948] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/533/ ---
[GitHub] carbondata issue #2734: [CARBONDATA-2946] Bloom filter backward compatibilit...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2734 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/532/ ---
[GitHub] carbondata issue #2734: [CARBONDATA-2946] Bloom filter backward compatibilit...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2734 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8602/ ---
[GitHub] carbondata issue #2726: [CARBONDATA-2948] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/356/ ---
[GitHub] carbondata pull request #2726: [CARBONDATA-2948] Float and Byte DataType sup...
Github user kunal642 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2726#discussion_r218800702 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java --- @@ -61,7 +61,7 @@ protected static final boolean unsafe = Boolean.parseBoolean(CarbonProperties.getInstance() .getProperty(CarbonCommonConstants.ENABLE_UNSAFE_COLUMN_PAGE, - CarbonCommonConstants.ENABLE_UNSAFE_COLUMN_PAGE_DEFAULT)); + "false")); --- End diff -- reverted ---
[jira] [Created] (CARBONDATA-2948) Support Float and Byte Datatypes for SDK and DataSource
Kunal Kapoor created CARBONDATA-2948: Summary: Support Float and Byte Datatypes for SDK and DataSource Key: CARBONDATA-2948 URL: https://issues.apache.org/jira/browse/CARBONDATA-2948 Project: CarbonData Issue Type: Improvement Reporter: Kunal Kapoor Assignee: Kunal Kapoor Currently float is supported by internally storing the data as double and changing the data type to Double. This poses some problems while using SparkCarbonFileFormat for reading the float type data. Internally as the data type is changed from Float to Double therefore the data is retrieved as a Double page instead of float. If the user tried to create a table using file format by specifying the datatype as float for any column then the query will fail. User is *restricted to use double to retrieve the data.* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2734: [CARBONDATA-2946] Bloom filter backward compatibilit...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2734 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/355/ ---
[jira] [Created] (CARBONDATA-2947) Adaptive encoding support for timestamp no dictionary
dhatchayani created CARBONDATA-2947: --- Summary: Adaptive encoding support for timestamp no dictionary Key: CARBONDATA-2947 URL: https://issues.apache.org/jira/browse/CARBONDATA-2947 Project: CarbonData Issue Type: Improvement Reporter: dhatchayani Assignee: dhatchayani -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2726: [WIP] Float and Byte DataType support
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2726#discussion_r218790962 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java --- @@ -61,7 +61,7 @@ protected static final boolean unsafe = Boolean.parseBoolean(CarbonProperties.getInstance() .getProperty(CarbonCommonConstants.ENABLE_UNSAFE_COLUMN_PAGE, - CarbonCommonConstants.ENABLE_UNSAFE_COLUMN_PAGE_DEFAULT)); + "false")); --- End diff -- Please revert the change ---
[GitHub] carbondata pull request #2734: [CARBONDATA-2946] Bloom filter backward compa...
GitHub user dhatchayani opened a pull request: https://github.com/apache/carbondata/pull/2734 [CARBONDATA-2946] Bloom filter backward compatibility with adaptive encoding and Refactor **(1)** Refactored the already existing code in ColumnPageWrapper to fill the data to vector **(2)** **Problem:** Bloom filter writing is like measure column. So backward compatibility is not ensured. If we are writing it as like measure column, filter queries will fail. **Solution:** Write the bloom also same as that of no dictionary column. Take care of null bitsets(i.e null values) as null would have been written as 0 as the page is written like that of measure. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dhatchayani/carbondata CARBONDATA-2946 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2734.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2734 commit 984eef1879d92514000a8f5e05bf82e935403066 Author: dhatchayani Date: 2018-09-19T12:37:11Z [CARBONDATA-2946] Bloom filter backward compatibility with adaptive encoding and Refactor ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/354/ ---
[jira] [Created] (CARBONDATA-2946) Bloom filter backward compatibility with adaptive encoding and Refactor
dhatchayani created CARBONDATA-2946: --- Summary: Bloom filter backward compatibility with adaptive encoding and Refactor Key: CARBONDATA-2946 URL: https://issues.apache.org/jira/browse/CARBONDATA-2946 Project: CarbonData Issue Type: Bug Reporter: dhatchayani Assignee: dhatchayani -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2927) Multiple issue fixes for varchar column and complex columns that grows more than 2MB
[ https://issues.apache.org/jira/browse/CARBONDATA-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2927. - Resolution: Fixed Fix Version/s: 1.5.0 > Multiple issue fixes for varchar column and complex columns that grows more > than 2MB > > > Key: CARBONDATA-2927 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2927 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > Fix For: 1.5.0 > > Time Spent: 7h 40m > Remaining Estimate: 0h > > *Fixed:* > *1. varchar data length is more than 2MB, buffer overflow exception (thread > local row buffer)* > *root* casue*: thread* loaclbuffer *was hardcoded with 2MB.* > *solution: grow dynamically based on the row size.* > *2. read data from carbon file having one row of varchar data with 150 MB > length is very slow.* > *root casue: At UnsafeDMStore, ensure memory is just incresing by 8KB each > time and lot of time malloc and free happens before reaching 150MB. hence > very slow performance.* > *solution: directly check and allocate the required size.* > *3. Jvm crash when data size is more than 128 MB in unsafe sort step.* > *root cause: unsafeCarbonRowPage is of 128MB, so if data is more than 128MB > for one row, we access block beyond allocated, leading to JVM crash.* > *solution: validate the size before access and prompt user to increase unsafe > memory. (by carbon property)* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2706: [CARBONDATA-2927] multiple issue fixes for varchar c...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2706 please take care about the loading performance compared with previous nio.buffer implementation. ---
[GitHub] carbondata pull request #2706: [CARBONDATA-2927] multiple issue fixes for va...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2706 ---
[GitHub] carbondata issue #2706: [CARBONDATA-2927] multiple issue fixes for varchar c...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2706 LGTM ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/353/ ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2726 retest this please ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/529/ ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8599/ ---
[GitHub] carbondata issue #2706: [CARBONDATA-2927] multiple issue fixes for varchar c...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2706 LGTM I am merging this PR. @ajantha-bhat Please start another discussion in the forum to support big column data up to 2GB for complex, varchar and string columns. And also make the page size configurable in terms of MB to avoid outofmemory while reading. ---
[GitHub] carbondata issue #2732: [WIP] lz4 as column compressor in final store
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2732 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8597/ ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/352/ ---
[GitHub] carbondata issue #2732: [WIP] lz4 as column compressor in final store
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2732 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/527/ ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/525/ ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8595/ ---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/528/ ---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8598/ ---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/351/ ---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2665 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/523/ ---
[GitHub] carbondata issue #2732: [WIP] lz4 as column compressor in final store
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2732 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/350/ ---
[GitHub] carbondata issue #2733: [CARBONDATA-2818] Upgrade presto integration version...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2733 @chenliang613 : All presto test case failed because of de-serialization error from antlr jar. In the pom.xml of presto db master code. https://github.com/prestodb/presto/blob/master/pom.xml antlr 4.7.1 is used. our pom has a different version. Hence this error. please change out presto pom.xml antlr version to **4.7.1** and re run the CI I have tested locally with this change. All test case passed ---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2665 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8593/ ---
[GitHub] carbondata issue #2732: [WIP] lz4 as column compressor in final store
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2732 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/349/ ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/348/ ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/347/ ---
[GitHub] carbondata pull request #2726: [WIP] Float and Byte DataType support
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2726#discussion_r218715543
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/ColumnPageEncoderMeta.java
---
@@ -132,6 +132,10 @@ private void writeMinMax(DataOutput out) throws
IOException {
out.writeDouble((Double) getMaxValue());
out.writeDouble((Double) getMinValue());
out.writeDouble(0d); // unique value is obsoleted, maintain for
compatibility
+} else if (dataType == DataTypes.FLOAT) {
+ out.writeDouble((Float) getMaxValue());
--- End diff --
used writeFloat instead of writeDouble
---
[GitHub] carbondata pull request #2726: [WIP] Float and Byte DataType support
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2726#discussion_r218715345
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/PrimitivePageStatsCollector.java
---
@@ -140,6 +149,10 @@ private PrimitivePageStatsCollector(DataType dataType)
{
minDouble = Double.POSITIVE_INFINITY;
maxDouble = Double.NEGATIVE_INFINITY;
decimal = 0;
+} else if (dataType == DataTypes.FLOAT) {
+ minFloat = Float.MIN_VALUE;
--- End diff --
done
---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8594/ ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/524/ ---
[GitHub] carbondata pull request #2726: [WIP] Float and Byte DataType support
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2726#discussion_r218715267
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/adaptive/AdaptiveFloatingCodec.java
---
@@ -139,15 +139,15 @@ public void encode(int rowId, long value) {
@Override
public void encode(int rowId, float value) {
if (targetDataType == DataTypes.BYTE) {
-encodedPage.putByte(rowId, (byte) (value * factor));
+encodedPage.putByte(rowId, (byte) (value * (float) factor));
--- End diff --
added another variable for float
---
[GitHub] carbondata pull request #2726: [WIP] Float and Byte DataType support
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2726#discussion_r218715201
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/LazyColumnPage.java
---
@@ -93,7 +93,22 @@ public double getDouble(int rowId) {
@Override
public float getFloat(int rowId) {
-throw new UnsupportedOperationException("internal error");
+DataType dataType = columnPage.getDataType();
--- End diff --
okay
---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2665 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/346/ ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/522/ ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8592/ ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user kevinjmh commented on the issue: https://github.com/apache/carbondata/pull/2654 OK ---
[GitHub] carbondata issue #2726: [WIP] Float and Byte DataType support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2726 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/345/ ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/2654 > I ran a test on table with bloom datamap created before applying this PR, and query it after this PR merged, but the answer is not correct. Can you check it? > > Procedure to reproduce: > > * switch master code before this PR merged > * create table with no-dict measure column (set the measure column as sort column) > * create bloom datamap on the measure column > * load some data into table > * query on the measure column, get a result > * switch to code after this PR merged > * do the same query and compare the result @kevinjmh Issue is reproduced and this is the issue with compatibility because of the data written in new store is of different format. That i will correct it in the next PR. ---
[GitHub] carbondata pull request #2665: [CARBONDATA-2897][DataMap] Optimize datamap c...
Github user kevinjmh commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2665#discussion_r218679293
--- Diff:
core/src/main/java/org/apache/carbondata/core/datamap/DataMapChooser.java ---
@@ -268,23 +238,38 @@ private ExpressionTuple selectDataMap(Expression
expression, List
private void extractColumnExpression(Expression expression,
List columnExpressions) {
-if (expression instanceof ColumnExpression) {
- columnExpressions.add((ColumnExpression) expression);
-} else if (expression instanceof MatchExpression) {
- // this is a special case for lucene
- // build a fake ColumnExpression to filter datamaps which contain
target column
- // a Lucene query string is alike "column:query term"
- String[] queryItems = expression.getString().split(":", 2);
- if (queryItems.length == 2) {
-columnExpressions.add(new ColumnExpression(queryItems[0], null));
- }
-} else if (expression != null) {
- List children = expression.getChildren();
- if (children != null && children.size() > 0) {
-for (Expression exp : children) {
- extractColumnExpression(exp, columnExpressions);
+switch (expression.getFilterExpressionType()) {
--- End diff --
Change to check by method `isSupport` in datamap factory
---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2665 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/521/ ---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2665 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8591/ ---
