[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2659 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/39/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2659 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6435/ ---
[GitHub] carbondata issue #2623: [CARBONDATA-2844] [CARBONDATA-2865] Pass SK/AK to ex...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2623 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8103/ ---
[GitHub] carbondata issue #2654: [CARBONDATA-2896] Adaptive Encoding for Primitive da...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2654 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6434/ ---
[jira] [Updated] (CARBONDATA-2896) Adaptive encoding for primitive data types
[ https://issues.apache.org/jira/browse/CARBONDATA-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhatchayani updated CARBONDATA-2896: Description: Currently Encoding and Decoding is present only for Dictionary, Measure Columns, but for no dictionary Primitive types encoding is *absent.* *Encoding is a technique used to reduce the storage size and after all these encoding, result will be compressed with snappy compression to further reduce the storage size.* *With this feature, we support encoding on the no dictionary primitive data types also.* > Adaptive encoding for primitive data types > -- > > Key: CARBONDATA-2896 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2896 > Project: CarbonData > Issue Type: New Feature >Reporter: dhatchayani >Assignee: dhatchayani >Priority: Major > > Currently Encoding and Decoding is present only for Dictionary, Measure > Columns, but for no dictionary Primitive types encoding is *absent.* > *Encoding is a technique used to reduce the storage size and after all these > encoding, result will be compressed with snappy compression to further reduce > the storage size.* > *With this feature, we support encoding on the no dictionary primitive data > types also.* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2896) Adaptive encoding for primitive data types
[ https://issues.apache.org/jira/browse/CARBONDATA-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhatchayani updated CARBONDATA-2896: Description: Currently Encoding and Decoding is present only for Dictionary, Measure Columns, but for no dictionary Primitive types encoding is *absent.* Encoding is a technique used to reduce the storage size and after all these encoding, result will be compressed with snappy compression to further reduce the storage size. With this feature, we support encoding on the no dictionary primitive data types also. was: Currently Encoding and Decoding is present only for Dictionary, Measure Columns, but for no dictionary Primitive types encoding is *absent.* *Encoding is a technique used to reduce the storage size and after all these encoding, result will be compressed with snappy compression to further reduce the storage size.* *With this feature, we support encoding on the no dictionary primitive data types also.* > Adaptive encoding for primitive data types > -- > > Key: CARBONDATA-2896 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2896 > Project: CarbonData > Issue Type: New Feature >Reporter: dhatchayani >Assignee: dhatchayani >Priority: Major > > Currently Encoding and Decoding is present only for Dictionary, Measure > Columns, but for no dictionary Primitive types encoding is *absent.* > Encoding is a technique used to reduce the storage size and after all these > encoding, result will be compressed with snappy compression to further reduce > the storage size. > With this feature, we support encoding on the no dictionary primitive data > types also. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2659 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6433/ ---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2628 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/38/ ---
[jira] [Created] (CARBONDATA-2896) Adaptive encoding for primitive data types
dhatchayani created CARBONDATA-2896: --- Summary: Adaptive encoding for primitive data types Key: CARBONDATA-2896 URL: https://issues.apache.org/jira/browse/CARBONDATA-2896 Project: CarbonData Issue Type: New Feature Reporter: dhatchayani Assignee: dhatchayani -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2623: [CARBONDATA-2844] [CARBONDATA-2865] Pass SK/AK to ex...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2623 retest this please ---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2628 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8101/ ---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2628 retest this please ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2659 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/37/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2659 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8100/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2659 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6432/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2659 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6431/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2659 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/36/ ---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2628 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/35/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2659 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6430/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2659 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8099/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2659 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8098/ ---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2628 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8097/ ---
[GitHub] carbondata pull request #2662: [WIP][CARBONDATA-2889]Add decoder based fallb...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2662#discussion_r213161218
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/FallbackDecoderBasedColumnPageEncoder.java
---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.core.datastore.page;
+
+import java.util.concurrent.Callable;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.TableSpec;
+import org.apache.carbondata.core.datastore.compression.CompressorFactory;
+import
org.apache.carbondata.core.datastore.page.encoding.EncodedColumnPage;
+import org.apache.carbondata.core.keygenerator.KeyGenerator;
+import org.apache.carbondata.core.keygenerator.factory.KeyGeneratorFactory;
+import
org.apache.carbondata.core.localdictionary.generator.LocalDictionaryGenerator;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.util.CarbonUtil;
+
+public class FallbackDecoderBasedColumnPageEncoder implements
Callable {
+ /**
+ * actual local dictionary generated column page
+ */
+ private EncodedColumnPage encodedColumnPage;
+
+ /**
+ * actual index in the page
+ * this is required as in a blocklet few pages will be local dictionary
+ * encoded and few pages will be plain text encoding
+ * in this case local dictionary encoded page
+ */
+ private int pageIndex;
+
+ private LocalDictionaryGenerator localDictionaryGenerator;
+
+ public FallbackDecoderBasedColumnPageEncoder(EncodedColumnPage
encodedColumnPage, int pageIndex,
+ LocalDictionaryGenerator localDictionaryGenerator) {
+this.encodedColumnPage = encodedColumnPage;
+this.pageIndex = pageIndex;
+this.localDictionaryGenerator = localDictionaryGenerator;
+ }
+
+ @Override public FallbackEncodedColumnPage call() throws Exception {
+
+// uncompress the encoded column page
+byte[] bytes = CompressorFactory.getInstance().getCompressor()
--- End diff --
emm, PR #2628 changed this. We should get the compressor from input
configuration or from the metadata.
---
[GitHub] carbondata pull request #2662: [WIP][CARBONDATA-2889]Add decoder based fallb...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2662#discussion_r213160857
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/blocklet/EncodedBlocklet.java
---
@@ -87,19 +91,24 @@ private void addPageMetadata(EncodedTablePage
encodedTablePage) {
* @param encodedTablePage
* encoded table page
*/
- private void addEncodedMeasurePage(EncodedTablePage encodedTablePage) {
+ private void addEncodedMeasurePage(EncodedTablePage encodedTablePage,
+ Map localDictionaryGeneratorMap) {
// for first page create new list
if (null == encodedMeasureColumnPages) {
encodedMeasureColumnPages = new ArrayList<>();
// adding measure pages
for (int i = 0; i < encodedTablePage.getNumMeasures(); i++) {
-BlockletEncodedColumnPage blockletEncodedColumnPage = new
BlockletEncodedColumnPage(null);
-
blockletEncodedColumnPage.addEncodedColumnColumnPage(encodedTablePage.getMeasure(i));
+BlockletEncodedColumnPage blockletEncodedColumnPage = new
BlockletEncodedColumnPage(null,
+Boolean.parseBoolean(CarbonProperties.getInstance()
--- End diff --
What if the configuration is changed during data loading? So that each
column page will have different configuration, will this be OK?
---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2628 retest this please ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2659 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/34/ ---
[GitHub] carbondata issue #2664: [CARBONDATA-2895] Fix Query result mismatch with Bat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2664 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/33/ ---
[GitHub] carbondata issue #2641: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2641 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/32/ ---
[GitHub] carbondata issue #2663: [CARBONDATA-2894] Add support for complex map type t...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2663 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/31/ ---
[GitHub] carbondata issue #2664: [CARBONDATA-2895] Fix Query result mismatch with Bat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2664 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8095/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2659 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6429/ ---
[GitHub] carbondata issue #2654: [WIP] Adaptive Encoding for Primitive data types
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/30/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2659 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8096/ ---
[GitHub] carbondata issue #2664: [CARBONDATA-2895] Fix Query result mismatch with Bat...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2664 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6428/ ---
[GitHub] carbondata issue #2664: [CARBONDATA-2895] Fix Query result mismatch with Bat...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2664 @jackylk , @ravipesala : Please review ---
[GitHub] carbondata pull request #2664: [CARBONDATA-2895] Fix Query result mismatch w...
GitHub user ajantha-bhat opened a pull request: https://github.com/apache/carbondata/pull/2664 [CARBONDATA-2895] Fix Query result mismatch with Batch-sort in save to disk (sort temp files) scenario. **probelm:** Query result mismatch with Batch-sort in save to disk (sort temp files) scenario. **scenario:** a) Configure batchsort but give batch size more than UnsafeMemoryManager.INSTANCE.getUsableMemory(). b) Load data that is greater than batch size. Observe that unsafeMemoryManager save to disk happened as it cannot process one batch. c) so load happens in 2 batch. d) When query the results. There result data rows is more than expected data rows. **root cause:** For each batch, createSortDataRows() will be called. Files saved to disk during sorting of previous batch was considered for this batch. **solution:** Files saved to disk during sorting of previous batch ,should not be considered for this batch. Hence use batchID as rangeID field of sorttempfiles. So getFilesToMergeSort() will select files of only this batch. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done. done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajantha-bhat/carbondata master_new Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2664.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2664 commit bad70a37508a2bad650aae2b150eecef59449a30 Author: ajantha-bhat Date: 2018-08-27T15:25:03Z [CARBONDATA-2895] Fix Query result mismatch with Batch-sort in save to disk (sort temp files) scenario. probelm: Query result mismatch with Batch-sort in save to disk (sort temp files) scenario. scenario: a) Configure batchsort but give batch size more than UnsafeMemoryManager.INSTANCE.getUsableMemory(). b) Load data that is greater than batch size. Observe that unsafeMemoryManager save to disk happened as it cannot process one batch. c) so load happens in 2 batch. d) When query the results. There result data rows is more than expected data rows. root cause: For each batch, createSortDataRows() will be called. Files saved to disk during sorting of previous batch was considered for this batch. solution: Files saved to disk during sorting of previous batch ,should not be considered for this batch. Hence use batchID as rangeID field of sorttempfiles. So getFilesToMergeSort() will select files of only this batch. ---
[jira] [Created] (CARBONDATA-2895) [Batch-sort]Query result mismatch with Batch-sort in save to disk (sort temp files) scenario.
Ajantha Bhat created CARBONDATA-2895: Summary: [Batch-sort]Query result mismatch with Batch-sort in save to disk (sort temp files) scenario. Key: CARBONDATA-2895 URL: https://issues.apache.org/jira/browse/CARBONDATA-2895 Project: CarbonData Issue Type: Bug Reporter: Ajantha Bhat Assignee: Ajantha Bhat probelm: Query result mismatch with Batch-sort in save to disk (sort temp files) scenario. scenario: a) Configure batchsort but give batch size more than UnsafeMemoryManager.INSTANCE.getUsableMemory(). b) Load data that is greater than batch size. Observe that unsafeMemoryManager save to disk happened as it cannot process one batch. c) so load happens in 2 batch. d) When query the results. There result data rows is more than expected data rows. root cause: For each batch, createSortDataRows() will be called. Files saved to disk during sorting of previous batch was considered for this batch. solution: Files saved to disk during sorting of previous batch ,should not be considered for this batch. Hence use batchID as rangeID field of sorttempfiles. So getFilesToMergeSort() will select files of only this batch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2663: [CARBONDATA-2894] Add support for complex map type t...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2663 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8094/ ---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2628 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/29/ ---
[GitHub] carbondata issue #2641: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2641 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8092/ ---
[GitHub] carbondata issue #2663: [CARBONDATA-2894] Add support for complex map type t...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2663 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6427/ ---
[GitHub] carbondata issue #2623: [CARBONDATA-2844] [CARBONDATA-2865] Pass SK/AK to ex...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2623 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/28/ ---
[GitHub] carbondata issue #2663: [CARBONDATA-2894] Add support for complex map type t...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2663 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6426/ ---
[GitHub] carbondata issue #2654: [WIP] Adaptive Encoding for Primitive data types
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8091/ ---
[GitHub] carbondata pull request #2663: [CARBONDATA-2894] Add support for complex map...
GitHub user manishgupta88 opened a pull request: https://github.com/apache/carbondata/pull/2663 [CARBONDATA-2894] Add support for complex map type through spark carbon file format API This PR supports loading querying complex map type through spark carbon file format API. **Note: This PR is dependent on PR #2649** - [ ] Any interfaces changed? No - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done Added test cases - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/manishgupta88/carbondata map_spark_carbon_file_support Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2663.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2663 commit e67bd0cb485c4bed15ce8ac3ef3be9b3a4f3798e Author: manishgupta88 Date: 2018-08-20T04:59:29Z Added support for Avro Map type using SDK commit 6db7f2a0d7c02406e0ecc9aa7ac69e2ec2e540a6 Author: manishgupta88 Date: 2018-08-27T13:47:21Z Add support for complex map type using spark carbon file format API ---
[jira] [Created] (CARBONDATA-2894) Add support for complex map type through spark carbon file format API
Manish Gupta created CARBONDATA-2894: Summary: Add support for complex map type through spark carbon file format API Key: CARBONDATA-2894 URL: https://issues.apache.org/jira/browse/CARBONDATA-2894 Project: CarbonData Issue Type: Sub-task Reporter: Manish Gupta Assignee: Manish Gupta -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2892) Data mismatch is seen in the Array-String and Array-Timestamp.
[ https://issues.apache.org/jira/browse/CARBONDATA-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Ravichandran updated CARBONDATA-2892: -- Attachment: (was: Array.csv) > Data mismatch is seen in the Array-String and Array-Timestamp. > -- > > Key: CARBONDATA-2892 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2892 > Project: CarbonData > Issue Type: Bug > Environment: 3 Node ANT. >Reporter: Prasanna Ravichandran >Priority: Major > Attachments: Array.csv > > > Data mismatch is seen in the Array-String and Array-Timestamp like mismatch > in data, order, date values. > *Test queries:* > drop table if exists array_com_hive; > create table array_com_hive (CUST_ID string, YEAR int, MONTH int, AGE int, > GENDER string, EDUCATED string, IS_MARRIED string, ARRAY_INT > array,ARRAY_STRING array,ARRAY_DATE array,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) row > format delimited fields terminated by ',' collection items terminated by '$'; > load data local inpath '/opt/csv/complex/Array.csv' into table array_com_hive; > drop table if exists array_com; > create table Array_com (CUST_ID string, YEAR int, MONTH int, AGE int, GENDER > string, EDUCATED string, IS_MARRIED string, ARRAY_INT array,ARRAY_STRING > array,ARRAY_DATE array,CARD_COUNT int,DEBIT_COUNT int, > CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) using carbon; > insert into Array_com select * from array_com_hive; > select * from array_com_hive order by CUST_ID ASC limit 3; > select * from array_com order by CUST_ID ASC limit 3; > *Expected result:* > There should be no data mismatch and data in table should be same as it is in > CSV file. > *Actual result:* > Data mismatch is seen. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2892) Data mismatch is seen in the Array-String and Array-Timestamp.
[ https://issues.apache.org/jira/browse/CARBONDATA-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Ravichandran updated CARBONDATA-2892: -- Attachment: Array.csv > Data mismatch is seen in the Array-String and Array-Timestamp. > -- > > Key: CARBONDATA-2892 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2892 > Project: CarbonData > Issue Type: Bug > Environment: 3 Node ANT. >Reporter: Prasanna Ravichandran >Priority: Major > Attachments: Array.csv > > > Data mismatch is seen in the Array-String and Array-Timestamp like mismatch > in data, order, date values. > *Test queries:* > drop table if exists array_com_hive; > create table array_com_hive (CUST_ID string, YEAR int, MONTH int, AGE int, > GENDER string, EDUCATED string, IS_MARRIED string, ARRAY_INT > array,ARRAY_STRING array,ARRAY_DATE array,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) row > format delimited fields terminated by ',' collection items terminated by '$'; > load data local inpath '/opt/csv/complex/Array.csv' into table array_com_hive; > drop table if exists array_com; > create table Array_com (CUST_ID string, YEAR int, MONTH int, AGE int, GENDER > string, EDUCATED string, IS_MARRIED string, ARRAY_INT array,ARRAY_STRING > array,ARRAY_DATE array,CARD_COUNT int,DEBIT_COUNT int, > CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) using carbon; > insert into Array_com select * from array_com_hive; > select * from array_com_hive order by CUST_ID ASC limit 3; > select * from array_com order by CUST_ID ASC limit 3; > *Expected result:* > There should be no data mismatch and data in table should be same as it is in > CSV file. > *Actual result:* > Data mismatch is seen. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2893) Job aborted during insert while loading the "Struct of Array" datatype values.
[ https://issues.apache.org/jira/browse/CARBONDATA-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Ravichandran updated CARBONDATA-2893: -- Attachment: structofarray.csv > Job aborted during insert while loading the "Struct of Array" datatype values. > -- > > Key: CARBONDATA-2893 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2893 > Project: CarbonData > Issue Type: Bug > Environment: 3 Node ANT. >Reporter: Prasanna Ravichandran >Priority: Major > Attachments: structofarray.csv > > > Job aborted during insert while loading the "Struct of Array" datatype values. > *Test queries:* > 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com_hive; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.026 seconds) > 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com_hive (CUST_ID string, YEAR > int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, > STRUCT_OF_ARRAY struct,sal1: > array,state: array,date1: array>,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT float, HQ_DEPOSIT double) row > format delimited fields terminated by ',' collection items terminated by '$' > map keys terminated by '&'; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.159 seconds) > 0: jdbc:hive2:> load data local inpath '/opt/csv/complex/structofarray.csv' > into table STRUCT_OF_ARRAY_com_hive; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.217 seconds) > 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.03 seconds) > 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com (CUST_ID string, YEAR int, > MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, > STRUCT_OF_ARRAY struct,sal1: > array,state: array,date1: array>,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) > using carbon; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.099 seconds) > 0: jdbc:hive2:> insert into STRUCT_OF_ARRAY_com select * from > STRUCT_OF_ARRAY_com_hive; > *Error: org.apache.spark.SparkException: Job aborted. (state=,code=0)* > > *Expected result:* > Insert should be success. > *Actual result:* > Insert is showing job aborted. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2893) Job aborted during insert while loading the "Struct of Array" datatype values.
[ https://issues.apache.org/jira/browse/CARBONDATA-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Ravichandran updated CARBONDATA-2893: -- Attachment: (was: structofarray.csv) > Job aborted during insert while loading the "Struct of Array" datatype values. > -- > > Key: CARBONDATA-2893 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2893 > Project: CarbonData > Issue Type: Bug > Environment: 3 Node ANT. >Reporter: Prasanna Ravichandran >Priority: Major > Attachments: structofarray.csv > > > Job aborted during insert while loading the "Struct of Array" datatype values. > *Test queries:* > 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com_hive; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.026 seconds) > 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com_hive (CUST_ID string, YEAR > int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, > STRUCT_OF_ARRAY struct,sal1: > array,state: array,date1: array>,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT float, HQ_DEPOSIT double) row > format delimited fields terminated by ',' collection items terminated by '$' > map keys terminated by '&'; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.159 seconds) > 0: jdbc:hive2:> load data local inpath '/opt/csv/complex/structofarray.csv' > into table STRUCT_OF_ARRAY_com_hive; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.217 seconds) > 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.03 seconds) > 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com (CUST_ID string, YEAR int, > MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, > STRUCT_OF_ARRAY struct,sal1: > array,state: array,date1: array>,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) > using carbon; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.099 seconds) > 0: jdbc:hive2:> insert into STRUCT_OF_ARRAY_com select * from > STRUCT_OF_ARRAY_com_hive; > *Error: org.apache.spark.SparkException: Job aborted. (state=,code=0)* > > *Expected result:* > Insert should be success. > *Actual result:* > Insert is showing job aborted. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2893) Job aborted during insert while loading the "Struct of Array" datatype values.
[ https://issues.apache.org/jira/browse/CARBONDATA-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Ravichandran updated CARBONDATA-2893: -- Attachment: structofarray.csv > Job aborted during insert while loading the "Struct of Array" datatype values. > -- > > Key: CARBONDATA-2893 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2893 > Project: CarbonData > Issue Type: Bug > Environment: 3 Node ANT. >Reporter: Prasanna Ravichandran >Priority: Major > Attachments: structofarray.csv > > > Job aborted during insert while loading the "Struct of Array" datatype values. > *Test queries:* > 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com_hive; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.026 seconds) > 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com_hive (CUST_ID string, YEAR > int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, > STRUCT_OF_ARRAY struct,sal1: > array,state: array,date1: array>,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT float, HQ_DEPOSIT double) row > format delimited fields terminated by ',' collection items terminated by '$' > map keys terminated by '&'; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.159 seconds) > 0: jdbc:hive2:> load data local inpath '/opt/csv/complex/structofarray.csv' > into table STRUCT_OF_ARRAY_com_hive; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.217 seconds) > 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.03 seconds) > 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com (CUST_ID string, YEAR int, > MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, > STRUCT_OF_ARRAY struct,sal1: > array,state: array,date1: array>,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) > using carbon; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.099 seconds) > 0: jdbc:hive2:> insert into STRUCT_OF_ARRAY_com select * from > STRUCT_OF_ARRAY_com_hive; > *Error: org.apache.spark.SparkException: Job aborted. (state=,code=0)* > > *Expected result:* > Insert should be success. > *Actual result:* > Insert is showing job aborted. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2893) Job aborted during insert while loading the "Struct of Array" datatype values.
[ https://issues.apache.org/jira/browse/CARBONDATA-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Ravichandran updated CARBONDATA-2893: -- Attachment: (was: arrayofstruct.csv) > Job aborted during insert while loading the "Struct of Array" datatype values. > -- > > Key: CARBONDATA-2893 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2893 > Project: CarbonData > Issue Type: Bug > Environment: 3 Node ANT. >Reporter: Prasanna Ravichandran >Priority: Major > Attachments: structofarray.csv > > > Job aborted during insert while loading the "Struct of Array" datatype values. > *Test queries:* > 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com_hive; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.026 seconds) > 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com_hive (CUST_ID string, YEAR > int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, > STRUCT_OF_ARRAY struct,sal1: > array,state: array,date1: array>,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT float, HQ_DEPOSIT double) row > format delimited fields terminated by ',' collection items terminated by '$' > map keys terminated by '&'; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.159 seconds) > 0: jdbc:hive2:> load data local inpath '/opt/csv/complex/structofarray.csv' > into table STRUCT_OF_ARRAY_com_hive; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.217 seconds) > 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.03 seconds) > 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com (CUST_ID string, YEAR int, > MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, > STRUCT_OF_ARRAY struct,sal1: > array,state: array,date1: array>,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) > using carbon; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.099 seconds) > 0: jdbc:hive2:> insert into STRUCT_OF_ARRAY_com select * from > STRUCT_OF_ARRAY_com_hive; > *Error: org.apache.spark.SparkException: Job aborted. (state=,code=0)* > > *Expected result:* > Insert should be success. > *Actual result:* > Insert is showing job aborted. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2893) Job aborted during insert while loading the "Struct of Array" datatype values.
[ https://issues.apache.org/jira/browse/CARBONDATA-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Ravichandran updated CARBONDATA-2893: -- Description: Job aborted during insert while loading the "Struct of Array" datatype values. *Test queries:* 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com_hive; +--+-+ |Result| +--+-+ +--+-+ No rows selected (0.026 seconds) 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com_hive (CUST_ID string, YEAR int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, STRUCT_OF_ARRAY struct,sal1: array,state: array,date1: array>,CARD_COUNT int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT float, HQ_DEPOSIT double) row format delimited fields terminated by ',' collection items terminated by '$' map keys terminated by '&'; +--+-+ |Result| +--+-+ +--+-+ No rows selected (0.159 seconds) 0: jdbc:hive2:> load data local inpath '/opt/csv/complex/structofarray.csv' into table STRUCT_OF_ARRAY_com_hive; +--+-+ |Result| +--+-+ +--+-+ No rows selected (0.217 seconds) 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com; +--+-+ |Result| +--+-+ +--+-+ No rows selected (0.03 seconds) 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com (CUST_ID string, YEAR int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, STRUCT_OF_ARRAY struct,sal1: array,state: array,date1: array>,CARD_COUNT int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) using carbon; +--+-+ |Result| +--+-+ +--+-+ No rows selected (0.099 seconds) 0: jdbc:hive2:> insert into STRUCT_OF_ARRAY_com select * from STRUCT_OF_ARRAY_com_hive; *Error: org.apache.spark.SparkException: Job aborted. (state=,code=0)* *Expected result:* Insert should be success. *Actual result:* Insert is showing job aborted. was: Job aborted during insert while loading the "Struct of Array" datatype values. *Test queries:* 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com_hive; +-+--+ | Result | +-+--+ +-+--+ No rows selected (0.026 seconds) 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com_hive (CUST_ID string, YEAR int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, STRUCT_OF_ARRAY struct,sal1: array,state: array,date1: array>,CARD_COUNT int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT float, HQ_DEPOSIT double) row format delimited fields terminated by ',' collection items terminated by '$' map keys terminated by '&'; +-+--+ | Result | +-+--+ +-+--+ No rows selected (0.159 seconds) 0: jdbc:hive2:> load data local inpath '/opt/csv/complex/structofarray.csv' into table STRUCT_OF_ARRAY_com_hive; +-+--+ | Result | +-+--+ +-+--+ No rows selected (0.217 seconds) 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com; +-+--+ | Result | +-+--+ +-+--+ No rows selected (0.03 seconds) 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com (CUST_ID string, YEAR int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, STRUCT_OF_ARRAY struct,sal1: array,state: array,date1: array>,CARD_COUNT int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) using carbon; +-+--+ | Result | +-+--+ +-+--+ No rows selected (0.099 seconds) 0: jdbc:hive2:> insert into STRUCT_OF_ARRAY_com select * from STRUCT_OF_ARRAY_com_hive; *Error: org.apache.spark.SparkException: Job aborted. (state=,code=0)* > Job aborted during insert while loading the "Struct of Array" datatype values. > -- > > Key: CARBONDATA-2893 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2893 > Project: CarbonData > Issue Type: Bug > Environment: 3 Node ANT. >Reporter: Prasanna Ravichandran >Priority: Major > Attachments: arrayofstruct.csv > > > Job aborted during insert while loading the "Struct of Array" datatype values. > *Test queries:* > 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com_hive; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.026 seconds) > 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com_hive (CUST_ID string, YEAR > int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, > STRUCT_OF_ARRAY struct,sal1: > array,state: array,date1: array>,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT float, HQ_DEPOSIT double) row > format delimited fields terminated by ',' collection items terminated by '$' > map keys terminated by '&'; > +--+-+ > |Result| > +--+-+ > +--+-+ > No rows selected (0.159 seconds) > 0:
[jira] [Created] (CARBONDATA-2893) Job aborted during insert while loading the "Struct of Array" datatype values.
Prasanna Ravichandran created CARBONDATA-2893: - Summary: Job aborted during insert while loading the "Struct of Array" datatype values. Key: CARBONDATA-2893 URL: https://issues.apache.org/jira/browse/CARBONDATA-2893 Project: CarbonData Issue Type: Bug Environment: 3 Node ANT. Reporter: Prasanna Ravichandran Attachments: arrayofstruct.csv Job aborted during insert while loading the "Struct of Array" datatype values. *Test queries:* 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com_hive; +-+--+ | Result | +-+--+ +-+--+ No rows selected (0.026 seconds) 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com_hive (CUST_ID string, YEAR int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, STRUCT_OF_ARRAY struct,sal1: array,state: array,date1: array>,CARD_COUNT int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT float, HQ_DEPOSIT double) row format delimited fields terminated by ',' collection items terminated by '$' map keys terminated by '&'; +-+--+ | Result | +-+--+ +-+--+ No rows selected (0.159 seconds) 0: jdbc:hive2:> load data local inpath '/opt/csv/complex/structofarray.csv' into table STRUCT_OF_ARRAY_com_hive; +-+--+ | Result | +-+--+ +-+--+ No rows selected (0.217 seconds) 0: jdbc:hive2:> drop table if exists STRUCT_OF_ARRAY_com; +-+--+ | Result | +-+--+ +-+--+ No rows selected (0.03 seconds) 0: jdbc:hive2:> create table STRUCT_OF_ARRAY_com (CUST_ID string, YEAR int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, STRUCT_OF_ARRAY struct,sal1: array,state: array,date1: array>,CARD_COUNT int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) using carbon; +-+--+ | Result | +-+--+ +-+--+ No rows selected (0.099 seconds) 0: jdbc:hive2:> insert into STRUCT_OF_ARRAY_com select * from STRUCT_OF_ARRAY_com_hive; *Error: org.apache.spark.SparkException: Job aborted. (state=,code=0)* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2641: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2641 retest this please ---
[GitHub] carbondata issue #2654: [WIP] Adaptive Encoding for Primitive data types
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/27/ ---
[jira] [Created] (CARBONDATA-2892) Data mismatch is seen in the Array-String and Array-Timestamp.
Prasanna Ravichandran created CARBONDATA-2892: - Summary: Data mismatch is seen in the Array-String and Array-Timestamp. Key: CARBONDATA-2892 URL: https://issues.apache.org/jira/browse/CARBONDATA-2892 Project: CarbonData Issue Type: Bug Environment: 3 Node ANT. Reporter: Prasanna Ravichandran Attachments: Array.csv Data mismatch is seen in the Array-String and Array-Timestamp like mismatch in data, order, date values. *Test queries:* drop table if exists array_com_hive; create table array_com_hive (CUST_ID string, YEAR int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, ARRAY_INT array,ARRAY_STRING array,ARRAY_DATE array,CARD_COUNT int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) row format delimited fields terminated by ',' collection items terminated by '$'; load data local inpath '/opt/csv/complex/Array.csv' into table array_com_hive; drop table if exists array_com; create table Array_com (CUST_ID string, YEAR int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, ARRAY_INT array,ARRAY_STRING array,ARRAY_DATE array,CARD_COUNT int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) using carbon; insert into Array_com select * from array_com_hive; select * from array_com_hive order by CUST_ID ASC limit 3; select * from array_com order by CUST_ID ASC limit 3; *Expected result:* There should be no data mismatch and data in table should be same as it is in CSV file. *Actual result:* Data mismatch is seen. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2654: [WIP] Adaptive Encoding for Primitive data types
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/2654 retest this please ---
[GitHub] carbondata issue #2623: [CARBONDATA-2844] [CARBONDATA-2865] Pass SK/AK to ex...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2623 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8089/ ---
[jira] [Created] (CARBONDATA-2891) Job aborted while loading long string 32k data into carbon table from hive
Rahul Singha created CARBONDATA-2891:
Summary: Job aborted while loading long string 32k data into
carbon table from hive
Key: CARBONDATA-2891
URL: https://issues.apache.org/jira/browse/CARBONDATA-2891
Project: CarbonData
Issue Type: Bug
Components: data-load
Reporter: Rahul Singha
_*Steps:*_
CREATE TABLE local1(id int, name string, description string,address string,
note string) using carbon options('long_string_columns'='description,note');
CREATE TABLE local_hive(id int, name string, description string,address string,
note string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
LOAD DATA local INPATH '/opt/csv/longStringData_100rec.csv' overwrite into
table local_hive;
insert into local1 select * from local_hive;
_*Expected result:*_
Data should get loaded.
_*Actual Result:*_
Error: org.apache.spark.SparkException: Job aborted. (state=,code=0)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2659 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/26/ ---
[GitHub] carbondata issue #2654: [WIP] Adaptive Encoding for Primitive data types
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2654 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8088/ ---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2628 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8090/ ---
[GitHub] carbondata issue #2628: [CARBONDATA-2851][CARBONDATA-2852] Support zstd as c...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2628 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6425/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2659 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8087/ ---
[GitHub] carbondata issue #2623: [CARBONDATA-2844] [CARBONDATA-2865] Pass SK/AK to ex...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2623 retest this please ---
[GitHub] carbondata issue #2623: [CARBONDATA-2844] [CARBONDATA-2865] Pass SK/AK to ex...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2623 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/25/ ---
[GitHub] carbondata issue #2654: [WIP] Adaptive Encoding for Primitive data types
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2654 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6424/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2659 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6423/ ---
[GitHub] carbondata issue #2635: [CARBONDATA-2856][BloomDataMap] Fix bug in bloom ind...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2635 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/24/ ---
[jira] [Created] (CARBONDATA-2890) Use CarbonLoadModelBuilder instead of new CarbonLoadModel instance
xuchuanyin created CARBONDATA-2890: -- Summary: Use CarbonLoadModelBuilder instead of new CarbonLoadModel instance Key: CARBONDATA-2890 URL: https://issues.apache.org/jira/browse/CARBONDATA-2890 Project: CarbonData Issue Type: Sub-task Reporter: xuchuanyin Currently to get an instance of CarbonLoadModel, we can: 1. directly new an instance and set the member one by one; 2. or use the CarbonLoadModelBuilder to build an instance However some of the members of CarbonLoadModel (such as ColumnCompressor, tableName) are required in the following procedure. For the 1st method, these members may be forgotten to initialize. While for the 2nd method, we can validate these members in the build method to ensure that these members are initialized. So here I proposed to only use the CarbonLoadModelBuilder to instantiate a CarbonLoadModel. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2623: [CARBONDATA-2844] [CARBONDATA-2865] Pass SK/AK to ex...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2623 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8085/ ---
[GitHub] carbondata issue #2659: [CARBONDATA-2887] Fix complex filters on spark carbo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2659 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6422/ ---
[GitHub] carbondata issue #2635: [CARBONDATA-2856][BloomDataMap] Fix bug in bloom ind...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2635 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8084/ ---
[GitHub] carbondata issue #2662: [WIP][CARBONDATA-2889]Add decoder based fallback mec...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2662 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/23/ ---
[GitHub] carbondata issue #2623: [CARBONDATA-2844] [CARBONDATA-2865] Pass SK/AK to ex...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2623 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6421/ ---
[GitHub] carbondata pull request #2661: [CARBONDATA-2888] Support multi level subfold...
Github user xuchuanyin commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2661#discussion_r212897009 --- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/SegmentIndexFileStore.java --- @@ -338,6 +338,24 @@ private MergedBlockIndex readMergeBlockIndex(ThriftReader thriftReader) throws I }); } + /** + * List all the index files of the segment. + * + * @param carbonFile directory + */ + public static void getCarbonIndexFilesRecursively(CarbonFile carbonFile, --- End diff -- Why not return the list instead of passing it as a parameter? ---
[GitHub] carbondata pull request #2661: [CARBONDATA-2888] Support multi level subfold...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2661#discussion_r212897508
--- Diff:
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
---
@@ -268,6 +257,18 @@ public boolean accept(CarbonFile file) {
return CarbonTable.buildFromTableInfo(tableInfoInfer);
}
+ private static CarbonFile getFirstIndexFile(CarbonFile tablePath) {
+CarbonFile[] carbonFiles = tablePath.listFiles();
+for (CarbonFile carbonFile : carbonFiles) {
+ if (carbonFile.isDirectory()) {
+return getFirstIndexFile(carbonFile);
+ } else if
(carbonFile.getName().endsWith(CarbonTablePath.INDEX_FILE_EXT)) {
+return carbonFile;
+ }
+}
+return null;
--- End diff --
why not just throw exception if the file does not exist?
---
[GitHub] carbondata issue #2635: [CARBONDATA-2856][BloomDataMap] Fix bug in bloom ind...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2635 retest this please ---
[GitHub] carbondata issue #2623: [CARBONDATA-2844] Pass SK/AK to executor by serializ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2623 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/22/ ---
[GitHub] carbondata issue #2662: [WIP][CARBONDATA-2889]Add decoder based fallback mec...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2662 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8083/ ---
[jira] [Assigned] (CARBONDATA-2885) Broadcast Issue and Small file distribution Issue
[ https://issues.apache.org/jira/browse/CARBONDATA-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar vishal reassigned CARBONDATA-2885: Assignee: Babulal > Broadcast Issue and Small file distribution Issue > - > > Key: CARBONDATA-2885 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2885 > Project: CarbonData > Issue Type: Bug >Reporter: Babulal >Assignee: Babulal >Priority: Major > > Carbon Relation size is getting calculated wrongly ( always 0 ) for External > Table. > Root Cause:- Because Tablestatus file is not present for external table > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2885) Broadcast Issue and Small file distribution Issue
[ https://issues.apache.org/jira/browse/CARBONDATA-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar vishal resolved CARBONDATA-2885. -- Resolution: Fixed > Broadcast Issue and Small file distribution Issue > - > > Key: CARBONDATA-2885 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2885 > Project: CarbonData > Issue Type: Bug >Reporter: Babulal >Assignee: Babulal >Priority: Major > > Carbon Relation size is getting calculated wrongly ( always 0 ) for External > Table. > Root Cause:- Because Tablestatus file is not present for external table > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2658: [Carbondata 2885]Broadcast Issue and Small fi...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2658 ---
[GitHub] carbondata issue #2623: [CARBONDATA-2844] Pass SK/AK to executor by serializ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2623 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8082/ ---
[GitHub] carbondata issue #2662: [WIP][CARBONDATA-2889]Add decoder based fallback mec...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2662 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6420/ ---
[GitHub] carbondata issue #2649: [CARBONDATA-2869] Add support for Avro Map data type...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2649 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/21/ ---
[GitHub] carbondata issue #2649: [CARBONDATA-2869] Add support for Avro Map data type...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2649 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8080/ ---
[jira] [Created] (CARBONDATA-2889) Support Decoder based fall back mechanism in Local Dictionary
Akash R Nilugal created CARBONDATA-2889: --- Summary: Support Decoder based fall back mechanism in Local Dictionary Key: CARBONDATA-2889 URL: https://issues.apache.org/jira/browse/CARBONDATA-2889 Project: CarbonData Issue Type: Sub-task Reporter: Akash R Nilugal Assignee: Akash R Nilugal Currently, when the fallback is initiated for a column page in case of local dictionary, we are keeping both encoded data and actual data in memory and then we form the new column page without dictionary encoding and then at last we free the Encoded Column Page. Because of this offheap memory footprint increases. We can reduce the offheap memory footprint. This can be done using decoder based fallback mechanism. This means, no need to keep the actual data along with encoded data in encoded column page. We can keep only encoded data and to form a new column page, get the dictionary data from encoded column page by uncompressing and using dictionary data get the actual data using local dictionary generator and put it in new column page created and compress it again and give to consumer for writing blocklet. The above process may slow down the loading, but it will reduces the memory footprint. So we can give a property which will decide whether to take current fallback procedure or decoder based fallback mechanism dring fallback -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2662: [CARBONDATA-2889]Add decoder based fallback m...
GitHub user akashrn5 opened a pull request: https://github.com/apache/carbondata/pull/2662 [CARBONDATA-2889]Add decoder based fallback mechanism in local dictionary to reduce memory footprint Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/akashrn5/incubator-carbondata fallback Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2662.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2662 commit 8729c673ed06ee16ed256270e31c495bd1568bfd Author: akashrn5 Date: 2018-08-20T04:59:26Z Add decoder based fallback mechanism in local dictionary to reduce memory footprint ---
[GitHub] carbondata issue #2623: [CARBONDATA-2844] Pass SK/AK to executor by serializ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2623 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6419/ ---
[GitHub] carbondata issue #2623: [CARBONDATA-2844] Pass SK/AK to executor by serializ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2623 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6418/ ---
