[jira] [Updated] (CARBONDATA-2729) Schema Compatibility problem between version 1.3.0 and 1.4.0
[
https://issues.apache.org/jira/browse/CARBONDATA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Shahid Khan updated CARBONDATA-2729:
-
Description:
The table schema created with 1.3 is not compatible with 1.4.0.
Steps:
Do the below operation on hive meta store.
1. Create table and load data in 1.3 version.
{code:java}
spark.sql("create database carbonversion_1_1");
spark.sql("use carbonversion_1_1");
spark.sql("""create table testIntType (c1 string,c2 int) STORED BY
'org.apache.carbondata.format'""")
spark.sql("insert into testIntType select 'a',200"){code}
2. Switch to 1.4 version and create data map on the table created in step 1.
{code:java}
spark.sql("use carbonversion_1_3");
spark.sql("create datamap dm8 on table testinttype3 using 'preaggregate' as
select c1, sum(c2) from testinttype3 group by c1")
{code}
stack trace
{code:java}
stack trace
Exception in thread "main" java.lang.NullPointerException
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalSchemaEvolution(ThriftWrapperSchemaConverterImpl.java:87)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableSchema(ThriftWrapperSchemaConverterImpl.java:277)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableInfo(ThriftWrapperSchemaConverterImpl.java:310)
at
org.apache.spark.sql.hive.CarbonHiveMetaStore.getThriftTableInfo(CarbonHiveMetaStore.scala:101){code}
was:
The table schema created with 1.3 is not compatible with 1.4.0.
stack trace
{code:java}
stack trace
Exception in thread "main" java.lang.NullPointerException
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalSchemaEvolution(ThriftWrapperSchemaConverterImpl.java:87)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableSchema(ThriftWrapperSchemaConverterImpl.java:277)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableInfo(ThriftWrapperSchemaConverterImpl.java:310)
at
org.apache.spark.sql.hive.CarbonHiveMetaStore.getThriftTableInfo(CarbonHiveMetaStore.scala:101){code}
> Schema Compatibility problem between version 1.3.0 and 1.4.0
>
>
> Key: CARBONDATA-2729
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2729
> Project: CarbonData
> Issue Type: Bug
> Components: file-format
>Affects Versions: 1.4.0
>Reporter: Mohammad Shahid Khan
>Assignee: Mohammad Shahid Khan
>Priority: Major
> Fix For: 1.4.1
>
>
> The table schema created with 1.3 is not compatible with 1.4.0.
> Steps:
> Do the below operation on hive meta store.
> 1. Create table and load data in 1.3 version.
> {code:java}
> spark.sql("create database carbonversion_1_1");
> spark.sql("use carbonversion_1_1");
> spark.sql("""create table testIntType (c1 string,c2 int) STORED BY
> 'org.apache.carbondata.format'""")
> spark.sql("insert into testIntType select 'a',200"){code}
> 2. Switch to 1.4 version and create data map on the table created in step 1.
> {code:java}
> spark.sql("use carbonversion_1_3");
> spark.sql("create datamap dm8 on table testinttype3 using 'preaggregate' as
> select c1, sum(c2) from testinttype3 group by c1")
> {code}
> stack trace
> {code:java}
> stack trace
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalSchemaEvolution(ThriftWrapperSchemaConverterImpl.java:87)
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableSchema(ThriftWrapperSchemaConverterImpl.java:277)
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableInfo(ThriftWrapperSchemaConverterImpl.java:310)
> at
> org.apache.spark.sql.hive.CarbonHiveMetaStore.getThriftTableInfo(CarbonHiveMetaStore.scala:101){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #2486: [CARBONDATA-2723][DataMap] Fix bugs in recreate data...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2486 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5785/ ---
[jira] [Updated] (CARBONDATA-2729) Schema Compatibility problem between version 1.3.0 and 1.4.0
[
https://issues.apache.org/jira/browse/CARBONDATA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Shahid Khan updated CARBONDATA-2729:
-
Description:
The table schema created with 1.3 is not compatible with 1.4.0.
stack trace
{code:java}
stack trace
Exception in thread "main" java.lang.NullPointerException
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalSchemaEvolution(ThriftWrapperSchemaConverterImpl.java:87)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableSchema(ThriftWrapperSchemaConverterImpl.java:277)
at
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableInfo(ThriftWrapperSchemaConverterImpl.java:310)
at
org.apache.spark.sql.hive.CarbonHiveMetaStore.getThriftTableInfo(CarbonHiveMetaStore.scala:101){code}
was:
The table schema created with 1.3 is not compatible with 1.4.0.
> Schema Compatibility problem between version 1.3.0 and 1.4.0
>
>
> Key: CARBONDATA-2729
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2729
> Project: CarbonData
> Issue Type: Bug
> Components: file-format
>Affects Versions: 1.4.0
>Reporter: Mohammad Shahid Khan
>Assignee: Mohammad Shahid Khan
>Priority: Major
> Fix For: 1.4.1
>
>
> The table schema created with 1.3 is not compatible with 1.4.0.
> stack trace
> {code:java}
> stack trace
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalSchemaEvolution(ThriftWrapperSchemaConverterImpl.java:87)
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableSchema(ThriftWrapperSchemaConverterImpl.java:277)
> at
> org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableInfo(ThriftWrapperSchemaConverterImpl.java:310)
> at
> org.apache.spark.sql.hive.CarbonHiveMetaStore.getThriftTableInfo(CarbonHiveMetaStore.scala:101){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #2354: [CARBONDATA-2720] Remove dead code
Github user gvramana commented on the issue: https://github.com/apache/carbondata/pull/2354 LGTM ---
[GitHub] carbondata issue #2354: [CARBONDATA-2720] Remove dead code
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2354 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5820/ ---
[GitHub] carbondata issue #2463: [CARBONDATA-2708][BloomDataMap] clear index file in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2463 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5818/ ---
[GitHub] carbondata issue #2354: [CARBONDATA-2720] Remove dead code
Github user sraghunandan commented on the issue: https://github.com/apache/carbondata/pull/2354 retest sdv please ---
[GitHub] carbondata issue #2493: [CARBONDATA-2637][DataMap] Fix bugs in rebuild datam...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2493 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5784/ ---
[GitHub] carbondata issue #2495: Added for kafka integration with Carbon StreamSQL
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2495 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2445: [CARBONDATA-2655][BloomDataMap] BloomFilter datamap ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2445 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5819/ ---
[GitHub] carbondata issue #2495: Added for kafka integration with Carbon StreamSQL
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2495 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2495: Added for kafka integration with Carbon StreamSQL
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2495 Can one of the admins verify this patch? ---
[GitHub] carbondata pull request #2495: Added for kafka integration with Carbon Strea...
GitHub user ajithme opened a pull request: https://github.com/apache/carbondata/pull/2495 Added for kafka integration with Carbon StreamSQL 1. Pass source table properties to streamReader.load() 2. Do not pass schema when sparkSession.readStream 3. Remove querySchema validation against sink as dataFrame made from kafka source will not have schema ( its written in value column of schema ) 4. Extract the dataframe from kafka source which contain actual data schema @ writeStream Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NO - [ ] Any backward compatibility impacted? NO - [ ] Document update required? Yes: Need to use CSV parser - [ ] Testing done Done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajithme/carbondata kafkaStreamSQLIntegration Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2495.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2495 commit 0560c5e69c61d6594a91994da918493335bd0cb4 Author: Ajith Date: 2018-07-12T03:47:22Z Added for kafka integration with Carbon StreamSQL 1. Pass source table properties to streamReader.load() 2. Do not pass schema when sparkSession.readStream 3. Remove querySchema validation against sink as dataFrame made from kafka source will not have schema ( its written in value column of schema ) 4. Extract the dataframe from kafka source which contain actual data schema @ writeStream ---
[GitHub] carbondata issue #2354: [CARBONDATA-2720] Remove dead code
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2354 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7043/ ---
[jira] [Created] (CARBONDATA-2736) Support Kafka Source for CREATE STREAM
Ajith S created CARBONDATA-2736: --- Summary: Support Kafka Source for CREATE STREAM Key: CARBONDATA-2736 URL: https://issues.apache.org/jira/browse/CARBONDATA-2736 Project: CarbonData Issue Type: Bug Reporter: Ajith S Attachments: image-2018-07-12-11-58-49-346.png When CREATE STREAM over a kafka source. It will throw a IllegalArgumentException !image-2018-07-12-11-58-49-346.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2489: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2489 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5783/ ---
[GitHub] carbondata issue #2445: [CARBONDATA-2655][BloomDataMap] BloomFilter datamap ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2445 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7042/ ---
[GitHub] carbondata issue #2493: [CARBONDATA-2637][DataMap] Fix bugs in rebuild datam...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2493 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5782/ ---
[GitHub] carbondata issue #2354: [CARBONDATA-2720] Remove dead code
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2354 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5817/ ---
[GitHub] carbondata issue #2463: [CARBONDATA-2708][BloomDataMap] clear index file in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2463 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7041/ ---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2490 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5816/ ---
[GitHub] carbondata issue #2354: [CARBONDATA-2720] Remove dead code
Github user sraghunandan commented on the issue: https://github.com/apache/carbondata/pull/2354 retest this please ---
[GitHub] carbondata pull request #2445: [CARBONDATA-2655][BloomDataMap] BloomFilter d...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2445#discussion_r201891618
--- Diff:
datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java
---
@@ -186,19 +188,39 @@ public void initIndexColumnConverters(CarbonTable
carbonTable, List models =
+ buildQueryModelForIn((ColumnExpression) left,
(ListExpression) right);
+ queryModels.addAll(models);
+}
+return queryModels;
+ } else if (left instanceof ListExpression && right instanceof
ColumnExpression) {
+column = ((ColumnExpression) right).getColumnName();
+if (this.name2Col.containsKey(column)) {
+ List models =
+ buildQueryModelForIn((ColumnExpression) right,
(ListExpression) left);
+ queryModels.addAll(models);
+}
+ }
--- End diff --
Each index datamap has a set of supported operators, for bloomfilter
datamap it supports `equal` and `in`, so there are only two branches here.
will add comment here
---
[GitHub] carbondata issue #2354: [CARBONDATA-2720] Remove dead code
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2354 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5781/ ---
[GitHub] carbondata pull request #2445: [CARBONDATA-2655][BloomDataMap] BloomFilter d...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2445#discussion_r201891274
--- Diff:
datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java
---
@@ -186,19 +188,39 @@ public void initIndexColumnConverters(CarbonTable
carbonTable, List models =
+ buildQueryModelForIn((ColumnExpression) left,
(ListExpression) right);
+ queryModels.addAll(models);
+}
+return queryModels;
+ } else if (left instanceof ListExpression && right instanceof
ColumnExpression) {
+column = ((ColumnExpression) right).getColumnName();
+if (this.name2Col.containsKey(column)) {
+ List models =
+ buildQueryModelForIn((ColumnExpression) right,
(ListExpression) left);
+ queryModels.addAll(models);
+}
+ }
--- End diff --
What if it does not fit into previous two if branch? can you add an else to
throw exception and explain it will not come to else?
---
[GitHub] carbondata issue #2445: [CARBONDATA-2655][BloomDataMap] BloomFilter datamap ...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2445 LGTM ---
[GitHub] carbondata issue #2354: [CARBONDATA-2720] Remove dead code
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2354 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7040/ ---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2490 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7039/ ---
[GitHub] carbondata issue #2463: [CARBONDATA-2708][BloomDataMap] clear index file in ...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2463 retest this please ---
[GitHub] carbondata issue #2486: [CARBONDATA-2723][DataMap] Fix bugs in recreate data...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2486 retest sdv please ---
[GitHub] carbondata issue #2487: [CARBONDATA-2735]Fixed Performance issue for complex...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2487 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5780/ ---
[GitHub] carbondata issue #2493: [HOTFIX][DataMap] Fix bugs in rebuild datamap
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2493 retest SDV please ---
[GitHub] carbondata issue #2489: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user sraghunandan commented on the issue: https://github.com/apache/carbondata/pull/2489 retest sdv please ---
[GitHub] carbondata issue #2493: [HOTFIX][DataMap] Fix bugs in rebuild datamap
Github user sraghunandan commented on the issue: https://github.com/apache/carbondata/pull/2493 retest sdv please ---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Github user kevinjmh commented on the issue: https://github.com/apache/carbondata/pull/2490 retest this please ---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2490 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5779/ ---
[GitHub] carbondata issue #2482: [CARBONDATA-2714] Support merge index files for the ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2482 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5814/ ---
[GitHub] carbondata issue #2472: [CARBONDATA-2717] fixed table id empty problem while...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2472 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5815/ ---
[GitHub] carbondata issue #2481: [CARBONDATA-2688][CarbonStore] Support SQL in REST A...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2481 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5778/ ---
[GitHub] carbondata issue #2472: [CARBONDATA-2717] fixed table id empty problem while...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2472 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5777/ ---
[GitHub] carbondata issue #2489: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2489 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5812/ ---
[GitHub] carbondata issue #2484: [WIP] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5813/ ---
[GitHub] carbondata issue #2494: [CARBONDATA-2734] Fix struct of date issue in create...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2494 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5810/ ---
[GitHub] carbondata issue #2488: [CARBONDATA-2724][DataMap]Unsupported create datamap...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2488 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5776/ ---
[GitHub] carbondata issue #2493: [HOTFIX][DataMap] Fix bugs in rebuild datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2493 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5811/ ---
[GitHub] carbondata issue #2489: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2489 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5775/ ---
[GitHub] carbondata issue #2489: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2489 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7037/ ---
[GitHub] carbondata issue #2487: [CARBONDATA-2735]Fixed Performance issue for complex...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2487 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5774/ ---
[GitHub] carbondata issue #2484: [WIP] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7038/ ---
[GitHub] carbondata issue #2481: [CARBONDATA-2688][CarbonStore] Support SQL in REST A...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2481 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5808/ ---
[GitHub] carbondata issue #2492: [CARBONDATA-2733]should support scalar sub query for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2492 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2472: [CARBONDATA-2717] fixed table id empty problem while...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2472 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7036/ ---
[GitHub] carbondata issue #2463: [CARBONDATA-2708][BloomDataMap] clear index file in ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2463 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5773/ ---
[GitHub] carbondata issue #2493: [HOTFIX][DataMap] Fix bugs in rebuild datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2493 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7034/ ---
[GitHub] carbondata issue #2484: [WIP] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7035/ ---
[GitHub] carbondata issue #2494: [CARBONDATA-2734] Fix struct of date issue in create...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2494 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7033/ ---
[GitHub] carbondata issue #2487: [CARBONDATA-2735]Fixed Performance issue for complex...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2487 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5806/ ---
[GitHub] carbondata issue #2484: [WIP] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5807/ ---
[GitHub] carbondata issue #2488: [CARBONDATA-2724][DataMap]Unsupported create datamap...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2488 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5805/ ---
[GitHub] carbondata issue #2481: [CARBONDATA-2688][CarbonStore] Support SQL in REST A...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2481 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5772/ ---
[GitHub] carbondata issue #2482: [CARBONDATA-2714] Support merge index files for the ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2482 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7031/ ---
[GitHub] carbondata issue #2492: [CARBONDATA-2733]should support scalar sub query for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2492 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2481: [CARBONDATA-2688][CarbonStore] Support SQL in REST A...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2481 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7032/ ---
[jira] [Resolved] (CARBONDATA-2712) Add fix for Local Dictionary Exclude for multi level complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar vishal resolved CARBONDATA-2712. -- Resolution: Fixed > Add fix for Local Dictionary Exclude for multi level complex columns > > > Key: CARBONDATA-2712 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2712 > Project: CarbonData > Issue Type: Sub-task >Reporter: Praveen M P >Assignee: Praveen M P >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2469: [CARBONDATA-2712] Added fix for Local Diction...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2469 ---
[GitHub] carbondata issue #2469: [CARBONDATA-2712] Added fix for Local Dictionary Exc...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2469 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5771/ ---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2490 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5803/ ---
[GitHub] carbondata issue #2489: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2489 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5804/ ---
[GitHub] carbondata issue #2487: [CARBONDATA-2735]Fixed Performance issue for complex...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2487 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7029/ ---
[GitHub] carbondata issue #2484: [WIP] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7030/ ---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2490 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7028/ ---
[GitHub] carbondata issue #2472: [CARBONDATA-2717] fixed table id empty problem while...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2472 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5770/ ---
[jira] [Assigned] (CARBONDATA-2735) Performance issue for complex array data type when number of elements in array is more
[ https://issues.apache.org/jira/browse/CARBONDATA-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar vishal reassigned CARBONDATA-2735: Assignee: kumar vishal > Performance issue for complex array data type when number of elements in > array is more > -- > > Key: CARBONDATA-2735 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2735 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal >Priority: Major > > *Problem* > Query is taking more time when number of elements in array type is more. > *Root Cause* > This is because offset of array values are stored in list and when number of > elements are high creating a big list to store offset is taking more time > *Solution:* > Instead of list use column page to store offset so when number of elements > are high there will not be any gc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2735) Performance issue for complex array data type when number of elements in array is more
kumar vishal created CARBONDATA-2735: Summary: Performance issue for complex array data type when number of elements in array is more Key: CARBONDATA-2735 URL: https://issues.apache.org/jira/browse/CARBONDATA-2735 Project: CarbonData Issue Type: Bug Reporter: kumar vishal *Problem* Query is taking more time when number of elements in array type is more. *Root Cause* This is because offset of array values are stored in list and when number of elements are high creating a big list to store offset is taking more time *Solution:* Instead of list use column page to store offset so when number of elements are high there will not be any gc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2494: [CARBONDATA-2734] Fix struct of date issue in...
GitHub user ajantha-bhat opened a pull request: https://github.com/apache/carbondata/pull/2494 [CARBONDATA-2734] Fix struct of date issue in create table problem: Struct of date is not supported currently in create table flow as date datatype check is missing during parsing. Hence child date column was not appended with parent name, leading to StringOutOfIndex exception. solution: Handle the date DataType as a complex child during column formation. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed?NA - [ ] Any backward compatibility impacted?NA - [ ] Document update required?NA - [ ] Testing done. Updated the UT. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajantha-bhat/carbondata issue_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2494.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2494 commit e8c6e49f12df5d2f4a17e1de05f8a14cf7605c94 Author: ajantha-bhat Date: 2018-07-11T11:58:45Z [CARBONDATA-2734] support struct of date in create table problem: Struct of date is not supported currently in create table flow as date datatype check is missing during parsing. Hence child date column was not appended with parent name, leading to StringOutOfIndex excpetion. solution: Handle the date DataType as a complex child during column formation. ---
[jira] [Updated] (CARBONDATA-2734) [BUG] support struct of date in create table
[ https://issues.apache.org/jira/browse/CARBONDATA-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-2734: - Description: Currently due to code issue. StringOutOfBound exception will be thrown > [BUG] support struct of date in create table > > > Key: CARBONDATA-2734 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2734 > Project: CarbonData > Issue Type: Sub-task >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > > Currently due to code issue. StringOutOfBound exception will be thrown -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2734) [BUG] support struct of date in create table
Ajantha Bhat created CARBONDATA-2734: Summary: [BUG] support struct of date in create table Key: CARBONDATA-2734 URL: https://issues.apache.org/jira/browse/CARBONDATA-2734 Project: CarbonData Issue Type: Sub-task Reporter: Ajantha Bhat Assignee: Ajantha Bhat -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2481: [CARBONDATA-2688][CarbonStore] Support SQL in REST A...
Github user QiangCai commented on the issue: https://github.com/apache/carbondata/pull/2481 Merge to carbonstore branch ---
[GitHub] carbondata issue #2472: [CARBONDATA-2717] fixed table id empty problem while...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2472 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7027/ ---
[GitHub] carbondata issue #2452: [CARBONDATA-2693][BloomDataMap]Fix bug for alter ren...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2452 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5802/ ---
[GitHub] carbondata issue #2463: [CARBONDATA-2708][BloomDataMap] clear index file in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2463 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5801/ ---
[GitHub] carbondata issue #2487: [WIP]Fixed Complex type array GC issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2487 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7026/ ---
[GitHub] carbondata issue #2484: [WIP] added hadoop conf to thread local
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2484 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5769/ ---
[GitHub] carbondata pull request #2493: [HOTFIX][DataMap] Fix bugs in rebuild datamap
GitHub user xuchuanyin opened a pull request: https://github.com/apache/carbondata/pull/2493 [HOTFIX][DataMap] Fix bugs in rebuild datamap In cluster mode, readCommitScope is null while rebuilding datamap for segments, this will cause NPE. Here we use the origin segment object whose readCommitScope is not null and will work fine. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xuchuanyin/carbondata 0711_bug_rebuild_segment Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2493.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2493 commit f33f0a1544c06fa71c187b3c07e0e31049171846 Author: xuchuanyin Date: 2018-07-11T14:24:54Z Fix bugs in rebuild datamap In cluster mode, readCommitScope is null while rebuilding datamap for segments, this will cause NPE. Here we use the origin segment object whose readCommitScope is not null and will work fine. ---
[GitHub] carbondata issue #2486: [CARBONDATA-2723][DataMap] Fix bugs in recreate data...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2486 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5768/ ---
[GitHub] carbondata issue #2492: [CARBONDATA-2733]should support scalar sub query for...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2492 Can one of the admins verify this patch? ---
[GitHub] carbondata pull request #2492: [CARBONDATA-2733]should support scalar sub qu...
GitHub user sandeep-katta opened a pull request: https://github.com/apache/carbondata/pull/2492 [CARBONDATA-2733]should support scalar sub query for partition tables 1)Multiple Projection in plan are supported 2)Scalar sub queries are supported for Carbon Tables ## What changes were proposed in this pull request? In this PR carbon tables will support scalar sub queries for Carbon Tables. If multiple projections are present in the plan,carbon should support it Highlights: In CarbonLateDecodeStrategy need to filter the queries which does not support sub-queries In CarbonLateDecodeStrategy if one of the projection of the plan is failed,catch the exception and continue ## How was this patch tested? UT Code You can merge this pull request into a Git repository by running: $ git pull https://github.com/sandeep-katta/carbondata sparkSupport Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2492.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2492 commit 8ea495718064ae5aa948dc088d939c66b5039297 Author: sandeep-katta Date: 2018-07-11T13:50:19Z [CARBONDATA-2733]Carbon should support scalar sub query for partition tables 1)Multiple Projection in plan are supported 2)Scalar sub queries are supported for Carbon Tables ---
[GitHub] carbondata issue #2488: [CARBONDATA-2724][DataMap]Unsupported create datamap...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2488 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7025/ ---
[jira] [Updated] (CARBONDATA-2733) Carbon should support scalar sub query for partition tables
[ https://issues.apache.org/jira/browse/CARBONDATA-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandeep katta updated CARBONDATA-2733: -- Issue Type: Bug (was: Improvement) > Carbon should support scalar sub query for partition tables > --- > > Key: CARBONDATA-2733 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2733 > Project: CarbonData > Issue Type: Bug >Reporter: sandeep katta >Priority: Major > > 1) Carbon needs to support multiple projections on spark2.3.1 > 2)As a part of SPARK-24085 spark supports scalar sub query ,so it is required > to support for carbon tables also -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2469: [CARBONDATA-2712] Added fix for Local Dictionary Exc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2469 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5800/ ---
[jira] [Created] (CARBONDATA-2733) Carbon should support scalar sub query for partition tables
sandeep katta created CARBONDATA-2733: - Summary: Carbon should support scalar sub query for partition tables Key: CARBONDATA-2733 URL: https://issues.apache.org/jira/browse/CARBONDATA-2733 Project: CarbonData Issue Type: Improvement Reporter: sandeep katta 1) Carbon needs to support multiple projections on spark2.3.1 2)As a part of SPARK-24085 spark supports scalar sub query ,so it is required to support for carbon tables also -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2491: [CARBONDATA-2700][BloomDataMap] Block dropping index...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2491 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2489: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2489 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7024/ ---
[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2485 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5767/ ---
[GitHub] carbondata issue #2491: [CARBONDATA-2700][BloomDataMap] Block dropping index...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2491 better to change the title of the PR ---
[GitHub] carbondata issue #2481: [CARBONDATA-2688][CarbonStore] Support SQL in REST A...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2481 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5799/ ---
[GitHub] carbondata issue #2491: [CARBONDATA-2700][BloomDataMap] Block dropping index...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2491 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2452: [CARBONDATA-2693][BloomDataMap]Fix bug for alter ren...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2452 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7023/ ---
[jira] [Assigned] (CARBONDATA-2682) create table with long_string_columns property
[
https://issues.apache.org/jira/browse/CARBONDATA-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lianganping reassigned CARBONDATA-2682:
---
Assignee: lianganping
> create table with long_string_columns property
> --
>
> Key: CARBONDATA-2682
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2682
> Project: CarbonData
> Issue Type: Sub-task
> Components: spark-integration
>Affects Versions: 1.4.1
>Reporter: lianganping
>Assignee: lianganping
>Priority: Minor
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> 1. create table with partition and no_inverted_index no long_string_columns
> should be blocked. eg: create table local_no_inverted_index0(id int,name
> string,description string,address string,note string) stored by 'carbondata'
> tblproperties('no_inverted_index'='note','long_string_columns'='note') ;
> create table local_partion(id int,name string,description string,address
> string) partition by(note string) stored by 'carbondata'
> tblproperties('long_string_columns'='note')
> 2. create table with duplicate long_string_column should be blocked.
> eg:create table local_longstringtbl(id int,name string,description
> string,address string,note string) stored by 'carbondata'
> tblproperties('long_string_columns'='note,note') ;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CARBONDATA-2732) Block create bloomfilter datamap index on column which its datatype is complex type
lianganping created CARBONDATA-2732: --- Summary: Block create bloomfilter datamap index on column which its datatype is complex type Key: CARBONDATA-2732 URL: https://issues.apache.org/jira/browse/CARBONDATA-2732 Project: CarbonData Issue Type: Sub-task Reporter: lianganping Assignee: lianganping -- This message was sent by Atlassian JIRA (v7.6.3#76005)
