[GitHub] [carbondata] kunal642 commented on a change in pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.
kunal642 commented on a change in pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961#discussion_r500753133
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/merge/CarbonMergeDataSetCommand.scala
##
@@ -106,18 +106,22 @@ case class CarbonMergeDataSetCommand(
// decide join type based on match conditions
val joinType = decideJoinType
-val joinColumn = mergeMatches.joinExpr.expr.asInstanceOf[EqualTo].left
- .asInstanceOf[UnresolvedAttribute].nameParts.tail.head
-// repartition the srsDs, if the target has bucketing and the bucketing
column and join column
-// are same
+val joinColumns = mergeMatches.joinExpr.expr.collect {
+ case unresolvedAttribute: UnresolvedAttribute if
unresolvedAttribute.nameParts.nonEmpty =>
+unresolvedAttribute.nameParts.tail.head
Review comment:
please write a comment explaining this logic
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (CARBONDATA-3872) IndexOutOfBoundsException in ResizableArray
[ https://issues.apache.org/jira/browse/CARBONDATA-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17209293#comment-17209293 ] Akshay commented on CARBONDATA-3872: Please provide the full query that is causing this failure. > IndexOutOfBoundsException in ResizableArray > --- > > Key: CARBONDATA-3872 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3872 > Project: CarbonData > Issue Type: Bug > Components: sql >Affects Versions: 1.5.1 >Reporter: SeaAndHill >Priority: Critical > Attachments: carbondata.png > > > carbondata 在通过in 查询时，in 是子查询，运行报数组越界， 对应spark 版本是2.2.1， hadoop 版本是 2.7.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (CARBONDATA-3872) IndexOutOfBoundsException in ResizableArray
[ https://issues.apache.org/jira/browse/CARBONDATA-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3872: --- Comment: was deleted (was: Please provide the query that is failing.) > IndexOutOfBoundsException in ResizableArray > --- > > Key: CARBONDATA-3872 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3872 > Project: CarbonData > Issue Type: Bug > Components: sql >Affects Versions: 1.5.1 >Reporter: SeaAndHill >Priority: Critical > Attachments: carbondata.png > > > carbondata 在通过in 查询时，in 是子查询，运行报数组越界， 对应spark 版本是2.2.1， hadoop 版本是 2.7.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-3872) IndexOutOfBoundsException in ResizableArray
[ https://issues.apache.org/jira/browse/CARBONDATA-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17209292#comment-17209292 ] Akshay commented on CARBONDATA-3872: Please provide the query that is failing. > IndexOutOfBoundsException in ResizableArray > --- > > Key: CARBONDATA-3872 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3872 > Project: CarbonData > Issue Type: Bug > Components: sql >Affects Versions: 1.5.1 >Reporter: SeaAndHill >Priority: Critical > Attachments: carbondata.png > > > carbondata 在通过in 查询时，in 是子查询，运行报数组越界， 对应spark 版本是2.2.1， hadoop 版本是 2.7.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-4016) NPE and FileNotFound in Show Segments and Insert Stage
[ https://issues.apache.org/jira/browse/CARBONDATA-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-4016. -- Resolution: Fixed > NPE and FileNotFound in Show Segments and Insert Stage > -- > > Key: CARBONDATA-4016 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4016 > Project: CarbonData > Issue Type: Bug > Components: flink-integration, spark-integration >Affects Versions: 2.0.1 >Reporter: Xingjun Hao >Priority: Minor > Fix For: 2.1.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > # Insert Stage, While Spark read Stages which are writting by Flink in the > meanwhile, JSONFORMAT EXCEPTION will be thrown. > # Show Segments with STAGE, when read stages which are writting by Flink or > deleting by spark. JSONFORMAT EXCEPTION will be thrown > # Show Segment will load partition info for non-partition table, which shall > be avoided. > # In getLastModifiedTime of TableStatus, if the loadendtime is empty, > getLastModifiedTime throw NPE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] asfgit closed pull request #3965: [CARBONDATA-4016] NPE and FileNotFound in Show Segments and Insert Stage
asfgit closed pull request #3965: URL: https://github.com/apache/carbondata/pull/3965 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #3965: [CARBONDATA-4016] NPE and FileNotFound in Show Segments and Insert Stage
kunal642 commented on pull request #3965: URL: https://github.com/apache/carbondata/pull/3965#issuecomment-704692149 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (CARBONDATA-4004) Wrong result in Presto select query after executing update
[ https://issues.apache.org/jira/browse/CARBONDATA-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4004: --- Summary: Wrong result in Presto select query after executing update (was: Wrong result in Presto select query in case of inverted index) > Wrong result in Presto select query after executing update > -- > > Key: CARBONDATA-4004 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4004 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Reporter: Akshay >Priority: Major > > Presto select query after update operation with inverted index enabled always > returns 0 rows as result. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4004) Wrong result in Presto select query after executing update
[ https://issues.apache.org/jira/browse/CARBONDATA-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4004: --- Description: Presto select query after update operation returns different number of rows. (was: Presto select query after update operation with inverted index enabled always returns 0 rows as result.) > Wrong result in Presto select query after executing update > -- > > Key: CARBONDATA-4004 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4004 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Reporter: Akshay >Priority: Major > > Presto select query after update operation returns different number of rows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3969: [WIP] document update
CarbonDataQA1 commented on pull request #3969: URL: https://github.com/apache/carbondata/pull/3969#issuecomment-704404206 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on pull request #3963: [CARBONDATA-4018]Fix CSV header validation not contains dimension columns
VenuReddy2103 commented on pull request #3963: URL: https://github.com/apache/carbondata/pull/3963#issuecomment-704376213 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3959: [CARBONDATA-4010] Doc changes for long strings.
CarbonDataQA1 commented on pull request #3959: URL: https://github.com/apache/carbondata/pull/3959#issuecomment-704325591 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4316/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3959: [CARBONDATA-4010] Doc changes for long strings.
CarbonDataQA1 commented on pull request #3959: URL: https://github.com/apache/carbondata/pull/3959#issuecomment-704324417 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2566/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG opened a new pull request #3969: [WIP] document update
ShreelekhyaG opened a new pull request #3969: URL: https://github.com/apache/carbondata/pull/3969 ### Why is this PR needed? Few document changes in dml, presto-guide. ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No ### Is any new testcase added? - No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on a change in pull request #3959: [CARBONDATA-4010] Doc changes for long strings.
nihal0107 commented on a change in pull request #3959: URL: https://github.com/apache/carbondata/pull/3959#discussion_r500254558 ## File path: docs/ddl-of-carbondata.md ## @@ -426,7 +426,8 @@ CarbonData DDL statements are documented here,which includes: - # String longer than 32000 characters In common scenarios, the length of string is less than 32000, - so carbondata stores the length of content using Short to reduce memory and space consumption. + so carbondata stores the length of content using Short to reduce memory and space consumption, and it handles the strings + which length greater than 32000 as bad record. Refer [bad record handling](https://github.com/apache/carbondata/blob/master/docs/dml-of-carbondata.md#bad-records-handling) section for better understanding. Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (CARBONDATA-2868) Create Table DDL support for Map DataType
[ https://issues.apache.org/jira/browse/CARBONDATA-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi Muthumurugesh resolved CARBONDATA-2868. -- Resolution: Fixed > Create Table DDL support for Map DataType > - > > Key: CARBONDATA-2868 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2868 > Project: CarbonData > Issue Type: Sub-task >Reporter: Indhumathi Muthumurugesh >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-2867) Load DDL support for Map DataType
[ https://issues.apache.org/jira/browse/CARBONDATA-2867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi Muthumurugesh resolved CARBONDATA-2867. -- Resolution: Fixed > Load DDL support for Map DataType > - > > Key: CARBONDATA-2867 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2867 > Project: CarbonData > Issue Type: Sub-task >Reporter: Indhumathi Muthumurugesh >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-3739) Select with order by columns not in projection gives wrong results
[ https://issues.apache.org/jira/browse/CARBONDATA-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi Muthumurugesh resolved CARBONDATA-3739. -- Resolution: Fixed > Select with order by columns not in projection gives wrong results > -- > > Key: CARBONDATA-3739 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3739 > Project: CarbonData > Issue Type: Bug >Reporter: Indhumathi Muthumurugesh >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-3594) Optimize getSplits() during compaction
[ https://issues.apache.org/jira/browse/CARBONDATA-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi Muthumurugesh resolved CARBONDATA-3594. -- Resolution: Fixed > Optimize getSplits() during compaction > -- > > Key: CARBONDATA-3594 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3594 > Project: CarbonData > Issue Type: Improvement >Reporter: Indhumathi Muthumurugesh >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-3750) Support segment level MInMax for Secondary Index
[ https://issues.apache.org/jira/browse/CARBONDATA-3750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi Muthumurugesh resolved CARBONDATA-3750. -- Resolution: Fixed > Support segment level MInMax for Secondary Index > > > Key: CARBONDATA-3750 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3750 > Project: CarbonData > Issue Type: Sub-task >Reporter: Indhumathi Muthumurugesh >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-3890) Fix MV case sensitive issues with ImplicitCastInputTypes and Add Doc for Show MV
[ https://issues.apache.org/jira/browse/CARBONDATA-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi Muthumurugesh resolved CARBONDATA-3890. -- Resolution: Fixed > Fix MV case sensitive issues with ImplicitCastInputTypes and Add Doc for Show > MV > > > Key: CARBONDATA-3890 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3890 > Project: CarbonData > Issue Type: Bug >Reporter: Indhumathi Muthumurugesh >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-3931) Secondary index with index column as DateType gives wrong results
[ https://issues.apache.org/jira/browse/CARBONDATA-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi Muthumurugesh resolved CARBONDATA-3931. -- Resolution: Fixed > Secondary index with index column as DateType gives wrong results > - > > Key: CARBONDATA-3931 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3931 > Project: CarbonData > Issue Type: Bug >Reporter: Indhumathi Muthumurugesh >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (CARBONDATA-3146) Support Load data using Json for CarbonSession
[ https://issues.apache.org/jira/browse/CARBONDATA-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi Muthumurugesh closed CARBONDATA-3146. Resolution: Later > Support Load data using Json for CarbonSession > -- > > Key: CARBONDATA-3146 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3146 > Project: CarbonData > Issue Type: New Feature >Reporter: Indhumathi Muthumurugesh >Assignee: Indhumathi Muthumurugesh >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] Indhumathi27 commented on pull request #3962: [CARBONDATA-4017]Fix the insert issue when the column name contains '\' and fix SI creation issue
Indhumathi27 commented on pull request #3962: URL: https://github.com/apache/carbondata/pull/3962#issuecomment-704178705 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3916: [CARBONDATA-3935]Support partition table transactional write in presto
CarbonDataQA1 commented on pull request #3916: URL: https://github.com/apache/carbondata/pull/3916#issuecomment-704164989 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4314/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (CARBONDATA-3867) Show materialized views command not documented in https://github.com/apache/carbondata/blob/master/docs/mv-guide.md
[ https://issues.apache.org/jira/browse/CARBONDATA-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Bhat closed CARBONDATA-3867. --- Fix Version/s: 2.1.0 Resolution: Fixed Updated in [https://github.com/apache/carbondata/blob/master/docs/mv-guide.md.] Defect closed. > Show materialized views command not documented in > https://github.com/apache/carbondata/blob/master/docs/mv-guide.md > --- > > Key: CARBONDATA-3867 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3867 > Project: CarbonData > Issue Type: Bug > Components: docs >Affects Versions: 2.0.0 > Environment: > https://github.com/apache/carbondata/blob/master/docs/mv-guide.md >Reporter: Chetan Bhat >Priority: Minor > Fix For: 2.1.0 > > > Show materialized views command not documented in > https://github.com/apache/carbondata/blob/master/docs/mv-guide.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (CARBONDATA-3949) Select filter query fails from presto-cli on MV table
[
https://issues.apache.org/jira/browse/CARBONDATA-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Bhat closed CARBONDATA-3949.
---
Fix Version/s: 2.1.0
Resolution: Fixed
Limitation Updated in docs -
[https://github.com/apache/carbondata/blob/master/docs/prestosql-guide.md]
Defect is closed.
> Select filter query fails from presto-cli on MV table
> -
>
> Key: CARBONDATA-3949
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3949
> Project: CarbonData
> Issue Type: Bug
> Components: presto-integration
>Affects Versions: 2.0.1
> Environment: Spark 2.4.5. PrestoSQL 316
>Reporter: Chetan Bhat
>Priority: Major
> Fix For: 2.1.0
>
>
> From sparksql create table , load data and create MV
> spark-sql> CREATE TABLE uniqdata(CUST_ID int,CUST_NAME
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp,
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10),
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2
> double,INTEGER_COLUMN1 int) STORED as carbondata
> TBLPROPERTIES('local_dictionary_enable'='true','local_dictionary_threshold'='1000');
> Time taken: 0.753 seconds
> spark-sql> LOAD DATA INPATH 'hdfs://hacluster/chetan/2000_UniqData.csv' into
> table uniqdata OPTIONS('DELIMITER'=',',
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> OK
> OK
> Time taken: 1.992 seconds
> spark-sql> CREATE MATERIALIZED VIEW mv1 as select cust_id, cust_name,
> count(cust_id) from uniqdata group by cust_id, cust_name;
> OK
> Time taken: 4.336 seconds
>
> From presto cli select filter query on table with MV fails.
> presto:chetan> select * from uniqdata where CUST_ID IS NULL or BIGINT_COLUMN1
> =1233720368578 or DECIMAL_COLUMN1 = 12345678901.123458 or Double_COLUMN1
> = 1.12345674897976E10 or INTEGER_COLUMN1 IS NULL ;
> Query 20200804_092703_00253_ed34h failed: Unable to get file status:
> *Log-*
> 2020-08-04T18:09:55.975+0800 INFO Query-20200804_100955_00300_ed34h-2642
> stdout 2020-08-04 18:09:55 WARN AbstractDFSCarbonFile:458 - Exception
> occurred: File
> hdfs://hacluster/user/sparkhive/warehouse/chetan.db/uniqdata_string/Metadata
> does not exist.
> java.io.FileNotFoundException: File
> hdfs://hacluster/user/sparkhive/warehouse/chetan.db/uniqdata_string/Metadata
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1058)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1118)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1115)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1125)
> at
> org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:270)
> at
> org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.listFiles(AbstractDFSCarbonFile.java:456)
> at
> org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.listFiles(AbstractDFSCarbonFile.java:559)
> at
> org.apache.carbondata.core.util.path.CarbonTablePath.getActualSchemaFilePath(CarbonTablePath.java:189)
> at
> org.apache.carbondata.core.util.path.CarbonTablePath.getSchemaFilePath(CarbonTablePath.java:168)
> at
> org.apache.carbondata.presto.impl.CarbonTableReader.updateSchemaTables(CarbonTableReader.java:147)
> at
> org.apache.carbondata.presto.impl.CarbonTableReader.getCarbonCache(CarbonTableReader.java:128)
> at
> org.apache.carbondata.presto.CarbondataSplitManager.getSplits(CarbondataSplitManager.java:145)
> at
> io.prestosql.spi.connector.classloader.ClassLoaderSafeConnectorSplitManager.getSplits(ClassLoaderSafeConnectorSplitManager.java:50)
> at io.prestosql.split.SplitManager.getSplits(SplitManager.java:85)
> at
> io.prestosql.sql.planner.DistributedExecutionPlanner$Visitor.visitScanAndFilter(DistributedExecutionPlanner.java:189)
> at
> io.prestosql.sql.planner.DistributedExecutionPlanner$Visitor.visitFilter(DistributedExecutionPlanner.java:257)
> at
> io.prestosql.sql.planner.DistributedExecutionPlanner$Visitor.visitFilter(DistributedExecutionPlanner.java:149)
> at io.prestosql.sql.planner.plan.FilterNode.accept(FilterNode.java:72)
> at
> io.prestosql.sql.planner.DistributedExecutionPlanner.doPlan(DistributedExecutionPlanner.java:119)
> at
>
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3916: [CARBONDATA-3935]Support partition table transactional write in presto
CarbonDataQA1 commented on pull request #3916: URL: https://github.com/apache/carbondata/pull/3916#issuecomment-704164039 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2564/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.
CarbonDataQA1 commented on pull request #3875: URL: https://github.com/apache/carbondata/pull/3875#issuecomment-704139470 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2562/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3875: [CARBONDATA-3934]Support write transactional table with presto.
CarbonDataQA1 commented on pull request #3875: URL: https://github.com/apache/carbondata/pull/3875#issuecomment-704138259 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4312/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (CARBONDATA-3806) Create bloom datamap fails with null pointer exception
[
https://issues.apache.org/jira/browse/CARBONDATA-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Bhat closed CARBONDATA-3806.
---
Fix Version/s: 2.1.0
Resolution: Fixed
fixed in latest Carbon 2.1 B06 build
> Create bloom datamap fails with null pointer exception
> --
>
> Key: CARBONDATA-3806
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3806
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 1.6.1
> Environment: Spark 2.3.2
>Reporter: Chetan Bhat
>Priority: Major
> Fix For: 2.1.0
>
>
> Create bloom datamap fails with null pointer exception
> create table brinjal_bloom (imei string,AMSize string,channelsId
> string,ActiveCountry string, Activecity string,gamePointId
> double,deviceInformationId double,productionDate Timestamp,deliveryDate
> timestamp,deliverycharge double) STORED BY 'carbondata'
> TBLPROPERTIES('table_blocksize'='1');
> LOAD DATA INPATH 'hdfs://hacluster/chetan/vardhandaterestruct.csv' INTO TABLE
> brinjal_bloom OPTIONS('DELIMITER'=',', 'QUOTECHAR'=
> '"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'=
> 'imei,deviceInformationId,AMSize,channelsId,ActiveCountry,Activecity,gamePointId,productionDate,deliveryDate,deliverycharge');
> 0: jdbc:hive2://10.20.255.171:23040/default> CREATE DATAMAP dm_brinjal4 ON
> TABLE brinjal_bloom USING 'bloomfilter' DMPROPERTIES ('INDEX_COLUMNS' =
> 'AMSize', 'BLOOM_SIZE'='64', 'BLOOM_FPP'='0.1');
> Error: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 0 in stage 210.0 failed 4 times, most recent failure: Lost task 0.3 in
> stage 210.0 (TID 1477, vm2, executor 2): java.lang.NullPointerException
> at
> org.apache.carbondata.core.datamap.Segment.getCommittedIndexFile(Segment.java:150)
> at
> org.apache.carbondata.core.util.BlockletDataMapUtil.getTableBlockUniqueIdentifiers(BlockletDataMapUtil.java:198)
> at
> org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapFactory.getTableBlockIndexUniqueIdentifiers(BlockletDataMapFactory.java:176)
> at
> org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapFactory.getDataMaps(BlockletDataMapFactory.java:154)
> at
> org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapFactory.getSegmentProperties(BlockletDataMapFactory.java:425)
> at
> org.apache.carbondata.datamap.IndexDataMapRebuildRDD.internalCompute(IndexDataMapRebuildRDD.scala:359)
> at org.apache.carbondata.spark.rdd.CarbonRDD.compute(CarbonRDD.scala:84)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:109)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Driver stacktrace: (state=,code=0)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CARBONDATA-3806) Create bloom datamap fails with null pointer exception
[
https://issues.apache.org/jira/browse/CARBONDATA-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17208575#comment-17208575
]
Nihal kumar ojha commented on CARBONDATA-3806:
--
This issue was handled in
PR: https://github.com/apache/carbondata/pull/3775
> Create bloom datamap fails with null pointer exception
> --
>
> Key: CARBONDATA-3806
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3806
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 1.6.1
> Environment: Spark 2.3.2
>Reporter: Chetan Bhat
>Priority: Major
>
> Create bloom datamap fails with null pointer exception
> create table brinjal_bloom (imei string,AMSize string,channelsId
> string,ActiveCountry string, Activecity string,gamePointId
> double,deviceInformationId double,productionDate Timestamp,deliveryDate
> timestamp,deliverycharge double) STORED BY 'carbondata'
> TBLPROPERTIES('table_blocksize'='1');
> LOAD DATA INPATH 'hdfs://hacluster/chetan/vardhandaterestruct.csv' INTO TABLE
> brinjal_bloom OPTIONS('DELIMITER'=',', 'QUOTECHAR'=
> '"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'=
> 'imei,deviceInformationId,AMSize,channelsId,ActiveCountry,Activecity,gamePointId,productionDate,deliveryDate,deliverycharge');
> 0: jdbc:hive2://10.20.255.171:23040/default> CREATE DATAMAP dm_brinjal4 ON
> TABLE brinjal_bloom USING 'bloomfilter' DMPROPERTIES ('INDEX_COLUMNS' =
> 'AMSize', 'BLOOM_SIZE'='64', 'BLOOM_FPP'='0.1');
> Error: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 0 in stage 210.0 failed 4 times, most recent failure: Lost task 0.3 in
> stage 210.0 (TID 1477, vm2, executor 2): java.lang.NullPointerException
> at
> org.apache.carbondata.core.datamap.Segment.getCommittedIndexFile(Segment.java:150)
> at
> org.apache.carbondata.core.util.BlockletDataMapUtil.getTableBlockUniqueIdentifiers(BlockletDataMapUtil.java:198)
> at
> org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapFactory.getTableBlockIndexUniqueIdentifiers(BlockletDataMapFactory.java:176)
> at
> org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapFactory.getDataMaps(BlockletDataMapFactory.java:154)
> at
> org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapFactory.getSegmentProperties(BlockletDataMapFactory.java:425)
> at
> org.apache.carbondata.datamap.IndexDataMapRebuildRDD.internalCompute(IndexDataMapRebuildRDD.scala:359)
> at org.apache.carbondata.spark.rdd.CarbonRDD.compute(CarbonRDD.scala:84)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:109)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Driver stacktrace: (state=,code=0)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3842: [CARBONDATA-3702] Clean temp index files in parallel in merge index flow
CarbonDataQA1 commented on pull request #3842: URL: https://github.com/apache/carbondata/pull/3842#issuecomment-704121200 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4311/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3959: [CARBONDATA-4010] Doc changes for long strings.
Indhumathi27 commented on a change in pull request #3959: URL: https://github.com/apache/carbondata/pull/3959#discussion_r500099594 ## File path: docs/ddl-of-carbondata.md ## @@ -426,7 +426,8 @@ CarbonData DDL statements are documented here,which includes: - # String longer than 32000 characters In common scenarios, the length of string is less than 32000, - so carbondata stores the length of content using Short to reduce memory and space consumption. + so carbondata stores the length of content using Short to reduce memory and space consumption, and it handles the strings + which length greater than 32000 as bad record. Refer [bad record handling](https://github.com/apache/carbondata/blob/master/docs/dml-of-carbondata.md#bad-records-handling) section for better understanding. Review comment: ```suggestion and it handles strings which have length greater than 32000 as a bad record. Refer [bad record handling](https://github.com/apache/carbondata/blob/master/docs/dml-of-carbondata.md#bad-records-handling) section for better understanding. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3959: [CARBONDATA-4010] Doc changes for long strings.
Indhumathi27 commented on a change in pull request #3959: URL: https://github.com/apache/carbondata/pull/3959#discussion_r500099594 ## File path: docs/ddl-of-carbondata.md ## @@ -426,7 +426,8 @@ CarbonData DDL statements are documented here,which includes: - # String longer than 32000 characters In common scenarios, the length of string is less than 32000, - so carbondata stores the length of content using Short to reduce memory and space consumption. + so carbondata stores the length of content using Short to reduce memory and space consumption, and it handles the strings + which length greater than 32000 as bad record. Refer [bad record handling](https://github.com/apache/carbondata/blob/master/docs/dml-of-carbondata.md#bad-records-handling) section for better understanding. Review comment: ```suggestion and it handles strings which have length greater than 32000 as a bad record. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3968: [WIP] Partition optimization
CarbonDataQA1 commented on pull request #3968: URL: https://github.com/apache/carbondata/pull/3968#issuecomment-704112498 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2563/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (CARBONDATA-3912) Clean file requests are failing in case of multiple load due to concurrent locking.
[ https://issues.apache.org/jira/browse/CARBONDATA-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal kumar ojha resolved CARBONDATA-3912. -- Fix Version/s: 2.1.0 Resolution: Fixed This issue was handled in PR: https://github.com/apache/carbondata/pull/3871 > Clean file requests are failing in case of multiple load due to concurrent > locking. > --- > > Key: CARBONDATA-3912 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3912 > Project: CarbonData > Issue Type: Bug >Reporter: Nihal kumar ojha >Priority: Minor > Fix For: 2.1.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > If multiple loads are fired at the same time then clean file requests are > failing due to failing in lock acquiring. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3968: [WIP] Partition optimization
CarbonDataQA1 commented on pull request #3968: URL: https://github.com/apache/carbondata/pull/3968#issuecomment-704110350 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4313/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (CARBONDATA-3972) Date/timestamp compatability between hive and carbon
[ https://issues.apache.org/jira/browse/CARBONDATA-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SHREELEKHYA GAMPA closed CARBONDATA-3972. - Resolution: Invalid > Date/timestamp compatability between hive and carbon > - > > Key: CARBONDATA-3972 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3972 > Project: CarbonData > Issue Type: Bug >Reporter: SHREELEKHYA GAMPA >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > To ensure the date/timestamp that is supported by hive also to be supported > by carbon. > Ex: -01-01 is accepted by hive as a valid record and converted to > 0001-01-01. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] maheshrajus opened a new pull request #3968: [WIP] Partition optimization
maheshrajus opened a new pull request #3968: URL: https://github.com/apache/carbondata/pull/3968 ### Why is this PR needed? Partition optimization ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (CARBONDATA-3797) Refresh materialized view command throws null pointer exception
[
https://issues.apache.org/jira/browse/CARBONDATA-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Bhat closed CARBONDATA-3797.
---
Fix Version/s: 2.1.0
Resolution: Fixed
Issue fixed in Carbon 2.1.0
> Refresh materialized view command throws null pointer exception
> ---
>
> Key: CARBONDATA-3797
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3797
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 2.0.0
> Environment: Spark 2.3.2, Spark 2.4.5
>Reporter: Chetan Bhat
>Priority: Major
> Fix For: 2.1.0
>
>
> Refresh materialized view command throws null pointer exception
> CREATE TABLE uniqdata_mv(CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
> string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
> bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
> decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
> int) STORED as carbondata
> TBLPROPERTIES('local_dictionary_enable'='true','local_dictionary_threshold'='1000');
> LOAD DATA INPATH 'hdfs://hacluster/chetan/2000_UniqData.csv' into table
> uniqdata_mv OPTIONS('DELIMITER'=',',
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> CREATE MATERIALIZED VIEW mv1 as select cust_id, cust_name, count(cust_id)
> from uniqdata_mv group by cust_id, cust_name;
> refresh MATERIALIZED VIEW mv1;
> Error: java.lang.NullPointerException (state=,code=0)
>
> *Exception-*
> 2020-05-06 00:50:59,941 | ERROR | [HiveServer2-Background-Pool: Thread-1822]
> | Error executing query, currentState RUNNING, |
> org.apache.spark.internal.Logging$class.logError(Logging.scala:91)
> java.lang.NullPointerException
> at org.apache.carbondata.view.MVRefresher$.refresh(MVRefresher.scala:62)
> at
> org.apache.spark.sql.execution.command.view.CarbonRefreshMVCommand.processData(CarbonRefreshMVCommand.scala:52)
> at
> org.apache.spark.sql.execution.command.DataCommand$$anonfun$run$2.apply(package.scala:132)
> at
> org.apache.spark.sql.execution.command.DataCommand$$anonfun$run$2.apply(package.scala:132)
> at
> org.apache.spark.sql.execution.command.Auditable$class.runWithAudit(package.scala:104)
> at
> org.apache.spark.sql.execution.command.DataCommand.runWithAudit(package.scala:130)
> at org.apache.spark.sql.execution.command.DataCommand.run(package.scala:132)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
> at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
> at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
> at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
> at
> org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
> at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)
> at org.apache.spark.sql.Dataset.(Dataset.scala:194)
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
> at
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:232)
> at
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:175)
> at
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:185)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3963: [CARBONDATA-4018]Fix CSV header validation not contains dimension columns
CarbonDataQA1 commented on pull request #3963: URL: https://github.com/apache/carbondata/pull/3963#issuecomment-704085705 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2560/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3963: [CARBONDATA-4018]Fix CSV header validation not contains dimension columns
CarbonDataQA1 commented on pull request #3963: URL: https://github.com/apache/carbondata/pull/3963#issuecomment-704080266 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4309/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (CARBONDATA-3834) Segment directory and segment file in metadata are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to false.
[
https://issues.apache.org/jira/browse/CARBONDATA-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venugopal Reddy K resolved CARBONDATA-3834.
---
Fix Version/s: 2.0.0
Resolution: Fixed
> Segment directory and segment file in metadata are not created for
> partitioned table when 'carbon.merge.index.in.segment' property is set to
> false.
> ---
>
> Key: CARBONDATA-3834
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3834
> Project: CarbonData
> Issue Type: Bug
> Components: hadoop-integration, spark-integration
>Affects Versions: 2.0.0
>Reporter: Venugopal Reddy K
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 8h
> Remaining Estimate: 0h
>
> *[Issue]*
> With the latest version of Carbon, Segment directory and segment file in
> metadata directory are not created for partitioned table when
> 'carbon.merge.index.in.segment' property is set to 'false'. And actual index
> files which were present in respective partition's '.tmp' directory are also
> deleted without moving them out to respective partition directory where its
> '.carbondata' file exist. Thus all the queries fails due to this problem.
> {color:#FF}Exception in thread "main"
> java.lang.NullPointerExceptionException in thread "main"
> java.lang.NullPointerException at
> org.apache.carbondata.core.readcommitter.TableStatusReadCommittedScope.getCommittedIndexFile(TableStatusReadCommittedScope.java:90)
> at
> org.apache.carbondata.core.index.Segment.getCommittedIndexFile(Segment.java:183)
> at
> org.apache.carbondata.core.util.BlockletIndexUtil.getTableBlockUniqueIdentifiers(BlockletIndexUtil.java:204)
> at {color}
>
> This issue was introduced from the resolution of an older optimization issue
> -CARBONDATA-3641-+[Should improve data loading performance for partition
> table]+
> i.e., with [https://github.com/apache/carbondata/pull/3535]
>
> *[Modification Suggestion]*
>
> If 'carbon.merge.index.in.segment' property is false, we can create the
> segment directory and segment file, and move the index file from respective
> partition's temp directory to partition directory where the .carbondata file
> exists.
> Note: This need to be done before the respective partition's .tmp directory
> is deleted. Otherwise, we loose the index files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3962: [CARBONDATA-4017]Fix the insert issue when the column name contains '\' and fix SI creation issue
CarbonDataQA1 commented on pull request #3962: URL: https://github.com/apache/carbondata/pull/3962#issuecomment-704070143 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2558/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3962: [CARBONDATA-4017]Fix the insert issue when the column name contains '\' and fix SI creation issue
CarbonDataQA1 commented on pull request #3962: URL: https://github.com/apache/carbondata/pull/3962#issuecomment-704068740 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4307/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 closed pull request #3956: [CARBONDATA-4010] Added documentation for set/unset of longStringColumns.
Karan980 closed pull request #3956: URL: https://github.com/apache/carbondata/pull/3956 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
CarbonDataQA1 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-704060595 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4310/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
CarbonDataQA1 commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-704059997 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2561/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #3914: [CARBONDATA-3979] Added Hive local dictionary support example
ShreelekhyaG commented on pull request #3914: URL: https://github.com/apache/carbondata/pull/3914#issuecomment-704058348 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on pull request #3963: [CARBONDATA-4018]Fix CSV header validation not contains dimension columns
Indhumathi27 commented on pull request #3963: URL: https://github.com/apache/carbondata/pull/3963#issuecomment-704058097 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
