[GitHub] [carbondata] CarbonDataQA commented on issue #3404: [CARBONDATA-3540] Delete all external segments when dropping table
CarbonDataQA commented on issue #3404: [CARBONDATA-3540] Delete all external segments when dropping table URL: https://github.com/apache/carbondata/pull/3404#issuecomment-540383825 Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/631/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA commented on issue #3404: [CARBONDATA-3540] Delete all external segments when dropping table
CarbonDataQA commented on issue #3404: [CARBONDATA-3540] Delete all external segments when dropping table URL: https://github.com/apache/carbondata/pull/3404#issuecomment-540351540 Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/631/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA commented on issue #3404: [CARBONDATA-3540] Delete all external segments when dropping table
CarbonDataQA commented on issue #3404: [CARBONDATA-3540] Delete all external segments when dropping table URL: https://github.com/apache/carbondata/pull/3404#issuecomment-540343575 Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/626/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (CARBONDATA-3521) optimize read property file code
[ https://issues.apache.org/jira/browse/CARBONDATA-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3521: - Fix Version/s: (was: 1.6.1) 2.0.0 > optimize read property file code > > > Key: CARBONDATA-3521 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3521 > Project: CarbonData > Issue Type: Improvement > Components: core >Affects Versions: 1.6.0 >Reporter: lamber-ken >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Some points > 1) change System.getProperty(key) to System.getProperty(key, default) > 2) optimize duplicated code, like CARBON_PROPERTIES_FILE_PATH_DEFAULT > 3) change default carbon properties file path -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3515) Limit local dictionary size to 10% of allowed blocklet size
[ https://issues.apache.org/jira/browse/CARBONDATA-3515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3515: - Fix Version/s: 1.6.1 > Limit local dictionary size to 10% of allowed blocklet size > --- > > Key: CARBONDATA-3515 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3515 > Project: CarbonData > Issue Type: Improvement >Reporter: Ajantha Bhat >Assignee: Kumar Vishal >Priority: Major > Fix For: 1.6.1 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > problem: currently local dictionary max size is 2GB, because of this for > varchar columns or long string columns, local dictionary can be of 2GB size. > so, as local dictionary is stored in blocklet. blocklet size will exceed 2 > GB, even though configured maximum blocklet size is 64MB. some places inter > overflow happens during casting. > > solution: limit the local dictionary size to 10% of maximum allowed blocklet > size -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3370) fix missing version of maven-duplicate-finder-plugin
[ https://issues.apache.org/jira/browse/CARBONDATA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3370: - Fix Version/s: (was: 1.6.1) 2.0.0 > fix missing version of maven-duplicate-finder-plugin > > > Key: CARBONDATA-3370 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3370 > Project: CarbonData > Issue Type: Improvement > Components: build >Affects Versions: 1.5.3 >Reporter: lamber-ken >Priority: Critical > Fix For: 2.0.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > fix missing version of maven-duplicate-finder-plugin in pom file -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3454) Optimize the performance of select coun(*) for index server
[ https://issues.apache.org/jira/browse/CARBONDATA-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3454: - Fix Version/s: 1.6.1 > Optimize the performance of select coun(*) for index server > --- > > Key: CARBONDATA-3454 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3454 > Project: CarbonData > Issue Type: Sub-task >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Minor > Fix For: 1.6.1 > > Time Spent: 13h 10m > Remaining Estimate: 0h > > Currently all the extended blocklets are being returned to the main driver in > case of count *.* But all this information is not required for count(*) case, > therefore the optimal thing would be to send only the requried info. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3491) Return updated/deleted rows count when execute update/delete sql
[ https://issues.apache.org/jira/browse/CARBONDATA-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3491: - Fix Version/s: 1.6.1 > Return updated/deleted rows count when execute update/delete sql > > > Key: CARBONDATA-3491 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3491 > Project: CarbonData > Issue Type: Improvement >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Minor > Fix For: 1.6.1 > > Time Spent: 10h 10m > Remaining Estimate: 0h > > Return updated/deleted rows count when execute update/delete sql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3511) Query time improvement by reducing the number of NameNode calls while having carbonindex files in the store
[ https://issues.apache.org/jira/browse/CARBONDATA-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3511: - Fix Version/s: 1.6.1 > Query time improvement by reducing the number of NameNode calls while having > carbonindex files in the store > --- > > Key: CARBONDATA-3511 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3511 > Project: CarbonData > Issue Type: Improvement >Reporter: dhatchayani >Priority: Major > Fix For: 1.6.1 > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3483) Can not run horizontal compaction when execute update sql
[ https://issues.apache.org/jira/browse/CARBONDATA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3483: - Fix Version/s: (was: 1.6.0) 1.6.1 > Can not run horizontal compaction when execute update sql > - > > Key: CARBONDATA-3483 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3483 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.5.3, 1.6.0, 1.5.4 >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Major > Fix For: 1.6.1 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > After PR#3166, horizontal compaction will not actually run when execute > update sql. > When it runs update sql and will run horizontal compaction if needs, it will > require update.lock and compaction.lock when execute > CarbonAlterTableCompactionCommand.alterTableForCompaction, but these two > locks already are locked when it starts to execute update sql. so it will > require locks failed and can't execute compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3477) Throw out exception when use sql: 'update table select\n...'
[
https://issues.apache.org/jira/browse/CARBONDATA-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajantha Bhat updated CARBONDATA-3477:
-
Fix Version/s: (was: 1.6.0)
1.6.1
> Throw out exception when use sql: 'update table select\n...'
>
>
> Key: CARBONDATA-3477
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3477
> Project: CarbonData
> Issue Type: Bug
>Reporter: Zhichao Zhang
>Assignee: Zhichao Zhang
>Priority: Minor
> Fix For: 1.6.1
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> When use below sql to update table:
> {code:java}
> UPDATE IUD_table2 a
> SET (a.IUD_table2_country, a.IUD_table2_salary) = (select
> b.IUD_table1_country, b.IUD_table1_salary from IUD_table1 b where
> b.IUD_table1_id = 8)
> WHERE a.IUD_table2_id < 6 or a.IUD_table2_id > 15{code}
> *It will throw out exception:*
> {code:java}
> Exception in thread "main" org.apache.spark.sql.AnalysisException: == Parse1
> ==
> mismatched input '.' expecting (line 2, pos 1)
> == SQL ==
> select select
> b.IUD_table1_country, b.IUD_table1_salary from IUD_table1 b where
> b.IUD_table1_id = 8 from iud_table2 a
> -^^^
> == Parse2 ==
> [1.1] failure: identifier matching regex (?i)ALTER expected
> select select
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3485) data loading is failed from S3 to hdfs table having ~2K carbonfiles
[ https://issues.apache.org/jira/browse/CARBONDATA-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3485: - Fix Version/s: 1.6.1 > data loading is failed from S3 to hdfs table having ~2K carbonfiles > - > > Key: CARBONDATA-3485 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3485 > Project: CarbonData > Issue Type: Bug >Reporter: Babulal >Priority: Major > Fix For: 1.6.1 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > data loading is failed from S3 to hdfs table having ~2K carbonfiles > org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException: > > at > org.apache.carbondata.processing.loading.sort.AbstractMergeSorter.checkError(AbstractMergeSorter.java:39) > at > org.apache.carbondata.processing.loading.sort.impl.UnsafeParallelReadMergeSorterImpl.sort(UnsafeParallelReadMergeSorterImpl.java:107) > at > org.apache.carbondata.processing.loading.steps.SortProcessorStepImpl.execute(SortProcessorStepImpl.java:55) > at > org.apache.carbondata.processing.loading.steps.DataWriterProcessorStepImpl.execute(DataWriterProcessorStepImpl.java:109) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3506) Alter table add, drop, rename and datatype change fails with hive compatile property
[ https://issues.apache.org/jira/browse/CARBONDATA-3506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3506: - Fix Version/s: 1.6.1 > Alter table add, drop, rename and datatype change fails with hive compatile > property > > > Key: CARBONDATA-3506 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3506 > Project: CarbonData > Issue Type: Bug >Reporter: Akash R Nilugal >Assignee: Akash R Nilugal >Priority: Major > Fix For: 1.6.1 > > Time Spent: 6h > Remaining Estimate: 0h > > 1. Alter table add, drop, rename and datatype change fails on partition table > with hive compatile property > when hive.metastore.disallow.incompatible.col.type.changes is set true, add > column or any alter fails on parition table in spark 2.2 and above > 2. when table has only two columns with one as partition column, if we allow > dropping of non partition column, then it will like table with all the > columns as partition column , which is invalid, and fails with above property > as true. so block this operation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA commented on issue #3406: [CARBONDATA-3516] Fixed compilation issue for mixed formats in Spark-2.1
CarbonDataQA commented on issue #3406: [CARBONDATA-3516] Fixed compilation issue for mixed formats in Spark-2.1 URL: https://github.com/apache/carbondata/pull/3406#issuecomment-540001207 Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/630/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (CARBONDATA-3490) Concurrent data load failure with carbondata FileNotFound exception
[ https://issues.apache.org/jira/browse/CARBONDATA-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3490: - Fix Version/s: (was: 1.6.0) 1.6.1 > Concurrent data load failure with carbondata FileNotFound exception > --- > > Key: CARBONDATA-3490 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3490 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Priority: Major > Fix For: 1.6.1 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Caused by: > org.apache.carbondata.core.datastore.exception.CarbonDataWriterException: > Problem while copying file from local store to carbon store > at > org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2750) > at > org.apache.carbondata.processing.store.writer.AbstractFactDataWriter.commitCurrentFile(AbstractFactDataWriter.java:283) > at > org.apache.carbondata.processing.store.writer.v3.CarbonFactDataWriterImplV3.closeWriter(CarbonFactDataWriterImplV3.java:393) > ... 11 more > Caused by: java.io.FileNotFoundException: > /tmp/carbon865982118689228_1/Fact/Part0/Segment_6/1/part-0-1_batchno0-0-6-1565329654844.carbondata > (No such file or directory) > at java.io.FileInputStream.open0(Native Method) > at java.io.FileInputStream.open(FileInputStream.java:195) > at java.io.FileInputStream.(FileInputStream.java:138) > at java.io.FileInputStream.(FileInputStream.java:93) > at > org.apache.carbondata.core.datastore.filesystem.LocalCarbonFile.getDataInputStream(LocalCarbonFile.java:309) > at > org.apache.carbondata.core.datastore.filesystem.LocalCarbonFile.getDataInputStream(LocalCarbonFile.java:299) > at > org.apache.carbondata.core.datastore.impl.FileFactory.getDataInputStream(FileFactory.java:179) > at > org.apache.carbondata.core.datastore.impl.FileFactory.getDataInputStream(FileFactory.java:175) > at > org.apache.carbondata.core.util.CarbonUtil.copyLocalFileToCarbonStore(CarbonUtil.java:2781) > at > org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2746) > ... 13 more > problem: When two load is happening concurrently, one load is cleaning the > temp directory of the concurrent load > cause: temp directory to store the carbon files is created using system.get > nano time, due to this two load have same store location. when one load is > completed, it cleaned the temp directory. causing dataload failure for other > load. > solution: > use UUID instead of nano time while creating the temp directory to have each > load a unique directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3482) Null pointer exception when concurrent select queries are executed from different beeline terminals.
[ https://issues.apache.org/jira/browse/CARBONDATA-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3482: - Fix Version/s: (was: 1.6.0) 1.6.1 > Null pointer exception when concurrent select queries are executed from > different beeline terminals. > > > Key: CARBONDATA-3482 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3482 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.6.1 > > Time Spent: 5.5h > Remaining Estimate: 0h > > # Beeline1: => create tables (1K ) > 2. Beeline2 => insert into table t2 (only 1 records ) till 7K > 3. Concurrent queries > q1 : select count(*) from t1 > q2 : select * from t1 limit 1 > q3 : select count(*) from t2 > q2 : select * from t2 limit 1 > > Exception: > java.lang.NullPointerException > at > org.apache.carbondata.core.indexstore.blockletindex.BlockDataMap.getFileFooterEntrySchema(BlockDataMap.java:1061) > at > org.apache.carbondata.core.indexstore.blockletindex.BlockDataMap.prune(BlockDataMap.java:727) > at > org.apache.carbondata.core.indexstore.blockletindex.BlockDataMap.prune(BlockDataMap.java:821) > at > org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapFactory.getAllBlocklets(BlockletDataMapFactory.java:446) > at > org.apache.carbondata.core.datamap.TableDataMap.pruneWithoutFilter(TableDataMap.java:156) > at > org.apache.carbondata.core.datamap.TableDataMap.prune(TableDataMap.java:143) > at > org.apache.carbondata.hadoop.api.CarbonInputFormat.getPrunedBlocklets(CarbonInputFormat.java:563) > at > org.apache.carbondata.hadoop.api.CarbonInputFormat.getDataBlocksOfSegment(CarbonInputFormat.java:471) > at > org.apache.carbondata.hadoop.api.CarbonTableInputFormat.getSplits(CarbonTableInputFormat.java:471) > at > org.apache.carbondata.hadoop.api.CarbonTableInputFormat.getSplits(CarbonTableInputFormat.java:199) > at > org.apache.carbondata.spark.rdd.CarbonScanRDD.internalGetPartitions(CarbonScanRDD.scala:141) > at > org.apache.carbondata.spark.rdd.CarbonRDD.getPartitions(CarbonRDD.scala:66) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:256) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:254) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3488) Check the file size after move local file to carbon path
[ https://issues.apache.org/jira/browse/CARBONDATA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3488: - Fix Version/s: (was: 1.6.0) 1.6.1 > Check the file size after move local file to carbon path > > > Key: CARBONDATA-3488 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3488 > Project: CarbonData > Issue Type: Improvement >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Minor > Fix For: 1.6.1 > > Time Spent: 7h > Remaining Estimate: 0h > > *Problem:* > One user met an issue: the row num saved in carbonindex file is non zero but > the file size of relevant carbondata file is 0. > > *Solution:* > In CarbonUtil.copyCarbonDataFileToCarbonStorePath, check the file size of > carbon file whether is the same as the size fo local file after move local > file to carbon path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3476) Read time and scan time stats shown wrong in executor log for filter query
[ https://issues.apache.org/jira/browse/CARBONDATA-3476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3476: - Fix Version/s: (was: 1.6.0) 1.6.1 > Read time and scan time stats shown wrong in executor log for filter query > -- > > Key: CARBONDATA-3476 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3476 > Project: CarbonData > Issue Type: Bug > Components: core >Reporter: Vikram Ahuja >Priority: Minor > Fix For: 1.6.1 > > Time Spent: 8h 20m > Remaining Estimate: 0h > > Problem: Read time and scan time stats shown wrong in executor log for filter > query > Root cause: Projection read time is added in scan time because of this scan > time and read time is not correct in stats > Solution: Added projection read time for both measure and dimension column in > read stats -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3487) wrong Input metrics (size/record) displayed in spark UI during insert into
[ https://issues.apache.org/jira/browse/CARBONDATA-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3487: - Fix Version/s: 1.6.1 > wrong Input metrics (size/record) displayed in spark UI during insert into > -- > > Key: CARBONDATA-3487 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3487 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Priority: Minor > Fix For: 1.6.1 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > create a carbon table > insert huge data (2 Billion row) to carbon table. > observe the metrics in spark UI. Both the size and record count in input > metrics is wrong during insert into scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3493) Carbon query fails when enable.query.statistics is true in specific scenario.
[ https://issues.apache.org/jira/browse/CARBONDATA-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3493: - Fix Version/s: (was: 1.6.0) 1.6.1 > Carbon query fails when enable.query.statistics is true in specific scenario. > - > > Key: CARBONDATA-3493 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3493 > Project: CarbonData > Issue Type: Bug >Reporter: Shivam Goyal >Priority: Minor > Fix For: 1.6.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3486) Serialization/ deserialization issue with Datatype
[ https://issues.apache.org/jira/browse/CARBONDATA-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3486: - Fix Version/s: (was: 1.6.0) 1.6.1 > Serialization/ deserialization issue with Datatype > -- > > Key: CARBONDATA-3486 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3486 > Project: CarbonData > Issue Type: Bug >Reporter: MANISH NALLA >Priority: Minor > Fix For: 1.6.1 > > Time Spent: 2.5h > Remaining Estimate: 0h > > When we use old store and do alter add sort columns on it then query on the > old segment, serialization/de-serialization issue comes for Filter Column of > Measure type which has been changed in Sort Column as it is being > de-serialized by ObjectSerialization. This fails the check and the query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3481) Multi-thread pruning fails when datamaps count is just near numOfThreadsForPruning
[ https://issues.apache.org/jira/browse/CARBONDATA-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3481: - Fix Version/s: (was: 1.6.0) 1.6.1 > Multi-thread pruning fails when datamaps count is just near > numOfThreadsForPruning > -- > > Key: CARBONDATA-3481 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3481 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Priority: Minor > Fix For: 1.6.1 > > Time Spent: 8h 10m > Remaining Estimate: 0h > > Problem : Multi-thread pruning fails when datamaps count is just near > numOfThreadsForPruning. > Cause : When the datamaps count is just near numOfThreadsForPruning, > As code is checking '>= ', last thread may not get the datamaps for prune. > Hence array out of index exception is thrown in this scenario. > There is no issues with higher number of datamaps. > solution: In this scenario launch threads based on the distribution value, > not on the hardcoded value > > steps to reproduce: > 5 index files with total 0.32 million data file (each has 50k, 120k, 50k, > 60k, 50k each) > with default thread count as 4, array of bound index is observed for below > line. > final List segmentDataMapGroups = > datamapListForEachThread.get(i); -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3478) Fix ArrayIndexOutOfBoundsException issue on compaction after alter rename operation
[
https://issues.apache.org/jira/browse/CARBONDATA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajantha Bhat updated CARBONDATA-3478:
-
Fix Version/s: (was: 1.6.0)
1.6.1
> Fix ArrayIndexOutOfBoundsException issue on compaction after alter rename
> operation
> ---
>
> Key: CARBONDATA-3478
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3478
> Project: CarbonData
> Issue Type: Bug
>Reporter: Indhumathi Muthumurugesh
>Priority: Major
> Fix For: 1.6.1
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Please find the steps to reproduce the issue:
> # Create table having dimension and measure columns
> # load data
> # rename / add column/ drop column in table
> # alter set tblproperties('sort_columns'=measure column',
> 'sort_scope'='local_sort')
> # load data
> # perform compaction and find the exception below
>
> Driver stacktrace:
> 2019-07-26 19:34:03 ERROR CarbonAlterTableCompactionCommand:345 - Exception
> in start compaction thread.
> java.lang.Exception: Exception in compaction Job aborted due to stage
> failure: Task 0 in stage 6.0 failed 1 times, most recent failure: Lost task
> 0.0 in stage 6.0 (TID 6, localhost, executor driver):
> java.lang.ArrayIndexOutOfBoundsException: 3
> at
> org.apache.carbondata.core.scan.wrappers.ByteArrayWrapper.getNoDictionaryKeyByIndex(ByteArrayWrapper.java:81)
> | |
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3462) Add usage and deployment document for index server
[ https://issues.apache.org/jira/browse/CARBONDATA-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3462: - Fix Version/s: (was: 1.6.0) 1.6.1 > Add usage and deployment document for index server > -- > > Key: CARBONDATA-3462 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3462 > Project: CarbonData > Issue Type: Sub-task >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.6.1 > > Time Spent: 11h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3474) Fix validate mvQuery having filter expression and correct error message
[ https://issues.apache.org/jira/browse/CARBONDATA-3474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3474: - Fix Version/s: (was: 1.6.0) 1.6.1 > Fix validate mvQuery having filter expression and correct error message > --- > > Key: CARBONDATA-3474 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3474 > Project: CarbonData > Issue Type: Bug >Reporter: Indhumathi Muthumurugesh >Priority: Minor > Fix For: 1.6.1 > > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3452) select query failure when substring on dictionary column with join
[ https://issues.apache.org/jira/browse/CARBONDATA-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajantha Bhat updated CARBONDATA-3452: - Fix Version/s: 1.6.1 > select query failure when substring on dictionary column with join > -- > > Key: CARBONDATA-3452 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3452 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Priority: Major > Fix For: 1.6.1 > > Time Spent: 13h 20m > Remaining Estimate: 0h > > h1. select query failure when substring on dictionary column with join > "select a.ch from (select substring(s1,1,2) as ch from t1) a join t2 h on > (a.ch = h.t2)" > problem: select query failure when substring on dictionary column with join. > cause: when dictionary include is present, data type is updated to int from > string in plan attribute. so substring was unresolved on int column. Join > operation try to reference this attribute which is unresolved. > solution: skip updating datatype if dictionary is included in the plan > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA commented on issue #3406: [CARBONDATA-3516] Fixed compilation issue for mixed formats in Spark-2.1
CarbonDataQA commented on issue #3406: [CARBONDATA-3516] Fixed compilation issue for mixed formats in Spark-2.1 URL: https://github.com/apache/carbondata/pull/3406#issuecomment-539973937 Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/625/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA commented on issue #3406: [CARBONDATA-3516] Fixed compilation issue for mixed formats in Spark-2.1
CarbonDataQA commented on issue #3406: [CARBONDATA-3516] Fixed compilation issue for mixed formats in Spark-2.1 URL: https://github.com/apache/carbondata/pull/3406#issuecomment-539955705 Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/624/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA commented on issue #3379: [WIP] delete data from a segment if the data is exists in other segments
CarbonDataQA commented on issue #3379: [WIP] delete data from a segment if the data is exists in other segments URL: https://github.com/apache/carbondata/pull/3379#issuecomment-539932469 Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/628/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA commented on issue #3379: [WIP] delete data from a segment if the data is exists in other segments
CarbonDataQA commented on issue #3379: [WIP] delete data from a segment if the data is exists in other segments URL: https://github.com/apache/carbondata/pull/3379#issuecomment-539923562 Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/628/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA commented on issue #3379: [WIP] delete data from a segment if the data is exists in other segments
CarbonDataQA commented on issue #3379: [WIP] delete data from a segment if the data is exists in other segments URL: https://github.com/apache/carbondata/pull/3379#issuecomment-539895843 Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/623/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA commented on issue #3375: [CARBONDATA-3512]Index Server Enhancement
CarbonDataQA commented on issue #3375: [CARBONDATA-3512]Index Server Enhancement URL: https://github.com/apache/carbondata/pull/3375#issuecomment-539848789 Build Success with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/627/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
