[GitHub] carbondata issue #2785: [CARBONDATA-2986] Table Properties are lost when mul...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2785 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/694/ ---
[GitHub] carbondata issue #2785: [CARBONDATA-2986] Table Properties are lost when mul...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2785 retest this please ---
[GitHub] carbondata issue #2772: [CARBONDATA-2977] Write uncompress_size to ChunkComp...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2772 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/693/ ---
[GitHub] carbondata issue #2793: [CARBONDATA-2990] Fixed JVM crash when rebuilding bl...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2793 @ravipesala Why does it only occur in cluster mode? Just in curious ---
[GitHub] carbondata issue #2772: [CARBONDATA-2977] Write uncompress_size to ChunkComp...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2772 retest this please ---
[GitHub] carbondata pull request #2776: [HOTFIX] Avoid adding empty table property
Github user jackylk closed the pull request at: https://github.com/apache/carbondata/pull/2776 ---
[GitHub] carbondata issue #2768: [CARBONDATA-2594] Do not add InvertedIndex in Encodi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2768 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/692/ ---
[GitHub] carbondata pull request #2791: [HOTFIX]correct the exception handling in loo...
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2791#discussion_r222537606
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
---
@@ -23,6 +23,7 @@ import java.net.URI
import scala.collection.mutable.ArrayBuffer
import org.apache.hadoop.fs.permission.{FsAction, FsPermission}
+import org.apache.hadoop.hive.ql.metadata.HiveException
--- End diff --
scalastyle fails if i give the space, i have checked it
---
[GitHub] carbondata pull request #2791: [HOTFIX]correct the exception handling in loo...
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2791#discussion_r222537366
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
---
@@ -208,7 +209,10 @@ class CarbonFileMetastore extends CarbonMetaStore {
try {
lookupRelation(tableIdentifier)(sparkSession)
} catch {
- case _: Exception =>
+ case ex: Exception =>
+if (ex.getCause.isInstanceOf[HiveException]) {
+ throw ex
+}
--- End diff --
here we arereturning false directly after catching exception, telling the
table does not exists, this is wrong, i have check the flow , we might get
hiveException(regarding permission) also if the user is not allowed to access
the table, that time also we return false and we will not get the proper error.
---
[GitHub] carbondata issue #2791: [HOTFIX]correct the exception handling in lookup rel...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2791 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/888/ ---
[GitHub] carbondata issue #2791: [HOTFIX]correct the exception handling in lookup rel...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2791 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8956/ ---
[GitHub] carbondata issue #2796: [CARBONDATA-2991]NegativeArraySizeException during q...
Github user sujith71955 commented on the issue: https://github.com/apache/carbondata/pull/2796 @BJangir @jackylk @chenliang613 @ravipesala I think UUID.randomUUID().toString() has some performance bottle necks even though its negligible, as it internally uses SecureRandom API which is a synchronized method. if possible explore more on this as our system is more performance driven. What about using JUG? There is also a time bound uuid generation which seems to work lot faster compare to normal UUID generation supported by java. Following link will give you a fair idea regarding implementation and performance comparison. http://www.dcalabresi.com/blog/java/generate-java-uuid-performance/ ---
[GitHub] carbondata issue #2796: [CARBONDATA-2991]NegativeArraySizeException during q...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2796 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/887/ ---
[GitHub] carbondata issue #2796: [CARBONDATA-2991]NegativeArraySizeException during q...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2796 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8955/ ---
[GitHub] carbondata pull request #2791: [HOTFIX]correct the exception handling in loo...
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2791#discussion_r222416107
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
---
@@ -23,6 +23,7 @@ import java.net.URI
import scala.collection.mutable.ArrayBuffer
import org.apache.hadoop.fs.permission.{FsAction, FsPermission}
+import org.apache.hadoop.hive.ql.metadata.HiveException
--- End diff --
nit: space is required, please check the rule
---
[GitHub] carbondata pull request #2791: [HOTFIX]correct the exception handling in loo...
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2791#discussion_r222415572
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
---
@@ -208,7 +209,10 @@ class CarbonFileMetastore extends CarbonMetaStore {
try {
lookupRelation(tableIdentifier)(sparkSession)
} catch {
- case _: Exception =>
+ case ex: Exception =>
+if (ex.getCause.isInstanceOf[HiveException]) {
+ throw ex
+}
--- End diff --
Didn't got your context of explanation is for which review comment?
---
[GitHub] carbondata issue #2791: [HOTFIX]correct the exception handling in lookup rel...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2791 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/691/ ---
[GitHub] carbondata pull request #2791: [HOTFIX]correct the exception handling in loo...
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2791#discussion_r222409727
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
---
@@ -208,7 +209,10 @@ class CarbonFileMetastore extends CarbonMetaStore {
try {
lookupRelation(tableIdentifier)(sparkSession)
} catch {
- case _: Exception =>
+ case ex: Exception =>
+if (ex.getCause.isInstanceOf[HiveException]) {
+ throw ex
+}
--- End diff --
i checked the flow and permission errors will be thrown by Hive as
HiveException and wrapped as analysis exception, and all the other exceptions
will be same as i have described in description, during that time we cannot
just catch exception and return false as table not exists. so i m checking as
HiveException
---
[GitHub] carbondata issue #2796: [CARBONDATA-2991]NegativeArraySizeException during q...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2796 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/690/ ---
[GitHub] carbondata pull request #2796: [CARBONDATA-2991]NegativeArraySizeException d...
GitHub user BJangir opened a pull request: https://github.com/apache/carbondata/pull/2796 [CARBONDATA-2991]NegativeArraySizeException during query execution Issue :- During Query Execution sometime NegativeArraySizeException Exception in Some Tasks . And sometime Executor is lost (JVM crash) Root Cause :- It is because existing memoryblock is removed while it was in-use. This happened because duplicate taskid generated. Sometime freed same memory addresses are assigned to another task which will initialize memory block to0 and this cause NegativeSizeArrayException whereas sometime freed memory will not be used any task of executor process but running task will try to access it and as that address is not part of process so JVM crash will happen. Solution :- Change taskID generation to UUID based instead of System.nanoTime() Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. Manual Testing Done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/BJangir/incubator-carbondata CARBONDATA-2991 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2796.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2796 commit b41b35f9fdd86e7080e2d03d3219e260bcad7d31 Author: BJangir Date: 2018-10-03T17:35:42Z [CARBONDATA-2991]NegativeArraySizeException during query execution ---
[GitHub] carbondata issue #2729: [WIP]Store optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2729 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/886/ ---
[GitHub] carbondata issue #2729: [WIP]Store optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2729 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8954/ ---
[GitHub] carbondata issue #2729: [WIP]Store optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2729 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/689/ ---
[GitHub] carbondata issue #2729: [WIP]Store optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2729 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8953/ ---
[GitHub] carbondata issue #2729: [WIP]Store optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2729 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/885/ ---
[GitHub] carbondata issue #2729: [WIP]Store optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2729 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/688/ ---
[GitHub] carbondata issue #2729: [WIP]Store optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2729 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8952/ ---
[GitHub] carbondata issue #2729: [WIP]Store optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2729 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/884/ ---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/883/ ---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8951/ ---
[GitHub] carbondata issue #2787: [HOTFIX] Fixed Between Query Data Mismatch issue for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2787 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/881/ ---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/687/ ---
[GitHub] carbondata issue #2787: [HOTFIX] Fixed Between Query Data Mismatch issue for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2787 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8949/ ---
[GitHub] carbondata pull request #2795: [HOTFIX] Changes to align printing of informa...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2795 ---
[GitHub] carbondata issue #2795: [HOTFIX] Changes to align printing of information in...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2795 LGTM ---
[jira] [Resolved] (CARBONDATA-2976) Support dumping column chunk meta in CarbonCli
[ https://issues.apache.org/jira/browse/CARBONDATA-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2976. - Resolution: Fixed Fix Version/s: 1.5.0 > Support dumping column chunk meta in CarbonCli > -- > > Key: CARBONDATA-2976 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2976 > Project: CarbonData > Issue Type: New Feature >Reporter: Jacky Li >Assignee: Jacky Li >Priority: Major > Fix For: 1.5.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2771: [CARBONDATA-2976] Support dumping column chun...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2771 ---
[jira] [Resolved] (CARBONDATA-2984) streaming throw NPE when there is no data in the task of a batch
[ https://issues.apache.org/jira/browse/CARBONDATA-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2984. - Resolution: Fixed Fix Version/s: 1.5.0 > streaming throw NPE when there is no data in the task of a batch > > > Key: CARBONDATA-2984 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2984 > Project: CarbonData > Issue Type: Bug >Reporter: QiangCai >Priority: Major > Fix For: 1.5.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > !746438440.jpg! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2782: [CARBONDATA-2984][Streaming] Fix NPE when the...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2782 ---
[GitHub] carbondata issue #2788: [Documentation] Readme updated with latest topics an...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2788 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8948/ ---
[GitHub] carbondata issue #2782: [CARBONDATA-2984][Streaming] Fix NPE when there is n...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2782 LGTM ---
[jira] [Resolved] (CARBONDATA-2987) Data mismatch after compaction with measure sort columns
[ https://issues.apache.org/jira/browse/CARBONDATA-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2987. - Resolution: Fixed > Data mismatch after compaction with measure sort columns > > > Key: CARBONDATA-2987 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2987 > Project: CarbonData > Issue Type: Improvement >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Critical > Time Spent: 2h 40m > Remaining Estimate: 0h > > problem: Data mismatch after compaction with measure sort columns > root cause : In compaction flow (DictionaryBasedResultCollector), in > ColumnPageWrapper inverted index mapping is not handled. Because of this, row > of no dictionary dimension columns gets data form other rows. > Hence the data mismatch > > solution: Handle inverted index mapping for DictionaryBasedResultCollector > flow in ColumnPageWrapper > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2784: [CARBONDATA-2987] Data mismatch after compact...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2784 ---
[GitHub] carbondata pull request #2790: [HOTFIX] carbon reader support open another r...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2790 ---
[GitHub] carbondata issue #2790: [HOTFIX] carbon reader support open another reader w...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2790 LGTM ---
[jira] [Resolved] (CARBONDATA-2967) Select is failing on pre-aggregate datamap when thrift server is restarted.
[ https://issues.apache.org/jira/browse/CARBONDATA-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2967. - Resolution: Fixed Fix Version/s: 1.5.0 > Select is failing on pre-aggregate datamap when thrift server is restarted. > --- > > Key: CARBONDATA-2967 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2967 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Major > Fix For: 1.5.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > *Problem:* > NullPointerException is thrown when select query is fired on a datamap. This > is because to access dictionary files of the parent table the child table > tries to get tablePath from CarbonTable object of the parent. Because the > metadata is not populated therefore NullpointerException is thrown. > > 1, 10.2.3.19, executor 1): java.lang.RuntimeException: Error while resolving > filter expression > at > org.apache.carbondata.core.metadata.schema.table.CarbonTable.resolveFilter(CarbonTable.java:1043) > > at > org.apache.carbondata.core.scan.model.QueryModelBuilder.build(QueryModelBuilder.java:322) > > at > org.apache.carbondata.hadoop.api.CarbonInputFormat.createQueryModel(CarbonInputFormat.java:632) > > at > org.apache.carbondata.spark.rdd.CarbonScanRDD.internalCompute(CarbonScanRDD.scala:419) > > at > org.apache.carbondata.spark.rdd.CarbonRDD.compute(CarbonRDD.scala:78) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) > at org.apache.spark.scheduler.Task.run(Task.scala:109) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.NullPointerException > at > org.apache.carbondata.core.scan.executor.util.QueryUtil.getTableIdentifierForColumn(QueryUtil.java:401) > > at > org.apache.carbondata.core.scan.filter.FilterUtil.getForwardDictionaryCache(FilterUtil.java:1416) > > at > org.apache.carbondata.core.scan.filter.FilterUtil.getFilterValues(FilterUtil.java:712) > > at > org.apache.carbondata.core.scan.filter.resolver.resolverinfo.visitor.DictionaryColumnVisitor.populateFilterResolvedInfo(DictionaryColumnVisitor.java:60) > > at > org.apache.carbondata.core.scan.filter.resolver.resolverinfo.DimColumnResolvedFilterInfo.populateFilterInfoBasedOnColumnType(DimColumnResolvedFilterInfo.java:119) > > at > org.apache.carbondata.core.scan.filter.resolver.ConditionalFilterResolverImpl.resolve(ConditionalFilterResolverImpl.java:107) > > at -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2786: [CARBONDATA-2967] Fixed NPE for preaggregate ...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2786 ---
[GitHub] carbondata issue #2788: [Documentation] Readme updated with latest topics an...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2788 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/880/ ---
[GitHub] carbondata issue #2786: [CARBONDATA-2967] Fixed NPE for preaggregate queries
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2786 LGTM ---
[jira] [Resolved] (CARBONDATA-2979) select count fails when carbondata file is written through SDK and read through sparkfileformat for complex datatype map(struct->array->map)
[ https://issues.apache.org/jira/browse/CARBONDATA-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2979. - Resolution: Fixed Fix Version/s: 1.5.0 > select count fails when carbondata file is written through SDK and read > through sparkfileformat for complex datatype map(struct->array->map) > > > Key: CARBONDATA-2979 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2979 > Project: CarbonData > Issue Type: Bug > Components: file-format >Affects Versions: 1.5.0 >Reporter: Rahul Singha >Assignee: Manish Gupta >Priority: Minor > Fix For: 1.5.0 > > Attachments: MapSchema_15_int.avsc > > Time Spent: 1h 50m > Remaining Estimate: 0h > > *Steps:* > create carabondata and carbonindex file using SDK > place the files in a hdfs location > Read files using spark file format > create table schema15_int using carbon location > 'hdfs://hacluster/user/rahul/map/mapschema15_int'; > Select count(*) from schema15_int; > *Actual Result:* > Error: org.apache.spark.SparkException: Job aborted due to stage failure: > Task 0 in stage 24.0 failed 4 times, most recent failure: Lost task 0.3 in > stage 24.0 (TID 34, BLR114238, executor 3): java.io.IOException: All the > files doesn't have same schema. Unsupported operation on nonTransactional > table. Check logs. > at > org.apache.carbondata.core.scan.executor.impl.AbstractQueryExecutor.updateColumns(AbstractQueryExecutor.java:276) > at > org.apache.carbondata.core.scan.executor.impl.AbstractQueryExecutor.getDataBlocks(AbstractQueryExecutor.java:234) > at > org.apache.carbondata.core.scan.executor.impl.AbstractQueryExecutor.initQuery(AbstractQueryExecutor.java:141) > at > org.apache.carbondata.core.scan.executor.impl.AbstractQueryExecutor.getBlockExecutionInfos(AbstractQueryExecutor.java:401) > at > org.apache.carbondata.core.scan.executor.impl.VectorDetailQueryExecutor.execute(VectorDetailQueryExecutor.java:44) > at > org.apache.carbondata.spark.vectorreader.VectorizedCarbonRecordReader.initialize(VectorizedCarbonRecordReader.java:143) > at > org.apache.spark.sql.carbondata.execution.datasources.SparkCarbonFileFormat$$anonfun$buildReaderWithPartitionValues$2.apply(SparkCarbonFileFormat.scala:395) > at > org.apache.spark.sql.carbondata.execution.datasources.SparkCarbonFileFormat$$anonfun$buildReaderWithPartitionValues$2.apply(SparkCarbonFileFormat.scala:361) > at > org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:124) > at > org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:174) > at > org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:105) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.scan_nextBatch$(Unknown > Source) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithoutKey$(Unknown > Source) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown > Source) > at > org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) > at > org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) > at > org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) > at org.apache.spark.scheduler.Task.run(Task.scala:108) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2774: [CARBONDATA-2979] select count fails when car...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2774 ---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8950/ ---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/882/ ---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/686/ ---
[GitHub] carbondata pull request #2794: [CARBONDATA-2985]Fix issues in Table level co...
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2794#discussion_r222330949
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
---
@@ -744,6 +744,7 @@ private static long getSizeOfSegment(String tablePath,
String segId) {
if (size >= 2) {
level1Size = noOfSegmentLevelsCount[0];
level2Size = noOfSegmentLevelsCount[1];
+ level2Size = level2Size == 1 ? 0 : level2Size;
--- End diff --
Please add a comment why to handle this
---
[GitHub] carbondata issue #2774: [CARBONDATA-2979] select count fails when carbondata...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2774 LGTM ---
[GitHub] carbondata pull request #2793: [CARBONDATA-2990] Fixed JVM crash when rebuil...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2793#discussion_r222318367
--- Diff:
core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java
---
@@ -217,6 +212,17 @@ public static MemoryBlock
allocateMemoryWithRetry(MemoryAllocator memoryAllocato
return baseBlock;
}
+ private MemoryAllocator getMemoryAllocator(MemoryType memoryType) {
+switch (memoryType) {
+ case ONHEAP:
+return MemoryAllocator.HEAP;
+ case OFFHEAP:
+return MemoryAllocator.UNSAFE;
+ default:
+return MemoryAllocator.UNSAFE;
--- End diff --
combine OFFHEAP and default case as one
`case OFFHEAP:
default:
return MemoryAllocator.UNSAFE;`
---
[jira] [Updated] (CARBONDATA-2991) NegativeArraySizeException during query execution
[
https://issues.apache.org/jira/browse/CARBONDATA-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Babulal updated CARBONDATA-2991:
Description:
During Query Execution sometime NegativeArraySizeException Exception in Some
Tasks . And sometime Executor is lost (JVM crash)
ava.lang.NegativeArraySizeException at
org.apache.carbondata.core.datastore.chunk.store.impl.unsafe.UnsafeVariableLengthDimesionDataChunkStore.getRow(UnsafeVariableLengthDimesionDataChunkStore.java:157)
at
org.apache.carbondata.core.datastore.chunk.impl.AbstractDimensionDataChunk.getChunkData(AbstractDimensionDataChunk.java:46)
at
org.apache.carbondata.core.scan.result.AbstractScannedResult.getNoDictionaryKeyArray(AbstractScannedResult.java:470)
at
org.apache.carbondata.core.scan.result.impl.NonFilterQueryScannedResult.getNoDictionaryKeyArray(NonFilterQueryScannedResult.java:102)
at
org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.collectData(DictionaryBasedResultCollector.java:101)
at
org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:51)
at
org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:32)
at
org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:49)
Issue Analysis :-
Possible Root Cause :- It is because existing memoryblock is removed while it
was in-use. This happened because duplicate taskid generated. Sometime freed
same memory addresses are assigned to another task which will initialize memory
block to0 and this cause NegativeSizeArrayException whereas sometime freed
memory will not be used any task of executor process but running task will try
to access it and as that address is not part of process so JVM crash will
happen.
*Steps to find cause*
*Add the code to create tasklist ids and add taskid to list on
setCarbonTaskInfo() , and if it is duplicate then Log a Warn message.*
Please check attachment.
Run the Query multiple time and found warn message in executor logs
2018-09-29 14:48:41,840 | INFO | [[Executor task launch worker for task
435242][partitionID:1;queryID:29971946625611231]] | [Executor task launch
worker for task 435242][partitionID:1;queryID:29971946625611231] Total memory
used after task 29971946381679677 is 0 Current tasks running now are : [] |
org.apache.carbondata.common.logging.impl.StandardLogService.logInfoMessage(StandardLogService.java:150)
2018-09-29 14:48:41,840 | INFO | [[Executor task launch worker for task
435242][partitionID:1;queryID:29971946625611231]] | Finished task 17091.0 in
stage 22.0 (TID 435242). 1412 bytes result sent to driver |
org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
{color:#FF}*2018-09-29 14:48:41,842 | WARN | [Executor task launch worker
for task 435393] | Executor task launch worker for task 435393 Already This
Task is is Present29971946637094373 |
org.apache.carbondata.common.logging.impl.StandardLogService.logWarnMessage(StandardLogService.java:168)*{color}
2018-09-29 14:48:41,842 | INFO | [dispatcher-event-loop-13] | Got assigned task
435395 | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
was:
During Query Execution sometime NegativeArraySizeException Exception in Some
Tasks . And sometime Executor is lost (JVM crash)
ava.lang.NegativeArraySizeException at
org.apache.carbondata.core.datastore.chunk.store.impl.unsafe.UnsafeVariableLengthDimesionDataChunkStore.getRow(UnsafeVariableLengthDimesionDataChunkStore.java:157)
at
org.apache.carbondata.core.datastore.chunk.impl.AbstractDimensionDataChunk.getChunkData(AbstractDimensionDataChunk.java:46)
at
org.apache.carbondata.core.scan.result.AbstractScannedResult.getNoDictionaryKeyArray(AbstractScannedResult.java:470)
at
org.apache.carbondata.core.scan.result.impl.NonFilterQueryScannedResult.getNoDictionaryKeyArray(NonFilterQueryScannedResult.java:102)
at
org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.collectData(DictionaryBasedResultCollector.java:101)
at
org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:51)
at
org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:32)
at
org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:49)
Issue Analysis :-
Possible Root Cause :- It is because existing memoryblock is removed while it
was in-use. This happened because duplicate taskid generated. Sometime freed
same memory addresses are assigned to another task which will initialize memory
block to0 and this cause NegativeSizeArrayException whereas sometime freed
memory will not be used any task of executor process but running task will try
[GitHub] carbondata issue #2795: [HOTFIX] Changes to align printing of information in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2795 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/878/ ---
[GitHub] carbondata issue #2795: [HOTFIX] Changes to align printing of information in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2795 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8946/ ---
[jira] [Updated] (CARBONDATA-2991) NegativeArraySizeException during query execution
[ https://issues.apache.org/jira/browse/CARBONDATA-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babulal updated CARBONDATA-2991: Attachment: Root_Cause_Find_Step.JPG > NegativeArraySizeException during query execution > -- > > Key: CARBONDATA-2991 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2991 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.4.0, 1.3.1 >Reporter: Babulal >Assignee: Babulal >Priority: Major > Attachments: Root_Cause_Find_Step.JPG > > > During Query Execution sometime NegativeArraySizeException Exception in Some > Tasks . And sometime Executor is lost (JVM crash) > > ava.lang.NegativeArraySizeException at > org.apache.carbondata.core.datastore.chunk.store.impl.unsafe.UnsafeVariableLengthDimesionDataChunkStore.getRow(UnsafeVariableLengthDimesionDataChunkStore.java:157) > at > org.apache.carbondata.core.datastore.chunk.impl.AbstractDimensionDataChunk.getChunkData(AbstractDimensionDataChunk.java:46) > at > org.apache.carbondata.core.scan.result.AbstractScannedResult.getNoDictionaryKeyArray(AbstractScannedResult.java:470) > at > org.apache.carbondata.core.scan.result.impl.NonFilterQueryScannedResult.getNoDictionaryKeyArray(NonFilterQueryScannedResult.java:102) > at > org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.collectData(DictionaryBasedResultCollector.java:101) > at > org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:51) > at > org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:32) > at > org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:49) > > > > Issue Analysis :- > Possible Root Cause :- It is because existing memoryblock is removed while it > was in-use. This happened because duplicate taskid generated. Sometime freed > same memory addresses are assigned to another task which will initialize > memory block to0 and this cause NegativeSizeArrayException whereas sometime > freed memory will not be used any task of executor process but running task > will try to access it and as that address is not part of process so JVM crash > will happen. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2991) NegativeArraySizeException during query execution
[ https://issues.apache.org/jira/browse/CARBONDATA-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babulal updated CARBONDATA-2991: Description: During Query Execution sometime NegativeArraySizeException Exception in Some Tasks . And sometime Executor is lost (JVM crash) ava.lang.NegativeArraySizeException at org.apache.carbondata.core.datastore.chunk.store.impl.unsafe.UnsafeVariableLengthDimesionDataChunkStore.getRow(UnsafeVariableLengthDimesionDataChunkStore.java:157) at org.apache.carbondata.core.datastore.chunk.impl.AbstractDimensionDataChunk.getChunkData(AbstractDimensionDataChunk.java:46) at org.apache.carbondata.core.scan.result.AbstractScannedResult.getNoDictionaryKeyArray(AbstractScannedResult.java:470) at org.apache.carbondata.core.scan.result.impl.NonFilterQueryScannedResult.getNoDictionaryKeyArray(NonFilterQueryScannedResult.java:102) at org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.collectData(DictionaryBasedResultCollector.java:101) at org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:51) at org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:32) at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:49) Issue Analysis :- Possible Root Cause :- It is because existing memoryblock is removed while it was in-use. This happened because duplicate taskid generated. Sometime freed same memory addresses are assigned to another task which will initialize memory block to0 and this cause NegativeSizeArrayException whereas sometime freed memory will not be used any task of executor process but running task will try to access it and as that address is not part of process so JVM crash will happen. was: During Query Execution sometime NegativeArraySizeException Exception in Some Tasks . And sometime Executor is lost (JVM crash) ava.lang.NegativeArraySizeException at org.apache.carbondata.core.datastore.chunk.store.impl.unsafe.UnsafeVariableLengthDimesionDataChunkStore.getRow(UnsafeVariableLengthDimesionDataChunkStore.java:157) at org.apache.carbondata.core.datastore.chunk.impl.AbstractDimensionDataChunk.getChunkData(AbstractDimensionDataChunk.java:46) at org.apache.carbondata.core.scan.result.AbstractScannedResult.getNoDictionaryKeyArray(AbstractScannedResult.java:470) at org.apache.carbondata.core.scan.result.impl.NonFilterQueryScannedResult.getNoDictionaryKeyArray(NonFilterQueryScannedResult.java:102) at org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.collectData(DictionaryBasedResultCollector.java:101) at org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:51) at org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:32) at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:49) Issue Analysis :- Possible Root Cause :- It is because existing memoryblock is removed while it was in-use. This happened because duplicate taskid generated. Sometime freed same memory addresses are assigned to another task which will initialize memory block to0 and this cause NegativeSizeArrayException whereas sometime freed memory will not be used any task of executor process but running task will try to access it and as that address is not part of process so JVM crash will happen. Method to find Cause :- !Root_Cause_Find_Step.JPG! !Log_Message.JPG! > NegativeArraySizeException during query execution > -- > > Key: CARBONDATA-2991 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2991 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.4.0, 1.3.1 >Reporter: Babulal >Priority: Major > > During Query Execution sometime NegativeArraySizeException Exception in Some > Tasks . And sometime Executor is lost (JVM crash) > > ava.lang.NegativeArraySizeException at > org.apache.carbondata.core.datastore.chunk.store.impl.unsafe.UnsafeVariableLengthDimesionDataChunkStore.getRow(UnsafeVariableLengthDimesionDataChunkStore.java:157) > at > org.apache.carbondata.core.datastore.chunk.impl.AbstractDimensionDataChunk.getChunkData(AbstractDimensionDataChunk.java:46) > at > org.apache.carbondata.core.scan.result.AbstractScannedResult.getNoDictionaryKeyArray(AbstractScannedResult.java:470) > at > org.apache.carbondata.core.scan.result.impl.NonFilterQueryScannedResult.getNoDictionaryKeyArray(NonFilterQueryScannedResult.java:102) > at >
[jira] [Assigned] (CARBONDATA-2991) NegativeArraySizeException during query execution
[ https://issues.apache.org/jira/browse/CARBONDATA-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babulal reassigned CARBONDATA-2991: --- Assignee: Babulal > NegativeArraySizeException during query execution > -- > > Key: CARBONDATA-2991 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2991 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.4.0, 1.3.1 >Reporter: Babulal >Assignee: Babulal >Priority: Major > > During Query Execution sometime NegativeArraySizeException Exception in Some > Tasks . And sometime Executor is lost (JVM crash) > > ava.lang.NegativeArraySizeException at > org.apache.carbondata.core.datastore.chunk.store.impl.unsafe.UnsafeVariableLengthDimesionDataChunkStore.getRow(UnsafeVariableLengthDimesionDataChunkStore.java:157) > at > org.apache.carbondata.core.datastore.chunk.impl.AbstractDimensionDataChunk.getChunkData(AbstractDimensionDataChunk.java:46) > at > org.apache.carbondata.core.scan.result.AbstractScannedResult.getNoDictionaryKeyArray(AbstractScannedResult.java:470) > at > org.apache.carbondata.core.scan.result.impl.NonFilterQueryScannedResult.getNoDictionaryKeyArray(NonFilterQueryScannedResult.java:102) > at > org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.collectData(DictionaryBasedResultCollector.java:101) > at > org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:51) > at > org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:32) > at > org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:49) > > > > Issue Analysis :- > Possible Root Cause :- It is because existing memoryblock is removed while it > was in-use. This happened because duplicate taskid generated. Sometime freed > same memory addresses are assigned to another task which will initialize > memory block to0 and this cause NegativeSizeArrayException whereas sometime > freed memory will not be used any task of executor process but running task > will try to access it and as that address is not part of process so JVM crash > will happen. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2793: [CARBONDATA-2990] Fixed JVM crash when rebuilding bl...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2793 @xuchuanyin It can happen only in a cluster, that's why it is not caught in UT. ---
[GitHub] carbondata issue #2787: [HOTFIX] Fixed Between Query Data Mismatch issue for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2787 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/685/ ---
[jira] [Created] (CARBONDATA-2991) NegativeArraySizeException during query execution
Babulal created CARBONDATA-2991: --- Summary: NegativeArraySizeException during query execution Key: CARBONDATA-2991 URL: https://issues.apache.org/jira/browse/CARBONDATA-2991 Project: CarbonData Issue Type: Bug Affects Versions: 1.3.1, 1.4.0 Reporter: Babulal During Query Execution sometime NegativeArraySizeException Exception in Some Tasks . And sometime Executor is lost (JVM crash) ava.lang.NegativeArraySizeException at org.apache.carbondata.core.datastore.chunk.store.impl.unsafe.UnsafeVariableLengthDimesionDataChunkStore.getRow(UnsafeVariableLengthDimesionDataChunkStore.java:157) at org.apache.carbondata.core.datastore.chunk.impl.AbstractDimensionDataChunk.getChunkData(AbstractDimensionDataChunk.java:46) at org.apache.carbondata.core.scan.result.AbstractScannedResult.getNoDictionaryKeyArray(AbstractScannedResult.java:470) at org.apache.carbondata.core.scan.result.impl.NonFilterQueryScannedResult.getNoDictionaryKeyArray(NonFilterQueryScannedResult.java:102) at org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.collectData(DictionaryBasedResultCollector.java:101) at org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:51) at org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:32) at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:49) Issue Analysis :- Possible Root Cause :- It is because existing memoryblock is removed while it was in-use. This happened because duplicate taskid generated. Sometime freed same memory addresses are assigned to another task which will initialize memory block to0 and this cause NegativeSizeArrayException whereas sometime freed memory will not be used any task of executor process but running task will try to access it and as that address is not part of process so JVM crash will happen. Method to find Cause :- !Root_Cause_Find_Step.JPG! !Log_Message.JPG! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2788: [Documentation] Readme updated with latest topics an...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2788 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/684/ ---
[GitHub] carbondata issue #2788: [Documentation] Readme updated with latest topics an...
Github user sgururajshetty commented on the issue: https://github.com/apache/carbondata/pull/2788 @sraghunandan & @kunal642 kindly review and merge the doc ---
[GitHub] carbondata issue #2795: [HOTFIX] Changes to align printing of information in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2795 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/683/ ---
[GitHub] carbondata pull request #2795: [HOTFIX] Changes to align printing of informa...
GitHub user manishgupta88 opened a pull request: https://github.com/apache/carbondata/pull/2795 [HOTFIX] Changes to align printing of information in explain command based on enable.query.statistics flag Now user need to set the flag enable.query.statistics = true to print the pruning and dataMap related information in the explain command. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/manishgupta88/carbondata explain_profiler_disable_default Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2795.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2795 commit 2b03fe91efd2ba144e6ef988750fc1e8c63d40f5 Author: manishgupta88 Date: 2018-10-03T12:43:45Z [HOTFIX] Changes to align printing of information in explain command based on enable.query.statistics flag ---
[GitHub] carbondata issue #2792: [CARBONDATA-2981] Support read primitive data type i...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2792 @xubo245 Please add link for CSDK-guide in README file. ---
[jira] [Resolved] (CARBONDATA-2952) Provide CarbonReader C++ interface for SDK
[ https://issues.apache.org/jira/browse/CARBONDATA-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-2952. -- Resolution: Fixed > Provide CarbonReader C++ interface for SDK > -- > > Key: CARBONDATA-2952 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2952 > Project: CarbonData > Issue Type: Sub-task > Components: other >Affects Versions: 1.5.0 >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Fix For: 1.5.0 > > Time Spent: 15h 40m > Remaining Estimate: 0h > > Provide CarbonReader C++ interface for SDK > 1.init carbonreader > 2.config data path and tablename > 3.configure projection > 4.build carbon reader > 5.hasNext > 6.readNextRow > 7.close -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2738 ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2738 LGTM. One minor comment -> Reference for CSDK-guide.md is missing from README due to which there is no way to navigate to the guide. Please handle this as part of #2792 ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2738 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/877/ ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2738 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8945/ ---
[jira] [Resolved] (CARBONDATA-2982) CarbonSchemaReader don't support Array
[
https://issues.apache.org/jira/browse/CARBONDATA-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manish Gupta resolved CARBONDATA-2982.
--
Resolution: Fixed
> CarbonSchemaReader don't support Array
> --
>
> Key: CARBONDATA-2982
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2982
> Project: CarbonData
> Issue Type: Bug
> Components: other
>Affects Versions: 1.5.0
>Reporter: xubo245
>Assignee: xubo245
>Priority: Major
> Fix For: 1.5.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> CarbonSchemaReader don't support Array
> When we read schema from index file and the data include array data
> type
> run org.apache.carbondata.examples.sdk.CarbonReaderExample :
> {code:java}
> Schema schema = CarbonSchemaReader
> .readSchemaInIndexFile(dataFiles[0].getAbsolutePath())
> .asOriginOrder();
> // Transform the schema
> String[] strings = new String[schema.getFields().length];
> for (int i = 0; i < schema.getFields().length; i++) {
> strings[i] = (schema.getFields())[i].getFieldName();
> System.out.println(strings[i] + "\t" +
> schema.getFields()[i].getSchemaOrdinal());
> }
> {code}
> and throw some exception:
> {code:java}
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> arrayfield.val0 -1
> stringfield 0
> shortfield1
> intfield 2
> longfield 3
> doublefield 4
> boolfield 5
> datefield 6
> timefield 7
> decimalfield 8
> varcharfield 9
> arrayfield10
> Complex child columns projection NOT supported through CarbonReader
> java.lang.UnsupportedOperationException: Complex child columns projection NOT
> supported through CarbonReader
> at
> org.apache.carbondata.sdk.file.CarbonReaderBuilder.build(CarbonReaderBuilder.java:155)
> at
> org.apache.carbondata.examples.sdk.CarbonReaderExample.main(CarbonReaderExample.java:110)
> {code}
> It print arrayfield.val0 -1, child schema
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata pull request #2780: [CARBONDATA-2982] CarbonSchemaReader support ...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2780 ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2738 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/682/ ---
[GitHub] carbondata pull request #2789: [HOTFIX] Fixed S3 metrics issue.
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2789 ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2738 retest this please ---
[GitHub] carbondata issue #2787: [HOTFIX] Fixed Between Query Data Mismatch issue for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2787 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8944/ ---
[GitHub] carbondata issue #2789: [HOTFIX] Fixed S3 metrics issue.
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/2789 LGTM ---
[GitHub] carbondata issue #2789: [HOTFIX] Fixed S3 metrics issue.
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2789 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8943/ ---
[GitHub] carbondata issue #2787: [HOTFIX] Fixed Between Query Data Mismatch issue for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2787 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/876/ ---
[GitHub] carbondata issue #2789: [HOTFIX] Fixed S3 metrics issue.
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2789 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/875/ ---
[GitHub] carbondata issue #2780: [CARBONDATA-2982] CarbonSchemaReader support array
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2780 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8942/ ---
[GitHub] carbondata issue #2786: [CARBONDATA-2967] Fixed NPE for preaggregate queries
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2786 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/872/ ---
[GitHub] carbondata issue #2789: [HOTFIX] Fixed S3 metrics issue.
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2789 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/681/ ---
[GitHub] carbondata issue #2780: [CARBONDATA-2982] CarbonSchemaReader support array
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2780 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/874/ ---
[GitHub] carbondata pull request #2791: [HOTFIX]correct the exception handling in loo...
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2791#discussion_r52487
--- Diff:
integration/spark2/src/main/spark2.1/org/apache/spark/sql/hive/CarbonSessionState.scala
---
@@ -141,7 +141,13 @@ class CarbonHiveSessionCatalog(
*/
override def lookupRelation(name: TableIdentifier,
alias: Option[String]): LogicalPlan = {
-val rtnRelation = super.lookupRelation(name, alias)
+val rtnRelation =
+ try {
+super.lookupRelation(name, alias)
+ } catch {
+case ex: Exception =>
--- End diff --
This change is not required. Catch and throwing same exception.
---
[GitHub] carbondata issue #2786: [CARBONDATA-2967] Fixed NPE for preaggregate queries
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2786 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8940/ ---
[GitHub] carbondata issue #2787: [HOTFIX] Fixed Between Query Data Mismatch issue for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2787 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/680/ ---
[GitHub] carbondata issue #2789: [HOTFIX] Fixed S3 metrics issue.
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/2789 retest this please ---
[GitHub] carbondata issue #2729: [WIP]Store optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2729 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8941/ ---
[GitHub] carbondata issue #2787: [HOTFIX] Fixed Between Query Data Mismatch issue for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2787 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8939/ ---
[GitHub] carbondata issue #2780: [CARBONDATA-2982] CarbonSchemaReader support array
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2780 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/679/ ---
[GitHub] carbondata issue #2729: [WIP]Store optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2729 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/873/ ---
[GitHub] carbondata issue #2729: [WIP]Store optimization
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2729 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/678/ ---
[GitHub] carbondata issue #2780: [CARBONDATA-2982] CarbonSchemaReader support array
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2780 retest this please ---
