[jira] [Updated] (CARBONDATA-4235) after alter add column when user does rename operation ,the select operation on struct type gives null value and childen of struct gives error
[ https://issues.apache.org/jira/browse/CARBONDATA-4235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Bhat updated CARBONDATA-4235: Description: *Queries –* drop table if exists test_rename; CREATE TABLE test_rename (str1 struct, str2 struct>, str3 struct>> comment 'struct', intfield int,arr1 array, arr2 array>, arr3 array, arr4 array> comment 'array') STORED AS carbondata; insert into test_rename values (named_struct('a', 2), named_struct('a', named_struct('b', 2)), named_struct('a', named_struct('b',named_struct('c', 2))), 1,array(1,2,3), array(array(1,2),array(3,4)), array('hello','world'), array(named_struct('a',45))); ALTER TABLE test_rename ADD COLUMNS(arr_1 ARRAY); alter table test_rename change str2 str22 struct>; select str22 from test_rename; select str22.a from test_rename; select str22.a.b from test_rename; Issue : after alter add column when user does rename operation ,the select operation on struct type gives null value and childen of struct gives error *Issue 1 : Exception trace on executing query –* 0: jdbc:hive2://vm2:22550/> select str22.a.b from test_rename; INFO : Execution ID: 2465 Error: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1100.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1100.0 (TID 10353) (vm1 executor 5): java.nio.BufferUnderflowException at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:155) at org.apache.carbondata.core.scan.complextypes.PrimitiveQueryType.getDataObject(PrimitiveQueryType.java:166) at org.apache.carbondata.core.scan.complextypes.PrimitiveQueryType.getDataObject(PrimitiveQueryType.java:147) at org.apache.carbondata.core.scan.complextypes.PrimitiveQueryType.getDataBasedOnColumn(PrimitiveQueryType.java:141) at org.apache.carbondata.core.scan.complextypes.StructQueryType.getDataBasedOnColumn(StructQueryType.java:160) at org.apache.carbondata.core.scan.complextypes.StructQueryType.getDataBasedOnColumn(StructQueryType.java:160) at org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.fillRow(DictionaryBasedResultCollector.java:316) at org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.fillDimensionData(DictionaryBasedResultCollector.java:288) at org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.collectResultInRow(DictionaryBasedResultCollector.java:159) at org.apache.carbondata.core.scan.processor.DataBlockIterator.next(DataBlockIterator.java:110) at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:58) at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:50) at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:32) at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.hasNext(ChunkRowIterator.java:56) at org.apache.carbondata.hadoop.CarbonRecordReader.nextKeyValue(CarbonRecordReader.java:127) at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.hasNext(CarbonScanRDD.scala:557) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:345) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:897) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:897) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:499) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1554) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:502) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Driver stacktrace: at
[jira] [Created] (CARBONDATA-4235) after alter add column when user does rename operation ,the select operation on struct type gives null value and childen of struct gives error
Chetan Bhat created CARBONDATA-4235: --- Summary: after alter add column when user does rename operation ,the select operation on struct type gives null value and childen of struct gives error Key: CARBONDATA-4235 URL: https://issues.apache.org/jira/browse/CARBONDATA-4235 Project: CarbonData Issue Type: Bug Components: sql Affects Versions: 2.2.0 Environment: Spark 3.1.1, Spark 2.4.5 Reporter: Chetan Bhat *Queries –* drop table if exists test_rename; CREATE TABLE test_rename (str1 struct, str2 struct>, str3 struct>> comment 'struct', intfield int,arr1 array, arr2 array>, arr3 array, arr4 array> comment 'array') STORED AS carbondata; insert into test_rename values (named_struct('a', 2), named_struct('a', named_struct('b', 2)), named_struct('a', named_struct('b',named_struct('c', 2))), 1,array(1,2,3), array(array(1,2),array(3,4)), array('hello','world'), array(named_struct('a',45))); ALTER TABLE test_rename ADD COLUMNS(arr_1 ARRAY); alter table test_rename change str2 str22 struct>; select str22 from test_rename; select str22.a from test_rename; select str22.a.b from test_rename; Issue : after alter add column when user does rename operation ,the select operation on struct type gives null value and childen of struct gives error *Exception trace on executing query –* 0: jdbc:hive2://vm2:22550/> select str22.a.b from test_rename; INFO : Execution ID: 2465 Error: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1100.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1100.0 (TID 10353) (vm1 executor 5): java.nio.BufferUnderflowException at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:155) at org.apache.carbondata.core.scan.complextypes.PrimitiveQueryType.getDataObject(PrimitiveQueryType.java:166) at org.apache.carbondata.core.scan.complextypes.PrimitiveQueryType.getDataObject(PrimitiveQueryType.java:147) at org.apache.carbondata.core.scan.complextypes.PrimitiveQueryType.getDataBasedOnColumn(PrimitiveQueryType.java:141) at org.apache.carbondata.core.scan.complextypes.StructQueryType.getDataBasedOnColumn(StructQueryType.java:160) at org.apache.carbondata.core.scan.complextypes.StructQueryType.getDataBasedOnColumn(StructQueryType.java:160) at org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.fillRow(DictionaryBasedResultCollector.java:316) at org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.fillDimensionData(DictionaryBasedResultCollector.java:288) at org.apache.carbondata.core.scan.collector.impl.DictionaryBasedResultCollector.collectResultInRow(DictionaryBasedResultCollector.java:159) at org.apache.carbondata.core.scan.processor.DataBlockIterator.next(DataBlockIterator.java:110) at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:58) at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:50) at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:32) at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.hasNext(ChunkRowIterator.java:56) at org.apache.carbondata.hadoop.CarbonRecordReader.nextKeyValue(CarbonRecordReader.java:127) at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.hasNext(CarbonScanRDD.scala:557) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:345) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:897) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:897) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:499) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1554) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:502) at
[jira] [Created] (CARBONDATA-4234) Alter change datatype at nested levels
Akshay created CARBONDATA-4234: -- Summary: Alter change datatype at nested levels Key: CARBONDATA-4234 URL: https://issues.apache.org/jira/browse/CARBONDATA-4234 Project: CarbonData Issue Type: Sub-task Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4199) Support renaming of map columns including nested levels
[ https://issues.apache.org/jira/browse/CARBONDATA-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4199: --- Summary: Support renaming of map columns including nested levels (was: Support renaming of map columns) > Support renaming of map columns including nested levels > --- > > Key: CARBONDATA-4199 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4199 > Project: CarbonData > Issue Type: Sub-task >Reporter: Akshay >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4233) Altering complex structure like add and drop at nested level
[ https://issues.apache.org/jira/browse/CARBONDATA-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4233: --- Parent: CARBONDATA-4160 Issue Type: Sub-task (was: Bug) > Altering complex structure like add and drop at nested level > > > Key: CARBONDATA-4233 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4233 > Project: CarbonData > Issue Type: Sub-task >Reporter: Akshay >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4233) Altering complex structure like add and drop at nested level
Akshay created CARBONDATA-4233: -- Summary: Altering complex structure like add and drop at nested level Key: CARBONDATA-4233 URL: https://issues.apache.org/jira/browse/CARBONDATA-4233 Project: CarbonData Issue Type: Bug Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4179) Support renaming of complex columns including nested levels(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4179: --- Summary: Support renaming of complex columns including nested levels(array/struct) (was: Support renaming of complex columns(array/struct)) > Support renaming of complex columns including nested levels(array/struct) > - > > Key: CARBONDATA-4179 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4179 > Project: CarbonData > Issue Type: Sub-task >Reporter: Akshay >Priority: Major > Fix For: 2.2.0 > > Time Spent: 21.5h > Remaining Estimate: 0h > > Complex columns supported for - *array and struct.* > For schema lets say- *str struct* the following points will be > considered - > 1) Rename parent name *str to str2* > alter table test_rename change str str2 struct > > 2) Rename child name from *a to b* > alter table test_rename change str str struct > > *NOTE-* > Rename operation fails if the structure of the complex column has been > altered. This check ensures the old and new columns are compatible with each > other. Meaning the *number of children* and *complex levels* should be > unaltered while attempting to rename. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4170) Support dropping of parent complex columns(for both single and multi-level array/struct/map)
[ https://issues.apache.org/jira/browse/CARBONDATA-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4170: --- Summary: Support dropping of parent complex columns(for both single and multi-level array/struct/map) (was: Support dropping of parent complex columns(array/struct/map)) > Support dropping of parent complex columns(for both single and multi-level > array/struct/map) > > > Key: CARBONDATA-4170 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4170 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Akshay >Priority: Major > Fix For: 2.2.0 > > Time Spent: 10h 40m > Remaining Estimate: 0h > > Drop complex columns(array/struct/map) from carbon table. For example - > arr1 array, struct1 struct, map1 map > Command - > ALTER TABLE DROP COLUMNS(arr1, struct1, map1) > Design document - > [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4232) Add missing doc change for secondary index.
Nihal kumar ojha created CARBONDATA-4232: Summary: Add missing doc change for secondary index. Key: CARBONDATA-4232 URL: https://issues.apache.org/jira/browse/CARBONDATA-4232 Project: CarbonData Issue Type: Bug Reporter: Nihal kumar ojha Doc changes were not handled for PR-4116 to leverage secondary index till segment level. -- This message was sent by Atlassian Jira (v8.3.4#803005)