[jira] [Created] (CARBONDATA-4275) alter drop column fails for all primitive datatype except integer on carbon table with spark 3.1.1
PRIYESH RANJAN created CARBONDATA-4275: -- Summary: alter drop column fails for all primitive datatype except integer on carbon table with spark 3.1.1 Key: CARBONDATA-4275 URL: https://issues.apache.org/jira/browse/CARBONDATA-4275 Project: CarbonData Issue Type: Bug Components: data-query Affects Versions: 2.2.0 Environment: Spark 3.1.1 Reporter: PRIYESH RANJAN Steps:- *Issue : alter drop column fails for all primitive datatype except integer on carbon table with spark 3.1.1* CREATE TABLE uniqdata_alter(CUST_ID int ,CUST_NAME string,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double, INTEGER_COLUMN1 int) stored as carbondata ; *Scenario 1*: alter table uniqdata_alter drop columns(Double_COLUMN1); *Scenario 2 :* alter table uniqdata_alter drop columns(cust_name); *Scenario 3:* alter table uniqdata_alter drop columns(DECIMAL_COLUMN2); *Scenario 4:* alter table uniqdata_alter drop columns(BIGINT_COLUMN2); *Scenario 5:* alter table uniqdata_alter drop columns(DOB); *+error message for all above scenario:+* 0: jdbc:hive2://10.21.19.14:23040/default> alter table uniqdata_alter drop columns(Double_COLUMN1); Error: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.carbondata.spark.exception.ProcessMetaDataException: operation failed for default.uniqdata_alter: Alter table drop column operation failed: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions : col at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:361) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:263) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:78) at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:62) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:43) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:263) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:258) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:272) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.carbondata.spark.exception.ProcessMetaDataException: operation failed for default.uniqdata_alter: Alter table drop column operation failed: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions : col at org.apache.spark.sql.execution.command.MetadataProcessOperation.throwMetadataException(package.scala:69) at org.apache.spark.sql.execution.command.MetadataProcessOperation.throwMetadataException$(package.scala:68) at org.apache.spark.sql.execution.command.MetadataCommand.throwMetadataException(package.scala:134) at org.apache.spark.sql.execution.command.schema.CarbonAlterTableDropColumnCommand.processMetadata(CarbonAlterTableDropColumnCommand.scala:216) at org.apache.spark.sql.execution.command.MetadataCommand.$anonfun$run$1(package.scala:137) at org.apache.spark.sql.execution.command.Auditable.runWithAudit(package.scala:118) at org.apache.spark.sql.execution.command.Auditable.runWithAudit$(package.scala:114) at org.apache.spark.sql.execution.command.MetadataCommand.runWithAudit(package.scala:134) at org.apache.spark.sql.execution.command.MetadataCommand.run(package.scala:137) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) at org.apache.spark.sql.execution.command.Ex
[jira] [Resolved] (CARBONDATA-4237) documentation issues in github master docs.
[ https://issues.apache.org/jira/browse/CARBONDATA-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi resolved CARBONDATA-4237. Fix Version/s: 2.3.0 Resolution: Fixed > documentation issues in github master docs. > --- > > Key: CARBONDATA-4237 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4237 > Project: CarbonData > Issue Type: Bug > Components: docs >Affects Versions: 2.2.0 > Environment: Contents verified on Spark 2.4.5 and Spark 3.1.1 >Reporter: PRIYESH RANJAN >Priority: Minor > Fix For: 2.3.0 > > > +Modification 1 :+ > [https://github.com/apache/carbondata/blob/master/docs/streaming-guide.md] > Streaming table don't support alter table operation(alter add columns, drop > column, rename column, change datatypes and rename table name) so In > Constraint section of this doc ,it can be added. > > 0: jdbc:hive2://100-112-148-186:22550/> alter table uniqdata_alter add > columns(id2 int); > Error: org.apache.hive.service.cli.HiveSQLException: Error running query: > org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException: > Alter table add column is not allowed for streaming table > 0: jdbc:hive2://100-112-148-186:22550/> alter table uniqdata_alter drop > columns(integer_column1); > Error: org.apache.hive.service.cli.HiveSQLException: Error running query: > org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException: > Alter table drop column is not allowed for streaming table. > 0: jdbc:hive2://100-112-148-186:22550/> ALTER TABLE uniqdata_alter rename TO > uniqdata_alterTable ; > Error: org.apache.hive.service.cli.HiveSQLException: Error running query: > org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException: > Alter rename table is not allowed for streaming table. > > +Modification 2 :+ > [https://github.com/apache/carbondata/blob/master/docs/file-structure-of-carbondata.md] > Since Metadata folder contain segment, tablestatus and schema folder so > dictionary file related content inside metadata folder can be removed from > doc. > eg : Metadata directory stores schema files, tablestatus and *dictionary > files (including .dict, .dictmeta and .sortindex).* These line from doc can > be modifed as Metadata directory stores schema files, tablestatus and > segments details. > > +Modification 3 :+ > [https://github.com/apache/carbondata/blob/master/docs/sdk-guide.md] > In the Quick Example section of following doc, it still converting date > datatype to Integer value and timestamp datatype to long value whereas now > they accept value as date and timestamp value respectively. > > {{while (reader.hasNext()) { > Object[] row = (Object[]) reader.readNextRow(); > System.out.println(String.format("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t", > i, row[0], row[1], row[2], row[3], row[4], row[5], > +*new Date((day * ((int) row[6]))), new Timestamp((long) row[7] / 1000)*+, > row[8] > )); > {{can be modified to}} > while (reader.hasNext()) { > Object[] row = (Object[]) reader.readNextRow(); > > System.out.println(String.format("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t", > i, row[0], row[1], row[2], row[3], row[4], row[5], +*row[6], row[7]*+, > row[8], row[9] > ));\{{}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-4236) Documentation correctness and link issues in https://github.com/apache/carbondata/blob/master/docs/
[ https://issues.apache.org/jira/browse/CARBONDATA-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi resolved CARBONDATA-4236. Fix Version/s: 2.3.0 Resolution: Fixed > Documentation correctness and link issues in > https://github.com/apache/carbondata/blob/master/docs/ > --- > > Key: CARBONDATA-4236 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4236 > Project: CarbonData > Issue Type: Bug > Components: docs >Affects Versions: 2.2.0 > Environment: docs with content and examples verified on Spark 2.4.5 > and Spark 3.1.1 compatible carbon. >Reporter: Chetan Bhat >Priority: Minor > Fix For: 2.3.0 > > > In the documentation link > https://github.com/apache/carbondata/blob/master/docs/ > Issue 1 :- > In link -> > https://github.com/apache/carbondata/blob/master/docs/configuration-parameters.md > the "See detail" links does not open the target > "http://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence"; > In link --> > https://github.com/apache/carbondata/blob/master/docs/documentation.md the > link "Apache CarbonData wiki" when clicked tries to open link > "https://cwiki.apache.org/confluence/display/CARBONDATA/CarbonData+Home"; the > target page cant be opened. Similarly the other links in the "External > Resources" section cant be opened due to the same error. > In link > https://github.com/apache/carbondata/blob/master/docs/faq.md#what-are-bad-records > the link "https://thrift.apache.org/docs/install"; when clicked does not open > the target page. > In link > https://github.com/apache/carbondata/blob/master/docs/quick-start-guide.md > when the "Spark website" link is clicked > https://spark.apache.org/downloads.html page is not opened. Also in same page > when the "Apache Spark Documentation" link is clicked the > "http://spark.apache.org/docs/latest/"; page is not opened. > In the link > https://github.com/apache/carbondata/blob/master/docs/release-guide.md > "Product Release Policy link" , "release signing guidelines" , "Apache Nexus > repository" and "repository.apache.org" when clicked the target pages are not > opening. > Issue 2:- > In link --> > https://github.com/apache/carbondata/blob/master/docs/configuration-parameters.md > the "To configure Ranges-based Compaction" to be changed to "To configure > Range-based Compaction" > Issue 3:- > In link --> > https://github.com/apache/carbondata/blob/master/docs/configuration-parameters.md > the "Making this true degrade the LOAD performance" to be changed to "Making > this true degrades the LOAD performance" > Issue 4 :- > In link --> > https://github.com/apache/carbondata/blob/master/docs/configuration-parameters.md > the "user an either set to true" to be changed to "user can either set to > true" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-4238) Documentation Issue in Github Docs Link https://github.com/apache/carbondata/blob/master/docs/ddl-of-carbondata.md#add-columns
[ https://issues.apache.org/jira/browse/CARBONDATA-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi resolved CARBONDATA-4238. Fix Version/s: 2.3.0 Resolution: Fixed > Documentation Issue in Github Docs Link > https://github.com/apache/carbondata/blob/master/docs/ddl-of-carbondata.md#add-columns > -- > > Key: CARBONDATA-4238 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4238 > Project: CarbonData > Issue Type: Bug > Components: docs >Affects Versions: 2.2.0 >Reporter: PURUJIT CHAUGULE >Priority: Minor > Fix For: 2.3.0 > > Attachments: Alter Add Complex.png, Alter Add > Complex_Error_message.png > > > [https://github.com/apache/carbondata/blob/master/docs/ddl-of-carbondata.md#add-columns] > * Example provided for Adding of only single-level Complex datatype > columns(only array and struct) has double level array column which is not > supported and needs to be changed to single level array column. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-4119) User Input for GeoID column not validated.
[ https://issues.apache.org/jira/browse/CARBONDATA-4119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Indhumathi resolved CARBONDATA-4119. Fix Version/s: 2.3.0 Resolution: Fixed > User Input for GeoID column not validated. > -- > > Key: CARBONDATA-4119 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4119 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 2.1.0 >Reporter: PURUJIT CHAUGULE >Priority: Minor > Fix For: 2.3.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > * User Input for geoId column can be paired to multiple pairs of source > columns values (correct internally calculated geoID values are different for > such above source columns values). > * The advantage of using geoID is not applicable when taking user input for > GeoId column is not validated and user input values may differ from actual > internally calculated values. GeoID value is only generated internally if > user does not input the geoID column. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-4240) Properties present in https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java which ar
[ https://issues.apache.org/jira/browse/CARBONDATA-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403711#comment-17403711 ] Pratyaksh Sharma commented on CARBONDATA-4240: -- There are few properties which are not getting used anywhere - {color:#172b4d}spark.carbon.datamanagement.driver{color} {color:#172b4d}carbon.lease.recovery.retry.count{color} {color:#172b4d}carbon.lease.recovery.retry.interval{color} {color:#172b4d}{color:#080808}carbon.number.of.cores.while.altPartition{color}{color} Also there are few properties which are getting only for testing purpose- {color:#172b4d}{color:#067d17}carbon.merge.index.in.segment{color}{color} {color:#172b4d}{color:#067d17}sort.inmemory.size.inmb{color}{color} > Properties present in > https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java > which are not present in open source doc > --- > > Key: CARBONDATA-4240 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4240 > Project: CarbonData > Issue Type: Bug > Components: docs >Affects Versions: 2.2.0 > Environment: Open source docs >Reporter: Chetan Bhat >Priority: Minor > > Properties present in > https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java > which are not present in open source doc as mentioned below. These > properties need to be updated in open source doc. > carbon.storelocation > carbon.blocklet.size > carbon.properties.filepath > carbon.date.format > carbon.complex.delimiter.level.1 > carbon.complex.delimiter.level.2 > carbon.complex.delimiter.level.3 > carbon.complex.delimiter.level.4 > carbon.lock.class > carbon.local.dictionary.enable > carbon.local.dictionary.decoder.fallback > spark.deploy.zookeeper.url > carbon.data.file.version > spark.carbon.hive.schema.store > spark.carbon.datamanagement.driver > spark.carbon.sessionstate.classname > spark.carbon.sqlastbuilder.classname > carbon.lease.recovery.retry.count > carbon.lease.recovery.retry.interval > carbon.index.schema.storage > carbon.merge.index.in.segment > carbon.number.of.cores.while.altPartition > carbon.minor.compaction.size > enable.unsafe.columnpage > carbon.lucene.compression.mode > sort.inmemory.size.inmb > is.driver.instance > carbon.input.metrics.update.interval > carbon.use.bitset.pipe.line > is.internal.load.call > carbon.lucene.index.stop.words > carbon.load.dateformat.setlenient.enable > carbon.infilter.subquery.pushdown.enable > broadcast.record.size > carbon.indexserver.tempfolder.deletetime -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CARBONDATA-4240) Properties present in https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java whi
[ https://issues.apache.org/jira/browse/CARBONDATA-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403711#comment-17403711 ] Pratyaksh Sharma edited comment on CARBONDATA-4240 at 8/24/21, 9:51 AM: There are few properties which are not getting used anywhere - {color:#172b4d}spark.carbon.datamanagement.driver{color} {color:#172b4d}carbon.lease.recovery.retry.count{color} {color:#172b4d}carbon.lease.recovery.retry.interval{color} {color:#172b4d}carbon.number.of.cores.while.altPartition{color} Also there are few properties which are getting used only for testing purpose- carbon.merge.index.in.segment sort.inmemory.size.inmb was (Author: pratyaksh): There are few properties which are not getting used anywhere - {color:#172b4d}spark.carbon.datamanagement.driver{color} {color:#172b4d}carbon.lease.recovery.retry.count{color} {color:#172b4d}carbon.lease.recovery.retry.interval{color} {color:#172b4d}{color:#080808}carbon.number.of.cores.while.altPartition{color}{color} Also there are few properties which are getting only for testing purpose- {color:#172b4d}{color:#067d17}carbon.merge.index.in.segment{color}{color} {color:#172b4d}{color:#067d17}sort.inmemory.size.inmb{color}{color} > Properties present in > https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java > which are not present in open source doc > --- > > Key: CARBONDATA-4240 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4240 > Project: CarbonData > Issue Type: Bug > Components: docs >Affects Versions: 2.2.0 > Environment: Open source docs >Reporter: Chetan Bhat >Priority: Minor > > Properties present in > https://github.com/apache/carbondata/blob/master/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java > which are not present in open source doc as mentioned below. These > properties need to be updated in open source doc. > carbon.storelocation > carbon.blocklet.size > carbon.properties.filepath > carbon.date.format > carbon.complex.delimiter.level.1 > carbon.complex.delimiter.level.2 > carbon.complex.delimiter.level.3 > carbon.complex.delimiter.level.4 > carbon.lock.class > carbon.local.dictionary.enable > carbon.local.dictionary.decoder.fallback > spark.deploy.zookeeper.url > carbon.data.file.version > spark.carbon.hive.schema.store > spark.carbon.datamanagement.driver > spark.carbon.sessionstate.classname > spark.carbon.sqlastbuilder.classname > carbon.lease.recovery.retry.count > carbon.lease.recovery.retry.interval > carbon.index.schema.storage > carbon.merge.index.in.segment > carbon.number.of.cores.while.altPartition > carbon.minor.compaction.size > enable.unsafe.columnpage > carbon.lucene.compression.mode > sort.inmemory.size.inmb > is.driver.instance > carbon.input.metrics.update.interval > carbon.use.bitset.pipe.line > is.internal.load.call > carbon.lucene.index.stop.words > carbon.load.dateformat.setlenient.enable > carbon.infilter.subquery.pushdown.enable > broadcast.record.size > carbon.indexserver.tempfolder.deletetime -- This message was sent by Atlassian Jira (v8.3.4#803005)