[jira] [Commented] (SPARK-26519) spark sql CHANGE COLUMN not working
[
https://issues.apache.org/jira/browse/SPARK-26519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733413#comment-16733413
]
Dongjoon Hyun commented on SPARK-26519:
---
+1 for [~hyukjin.kwon].
[~sumanGorantla]. `Operation not allowed` means this is a new feature to Spark
instead of `BUG`. We usually don't say like Hive/Impala has a bug when they
don't support Spark commands like `CACHE TABLE` syntax.
> spark sql CHANGE COLUMN not working
> --
>
> Key: SPARK-26519
> URL: https://issues.apache.org/jira/browse/SPARK-26519
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
> Environment: !image-2019-01-02-14-25-34-594.png!
>Reporter: suman gorantla
>Priority: Major
> Attachments: sparksql error.PNG
>
>
> Dear Team,
> with spark sql I am unable to change the newly added column() position after
> an existing column in the table (old_column) of a hive external table please
> see the screenshot as in below
> scala> sql("ALTER TABLE enterprisedatalakedev.tmptst ADD COLUMNs (new_col
> STRING)")
> res14: org.apache.spark.sql.DataFrame = []
> sql("ALTER TABLE enterprisedatalakedev.tmptst CHANGE COLUMN new_col new_col
> STRING AFTER old_col ")
> org.apache.spark.sql.catalyst.parser.ParseException:
> Operation not allowed: ALTER TABLE table [PARTITION partition_spec] CHANGE
> COLUMN ... FIRST | AFTER otherCol(line 1, pos 0)
> == SQL ==
> ALTER TABLE enterprisedatalakedev.tmptst CHANGE COLUMN new_col new_col
> STRING AFTER old_col
> ^^^
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.operationNotAllowed(ParserUtils.scala:39)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitChangeColumn$1.apply(SparkSqlParser.scala:934)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitChangeColumn$1.apply(SparkSqlParser.scala:928)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitChangeColumn(SparkSqlParser.scala:928)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitChangeColumn(SparkSqlParser.scala:55)
> at
> org.apache.spark.sql.catalyst.parser.SqlBaseParser$ChangeColumnContext.accept(SqlBaseParser.java:1485)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:71)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:71)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:70)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:69)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:68)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:97)
> at
> org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:68)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:623)
> ... 48 elided
> !image-2019-01-02-14-25-40-980.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26519) spark sql CHANGE COLUMN not working
[
https://issues.apache.org/jira/browse/SPARK-26519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732758#comment-16732758
]
Hyukjin Kwon commented on SPARK-26519:
--
It's a duplicate of SPARK-23890 or SPARK-24602.
> spark sql CHANGE COLUMN not working
> --
>
> Key: SPARK-26519
> URL: https://issues.apache.org/jira/browse/SPARK-26519
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
> Environment: !image-2019-01-02-14-25-34-594.png!
>Reporter: suman gorantla
>Priority: Major
> Attachments: sparksql error.PNG
>
>
> Dear Team,
> with spark sql I am unable to change the newly added column() position after
> an existing column in the table (old_column) of a hive external table please
> see the screenshot as in below
> scala> sql("ALTER TABLE enterprisedatalakedev.tmptst ADD COLUMNs (new_col
> STRING)")
> res14: org.apache.spark.sql.DataFrame = []
> sql("ALTER TABLE enterprisedatalakedev.tmptst CHANGE COLUMN new_col new_col
> STRING AFTER old_col ")
> org.apache.spark.sql.catalyst.parser.ParseException:
> Operation not allowed: ALTER TABLE table [PARTITION partition_spec] CHANGE
> COLUMN ... FIRST | AFTER otherCol(line 1, pos 0)
> == SQL ==
> ALTER TABLE enterprisedatalakedev.tmptst CHANGE COLUMN new_col new_col
> STRING AFTER old_col
> ^^^
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.operationNotAllowed(ParserUtils.scala:39)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitChangeColumn$1.apply(SparkSqlParser.scala:934)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitChangeColumn$1.apply(SparkSqlParser.scala:928)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitChangeColumn(SparkSqlParser.scala:928)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitChangeColumn(SparkSqlParser.scala:55)
> at
> org.apache.spark.sql.catalyst.parser.SqlBaseParser$ChangeColumnContext.accept(SqlBaseParser.java:1485)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:71)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:71)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:70)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:69)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:68)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:97)
> at
> org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:68)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:623)
> ... 48 elided
> !image-2019-01-02-14-25-40-980.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26519) spark sql CHANGE COLUMN not working
[
https://issues.apache.org/jira/browse/SPARK-26519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732714#comment-16732714
]
suman gorantla commented on SPARK-26519:
Dear Hyun,
Below command was successfully working in hive/ impala but failing in spark
sql and spark submit . Found the same error message in the logs as well .
This seems to be in correct behavior. Please let me know the reason
ALTER TABLE enterprisedatalakedev.tmptst CHANGE COLUMN new_col new_col
STRING AFTER old_col
--
సుమన్ గోరంట్ల
> spark sql CHANGE COLUMN not working
> --
>
> Key: SPARK-26519
> URL: https://issues.apache.org/jira/browse/SPARK-26519
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
> Environment: !image-2019-01-02-14-25-34-594.png!
>Reporter: suman gorantla
>Priority: Major
> Attachments: sparksql error.PNG
>
>
> Dear Team,
> with spark sql I am unable to change the newly added column() position after
> an existing column in the table (old_column) of a hive external table please
> see the screenshot as in below
> scala> sql("ALTER TABLE enterprisedatalakedev.tmptst ADD COLUMNs (new_col
> STRING)")
> res14: org.apache.spark.sql.DataFrame = []
> sql("ALTER TABLE enterprisedatalakedev.tmptst CHANGE COLUMN new_col new_col
> STRING AFTER old_col ")
> org.apache.spark.sql.catalyst.parser.ParseException:
> Operation not allowed: ALTER TABLE table [PARTITION partition_spec] CHANGE
> COLUMN ... FIRST | AFTER otherCol(line 1, pos 0)
> == SQL ==
> ALTER TABLE enterprisedatalakedev.tmptst CHANGE COLUMN new_col new_col
> STRING AFTER old_col
> ^^^
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.operationNotAllowed(ParserUtils.scala:39)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitChangeColumn$1.apply(SparkSqlParser.scala:934)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitChangeColumn$1.apply(SparkSqlParser.scala:928)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitChangeColumn(SparkSqlParser.scala:928)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitChangeColumn(SparkSqlParser.scala:55)
> at
> org.apache.spark.sql.catalyst.parser.SqlBaseParser$ChangeColumnContext.accept(SqlBaseParser.java:1485)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:71)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:71)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:70)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:69)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:68)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:97)
> at
> org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:68)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:623)
> ... 48 elided
> !image-2019-01-02-14-25-40-980.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26519) spark sql CHANGE COLUMN not working
[
https://issues.apache.org/jira/browse/SPARK-26519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732528#comment-16732528
]
Dongjoon Hyun commented on SPARK-26519:
---
Hi, [~sumanGorantla]. It's not a bug, isn't it? Please see the log.
{code}
Operation not allowed: ALTER TABLE table [PARTITION partition_spec] CHANGE
COLUMN ... FIRST | AFTER otherCol(line 1, pos 0)
{code}
> spark sql CHANGE COLUMN not working
> --
>
> Key: SPARK-26519
> URL: https://issues.apache.org/jira/browse/SPARK-26519
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell, Spark Submit
>Affects Versions: 2.1.0
> Environment: !image-2019-01-02-14-25-34-594.png!
>Reporter: suman gorantla
>Priority: Major
> Attachments: sparksql error.PNG
>
>
> Dear Team,
> with spark sql I am unable to change the newly added column() position after
> an existing column in the table (old_column) of a hive external table please
> see the screenshot as in below
> scala> sql("ALTER TABLE enterprisedatalakedev.tmptst ADD COLUMNs (new_col
> STRING)")
> res14: org.apache.spark.sql.DataFrame = []
> sql("ALTER TABLE enterprisedatalakedev.tmptst CHANGE COLUMN new_col new_col
> STRING AFTER old_col ")
> org.apache.spark.sql.catalyst.parser.ParseException:
> Operation not allowed: ALTER TABLE table [PARTITION partition_spec] CHANGE
> COLUMN ... FIRST | AFTER otherCol(line 1, pos 0)
> == SQL ==
> ALTER TABLE enterprisedatalakedev.tmptst CHANGE COLUMN new_col new_col
> STRING AFTER old_col
> ^^^
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.operationNotAllowed(ParserUtils.scala:39)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitChangeColumn$1.apply(SparkSqlParser.scala:934)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitChangeColumn$1.apply(SparkSqlParser.scala:928)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitChangeColumn(SparkSqlParser.scala:928)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitChangeColumn(SparkSqlParser.scala:55)
> at
> org.apache.spark.sql.catalyst.parser.SqlBaseParser$ChangeColumnContext.accept(SqlBaseParser.java:1485)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:71)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:71)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:70)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:69)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:68)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:97)
> at
> org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:68)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:623)
> ... 48 elided
> !image-2019-01-02-14-25-40-980.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
