[jira] [Updated] (SPARK-24430) CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'.
[ https://issues.apache.org/jira/browse/SPARK-24430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Glushak updated SPARK-24430: -- Description: When I executes following SQL statement: {code:java} spark.sql('CREATE VIEW view_12 AS SELECT * FROM ( SELECT * FROM table1 UNION ALL SELECT * FROM table2 ) UT') {code} It successfully creates view in HIVE, which I can query via Apache Spark. However if I'm trying to query the same view directly via HIVE, I've got an error: {code:java} org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 6:6 Failed to recognize predicate 'UNION'. Failed rule: 'identifier' in subquery source {code} *Investigation* Under hood, spark generate following SQL statement for this view: {code:java} CREATE VIEW `view_12` AS SELECT * FROM (SELECT * FROM ( (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS gen_subquery_0) UNION ALL (SELECT * FROM (SELECT * FROM `db1`.`tabl2`) AS gen_subquery_1) ) AS UT ) AS UT {code} If I try to execute this statement in HIVE it fails with the same reason. The easiest way to fix it, is to remove parentheses from outer queries on lines 5 and 7. was: When I executes following SQL statement: {code:java} spark.sql('CREATE VIEW view_12 AS SELECT * FROM ( SELECT * FROM table1 UNION ALL SELECT * FROM table2 ) UT') {code} It successfully creates view in HIVE, which I can query via Apache Spark. However if I'm trying to query the same view directly via HIVE, I've got an error: {code:java} org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 6:6 Failed to recognize predicate 'UNION'. Failed rule: 'identifier' in subquery source {code} *Investigation* Under hood, spark generate following SQL statement for this view: {code:java} CREATE VIEW `view_12` AS SELECT * FROM (SELECT * FROM ( (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS gen_subquery_0) UNION ALL (SELECT * FROM (SELECT * FROM `db1`.`tabl2`) AS gen_subquery_1) ) AS UT ) AS UT {code} If I try to execute this statement in HIVE it fails with the same reason. The easiest way to fix it, is to unwrap outer queries on lines 5 and 7. > CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'. > > > Key: SPARK-24430 > URL: https://issues.apache.org/jira/browse/SPARK-24430 > Project: Spark > Issue Type: Request > Components: Spark Core, SQL >Affects Versions: 2.2.1 >Reporter: Volodymyr Glushak >Priority: Major > > When I executes following SQL statement: > {code:java} > spark.sql('CREATE VIEW view_12 AS > SELECT * FROM ( > SELECT * FROM table1 > UNION ALL > SELECT * FROM table2 > ) UT') > {code} > > It successfully creates view in HIVE, which I can query via Apache Spark. > However if I'm trying to query the same view directly via HIVE, I've got an > error: > {code:java} > org.apache.hive.service.cli.HiveSQLException: Error while compiling > statement: FAILED: ParseException line 6:6 Failed to recognize predicate > 'UNION'. > Failed rule: 'identifier' in subquery source > {code} > > *Investigation* > Under hood, spark generate following SQL statement for this view: > {code:java} > CREATE VIEW `view_12` AS > SELECT * > FROM (SELECT * FROM > ( > (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS > gen_subquery_0) > UNION ALL > (SELECT * FROM (SELECT * FROM `db1`.`tabl2`) AS > gen_subquery_1) > ) AS UT > ) AS UT > {code} > If I try to execute this statement in HIVE it fails with the same reason. > The easiest way to fix it, is to remove parentheses from outer queries on > lines 5 and 7. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24430) CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'.
[ https://issues.apache.org/jira/browse/SPARK-24430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Glushak updated SPARK-24430: -- Description: When I executes following SQL statement: {code:java} spark.sql('CREATE VIEW view_12 AS SELECT * FROM ( SELECT * FROM table1 UNION ALL SELECT * FROM table2 ) UT') {code} It successfully creates view in HIVE, which I can query via Apache Spark. However if I'm trying to query the same view directly via HIVE, I've got an error: {code:java} org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 6:6 Failed to recognize predicate 'UNION'. Failed rule: 'identifier' in subquery source {code} *Investigation* Under hood, spark generate following SQL statement for this view: {code:java} CREATE VIEW `view_12` AS SELECT * FROM (SELECT * FROM ( (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS gen_subquery_0) UNION ALL (SELECT * FROM (SELECT * FROM `db1`.`tabl2`) AS gen_subquery_1) ) AS UT ) AS UT {code} If I try to execute this statement in HIVE it fails with the same reason. The easiest way to fix it, is to unwrap outer queries on lines 5 and 7. was: When I executes following SQL statement: {code} spark.sql('CREATE VIEW view_12 AS SELECT * FROM ( SELECT * FROM table1 UNION ALL SELECT * FROM table2 ) UT') {code} It successfully creates view in HIVE, which I can query via Apache Spark. However if I'm trying to query the same view directly via HIVE, I've got an error: {code} org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 6:6 Failed to recognize predicate 'UNION'. Failed rule: 'identifier' in subquery source {code} *Investigation* Under hood, spark generate following SQL statement for this view: {code} CREATE VIEW `view_12` AS SELECT * FROM (SELECT * FROM ( (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS gen_subquery_0) UNION ALL (SELECT * FROM (SELECT * FROM `db1`.`tabl2`) AS gen_subquery_1) ) AS UT ) AS UT {code} If I try to executes this statement in HIVE it fails with the same reason. The easiest way to fix it, is to unwrap outer queries on lines 5 and 7. > CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'. > > > Key: SPARK-24430 > URL: https://issues.apache.org/jira/browse/SPARK-24430 > Project: Spark > Issue Type: Request > Components: Spark Core, SQL >Affects Versions: 2.2.1 >Reporter: Volodymyr Glushak >Priority: Major > > When I executes following SQL statement: > {code:java} > spark.sql('CREATE VIEW view_12 AS > SELECT * FROM ( > SELECT * FROM table1 > UNION ALL > SELECT * FROM table2 > ) UT') > {code} > > It successfully creates view in HIVE, which I can query via Apache Spark. > However if I'm trying to query the same view directly via HIVE, I've got an > error: > {code:java} > org.apache.hive.service.cli.HiveSQLException: Error while compiling > statement: FAILED: ParseException line 6:6 Failed to recognize predicate > 'UNION'. > Failed rule: 'identifier' in subquery source > {code} > > *Investigation* > Under hood, spark generate following SQL statement for this view: > {code:java} > CREATE VIEW `view_12` AS > SELECT * > FROM (SELECT * FROM > ( > (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS > gen_subquery_0) > UNION ALL > (SELECT * FROM (SELECT * FROM `db1`.`tabl2`) AS > gen_subquery_1) > ) AS UT > ) AS UT > {code} > If I try to execute this statement in HIVE it fails with the same reason. > The easiest way to fix it, is to unwrap outer queries on lines 5 and 7. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24430) CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'.
[ https://issues.apache.org/jira/browse/SPARK-24430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495374#comment-16495374 ] Volodymyr Glushak commented on SPARK-24430: --- I wonder, if that is legal code, and this request should be moved to HIVE JIRA. > CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'. > > > Key: SPARK-24430 > URL: https://issues.apache.org/jira/browse/SPARK-24430 > Project: Spark > Issue Type: Request > Components: Spark Core, SQL >Affects Versions: 2.2.1 >Reporter: Volodymyr Glushak >Priority: Major > > When I executes following SQL statement: > {code} > spark.sql('CREATE VIEW view_12 AS > SELECT * FROM ( > SELECT * FROM table1 > UNION ALL > SELECT * FROM table2 > ) UT') > {code} > > It successfully creates view in HIVE, which I can query via Apache Spark. > However if I'm trying to query the same view directly via HIVE, I've got an > error: > {code} > org.apache.hive.service.cli.HiveSQLException: Error while compiling > statement: FAILED: ParseException line 6:6 Failed to recognize predicate > 'UNION'. > Failed rule: 'identifier' in subquery source > {code} > > *Investigation* > Under hood, spark generate following SQL statement for this view: > {code} > CREATE VIEW `view_12` AS > SELECT * > FROM (SELECT * FROM > ( > (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS > gen_subquery_0) > UNION ALL > (SELECT * FROM (SELECT * FROM `db1`.`tabl2`) AS > gen_subquery_1) > ) AS UT > ) AS UT > {code} > If I try to executes this statement in HIVE it fails with the same reason. > The easiest way to fix it, is to unwrap outer queries on lines 5 and 7. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24430) CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'.
Volodymyr Glushak created SPARK-24430: - Summary: CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'. Key: SPARK-24430 URL: https://issues.apache.org/jira/browse/SPARK-24430 Project: Spark Issue Type: Request Components: Spark Core, SQL Affects Versions: 2.2.1 Reporter: Volodymyr Glushak When I executes following SQL statement: {code} spark.sql('CREATE VIEW view_12 AS SELECT * FROM ( SELECT * FROM table1 UNION ALL SELECT * FROM table2 ) UT') {code} It successfully creates view in HIVE, which I can query via Apache Spark. However if I'm trying to query the same view directly via HIVE, I've got an error: {code} org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 6:6 Failed to recognize predicate 'UNION'. Failed rule: 'identifier' in subquery source {code} *Investigation* Under hood, spark generate following SQL statement for this view: {code} CREATE VIEW `view_12` AS SELECT * FROM (SELECT * FROM ( (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS gen_subquery_0) UNION ALL (SELECT * FROM (SELECT * FROM `db1`.`tabl2`) AS gen_subquery_1) ) AS UT ) AS UT {code} If I try to executes this statement in HIVE it fails with the same reason. The easiest way to fix it, is to unwrap outer queries on lines 5 and 7. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-23993) Support DESC FORMATTED table_name column_name
[ https://issues.apache.org/jira/browse/SPARK-23993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Glushak resolved SPARK-23993. --- Resolution: Duplicate Fix Version/s: 2.3.0 Duplicate of SPARK-17642. Thanks [~ksunitha] for investigation. > Support DESC FORMATTED table_name column_name > - > > Key: SPARK-23993 > URL: https://issues.apache.org/jira/browse/SPARK-23993 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.1.2 >Reporter: Volodymyr Glushak >Priority: Major > Fix For: 2.3.0 > > > Hive and Spark both supports: > {code} > DESC FORMATTED table_name{code} > which gives table metadata. > If you want to get metadata for particular column in hive you can execute: > {code} > DESC FORMATTED table_name column_name{code} > Thos is not supported in Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23993) Support DESC FORMATTED table_name column_name
Volodymyr Glushak created SPARK-23993: - Summary: Support DESC FORMATTED table_name column_name Key: SPARK-23993 URL: https://issues.apache.org/jira/browse/SPARK-23993 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.1.2 Reporter: Volodymyr Glushak Hive and Spark both supports: {code} DESC FORMATTED table_name{code} which gives table metadata. If you want to get metadata for particular column in hive you can execute: {code} DESC FORMATTED table_name column_name{code} Thos is not supported in Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-15276) CREATE TABLE with LOCATION should imply EXTERNAL
[ https://issues.apache.org/jira/browse/SPARK-15276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415825#comment-16415825 ] Volodymyr Glushak edited comment on SPARK-15276 at 3/27/18 3:55 PM: [~andrewor14], HIVE does not "externalise" table if LOCATION specified. (I reckon Impala neither) Why does Apache Spark introduce different behaviour? was (Author: rumoku): [~andrewor14], HIVE does not "externalise" table if LOCATION specified. Why does Apache Spark introduce different behaviour? > CREATE TABLE with LOCATION should imply EXTERNAL > > > Key: SPARK-15276 > URL: https://issues.apache.org/jira/browse/SPARK-15276 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.0 >Reporter: Andrew Or >Assignee: Andrew Or >Priority: Major > Fix For: 2.0.0 > > > If the user runs `CREATE TABLE some_table ... LOCATION /some/path`, then this > will still be a managed table even though the table's data is stored at > /some/path. The problem is that when we drop the table we'll also delete the > data /some/path. This could cause problems if /some/path contains existing > data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-15276) CREATE TABLE with LOCATION should imply EXTERNAL
[ https://issues.apache.org/jira/browse/SPARK-15276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415825#comment-16415825 ] Volodymyr Glushak edited comment on SPARK-15276 at 3/27/18 3:51 PM: [~andrewor14], HIVE does not "externalise" table if LOCATION specified. Why does Apache Spark introduce different behaviour? was (Author: rumoku): [~andrewor14], HIVE is not adding EXTERNAL keyword if LOCATION specified. Why does Apache Spark introduce different behaviour? > CREATE TABLE with LOCATION should imply EXTERNAL > > > Key: SPARK-15276 > URL: https://issues.apache.org/jira/browse/SPARK-15276 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.0 >Reporter: Andrew Or >Assignee: Andrew Or >Priority: Major > Fix For: 2.0.0 > > > If the user runs `CREATE TABLE some_table ... LOCATION /some/path`, then this > will still be a managed table even though the table's data is stored at > /some/path. The problem is that when we drop the table we'll also delete the > data /some/path. This could cause problems if /some/path contains existing > data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15276) CREATE TABLE with LOCATION should imply EXTERNAL
[ https://issues.apache.org/jira/browse/SPARK-15276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415825#comment-16415825 ] Volodymyr Glushak commented on SPARK-15276: --- [~andrewor14], HIVE is not adding EXTERNAL keyword if LOCATION specified. Why does Apache Spark introduce different behaviour? > CREATE TABLE with LOCATION should imply EXTERNAL > > > Key: SPARK-15276 > URL: https://issues.apache.org/jira/browse/SPARK-15276 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.0 >Reporter: Andrew Or >Assignee: Andrew Or >Priority: Major > Fix For: 2.0.0 > > > If the user runs `CREATE TABLE some_table ... LOCATION /some/path`, then this > will still be a managed table even though the table's data is stored at > /some/path. The problem is that when we drop the table we'll also delete the > data /some/path. This could cause problems if /some/path contains existing > data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org