[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136550#comment-17136550 ] Lin Gang Deng commented on SPARK-31955: --- [~dengzh], the version of beeline in spark2.4 and before is 1.2.1, spark3.0 upgrade the beeline to 2.3.7. For spark，The issue was fixed in spark3.0. > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Major > Fix For: 3.0.0 > > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136535#comment-17136535 ] Zhihua Deng commented on SPARK-31955: - The issue seems have been fixed in [HIVE-10541|https://issues.apache.org/jira/browse/HIVE-10541]. > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Major > Fix For: 3.0.0 > > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[
https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136527#comment-17136527
]
Yuming Wang commented on SPARK-31955:
-
The issue fixed by upgrading the beeline to 2.3.7. How to reproduce this issue:
*Prepare table*
{code:sql}
create table test_beeline using parquet as select id from range(5);
{code}
*Prepare SQL*
{code:sql}
echo -en "select * from test_beeline\n where id=2;" >> test.sql
{code}
*Spark 2.4*:
{noformat}
[root@spark-3267648 spark-2.4.4-bin-hadoop2.7]# bin/beeline -u
"jdbc:hive2://localhost:1" -f /root/spark-3.0.0-bin-hadoop3.2/test.sql
Connecting to jdbc:hive2://localhost:1
log4j:WARN No appenders could be found for logger (org.apache.hive.jdbc.Utils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Connected to: Spark SQL (version 3.0.0)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:1> select * from test_beeline
0: jdbc:hive2://localhost:1> where id=2;+-+--+
| id |
+-+--+
| 0 |
| 2 |
| 1 |
| 3 |
| 4 |
+-+--+
5 rows selected (5.622 seconds)
0: jdbc:hive2://localhost:1> where id=2;
Closing: 0: jdbc:hive2://localhost:1
{noformat}
*Spark 3.0*:
{noformat}
[root@spark-3267648 spark-3.0.0-bin-hadoop3.2]# bin/beeline -u
"jdbc:hive2://localhost:1" -f test.sql
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Connecting to jdbc:hive2://localhost:1
Connected to: Spark SQL (version 3.0.0)
Driver: Hive JDBC (version 2.3.7)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:1> select * from test_beeline
. . . . . . . . . . . . . . . .> where id=2;
+-+
| id |
+-+
| 2 |
+-+
1 row selected (7.749 seconds)
0: jdbc:hive2://localhost:1> Closing: 0: jdbc:hive2://localhost:1
{noformat}
> Beeline discard the last line of the sql file when submited to thriftserver
> via beeline
>
>
> Key: SPARK-31955
> URL: https://issues.apache.org/jira/browse/SPARK-31955
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.4
>Reporter: Lin Gang Deng
>Priority: Major
> Fix For: 3.0.0
>
>
> I submitted a sql file on beeline and the result returned is wrong. After
> many tests, it was found that the sql executed by Spark would discard the
> last line.This should be beeline's bug parsing sql file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133898#comment-17133898 ] Lin Gang Deng commented on SPARK-31955: --- [~hyukjin.kwon] , the difference between me and [~younggyuchun] 's sql file is that there is no EOL at the end of my script. My example is the exact reproducer. > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Major > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133894#comment-17133894 ] Yuming Wang commented on SPARK-31955: - It seems this is a Hive issue. Maybe we should fix it on the Hive side. > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Major > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133884#comment-17133884 ] Hyukjin Kwon commented on SPARK-31955: -- I am going to leave it resolved until enough information is provided to analyze further, for JIRA management purpose. > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Major > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133883#comment-17133883 ] Hyukjin Kwon commented on SPARK-31955: -- [~denglg] Please show the __exact__ reproducer. From reading [~younggyuchun], it doesn't look clear what issue you mean. Also please check if the behaviours are consistent with beeline in Hive. If it also exists in Hive, this isn't a Spark specific issue. > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Major > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133822#comment-17133822 ] Lin Gang Deng commented on SPARK-31955: --- [~dongjoon] As you said, EOL is the key to the problem. Sometimes, there is no newline character when SQL is submitted, or the newline character is removed by the third-party component. Maybe, spark or beeline should correctly parse SQL, whether or not there is EOL. Otherwise, users will be bothered by the wrong result. > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Major > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133521#comment-17133521 ] Dongjoon Hyun commented on SPARK-31955: --- [~denglg]. Please add a new line character at the end of the line. Your script doesn't have it. Thank you for investigating, [~younggyuchun]. > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Major > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[
https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133298#comment-17133298
]
YoungGyu Chun commented on SPARK-31955:
---
[~denglg]
I cannot reproduce this locally. :
Whole data:
{code:java}
0: jdbc:hive2://localhost:1> select * from info_dev.beeline_test;
+--++--+
| beeline_test.id | beeline_test.name |
+--++--+
| 1| aaa|
| 2| bbb|
| 3| ccc|
| 1| aaa|
+--++--+
4 rows selected (0.239 seconds)
0: jdbc:hive2://localhost:1>
{code}
test2.sql:
{code:java}
jun562@CHUNYLT:~/spark-2.4.4-bin-hadoop2.7/bin$ cat test2.sql
select * from info_dev.beeline_test where name='bbb'
jun562@CHUNYLT:~/spark-2.4.4-bin-hadoop2.7/bin$
{code}
Execute a test2.sql file on Beeline by running a "run" command:
{code:java}
Connected to: Apache Hive (version 1.2.1)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:1> !run
/home/jun562/apache-hive-1.2.1-bin/bin/test2.sql
>>> select * from info_dev.beeline_test where name='bbb';
+--++--+
| beeline_test.id | beeline_test.name |
+--++--+
| 2| bbb|
+--++--+
1 row selected (0.406 seconds)
0: jdbc:hive2://localhost:1>
{code}
Execute SQL on Beeline:
{code:java}
0: jdbc:hive2://localhost:1> select * from info_dev.beeline_test where
name='bbb';
+--++--+
| beeline_test.id | beeline_test.name |
+--++--+
| 2| bbb|
+--++--+
1 row selected (0.233 seconds)
0: jdbc:hive2://localhost:1>
{code}
cc [~dongjoon] [~hyukjin.kwon] [~srowen]
> Beeline discard the last line of the sql file when submited to thriftserver
> via beeline
>
>
> Key: SPARK-31955
> URL: https://issues.apache.org/jira/browse/SPARK-31955
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.4
>Reporter: Lin Gang Deng
>Priority: Major
>
> I submitted a sql file on beeline and the result returned is wrong. After
> many tests, it was found that the sql executed by Spark would discard the
> last line.This should be beeline's bug parsing sql file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133226#comment-17133226 ] YoungGyu Chun commented on SPARK-31955: --- hmm... I will look into this > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Major > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[
https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132812#comment-17132812
]
Lin Gang Deng commented on SPARK-31955:
---
[~younggyuchun] No results.
{code:java}
// code placeholder
[test@192.168.0.1 denglg]$ cat -A test3.sql
select * from info_dev.beeline_test where name='bbb'[test@192.168.0.1 denglg]$
{code}
{code:java}
// code placeholder
0: jdbc:hive2://spark-sql.hadoo> select * from info_dev.beeline_test where
name='bbb'
Closing: 0:
jdbc:hive2://spark-sql.hadoop.srv:1/;principal=xxx?mapreduce.job.queuename=xxx
[test@192.168.0.1 denglg]$
{code}
> Beeline discard the last line of the sql file when submited to thriftserver
> via beeline
>
>
> Key: SPARK-31955
> URL: https://issues.apache.org/jira/browse/SPARK-31955
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.4
>Reporter: Lin Gang Deng
>Priority: Blocker
>
> I submitted a sql file on beeline and the result returned is wrong. After
> many tests, it was found that the sql executed by Spark would discard the
> last line.This should be beeline's bug parsing sql file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132566#comment-17132566 ] YoungGyu Chun commented on SPARK-31955: --- Try this rather than add a space between select statement and where statement: select * from info_dev.beeline_test where name='bbb' > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Blocker > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17132549#comment-17132549 ] YoungGyu Chun commented on SPARK-31955: --- Thanks, I will work on this > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Blocker > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[
https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130795#comment-17130795
]
Lin Gang Deng commented on SPARK-31955:
---
{code:java}
// code placeholder
0: jdbc:hive2://hadoop.spark-sql.hadoo> select * from info_dev.beeline_test;
+-+---+--+
| id | name |
+-+---+--+
| 3 | ccc |
| 2 | bbb |
| 1 | aaa |
+-+---+--+
3 rows selected (1.402 seconds)
{code}
Then sql as bellows,
{code:java}
// code placeholder
[test@192.168.0.1 denglg]$ cat -A test2.sql
select * from info_dev.beeline_test$
where name='bbb';[test@192.168.0.1 denglg]$
{code}
Result as bellows,
{code:java}
// code placeholder
0: jdbc:hive2://spark-sql.hadoo> select * from info_dev.beeline_test
0: jdbc:hive2://spark-sql.hadoo> where name='bbb';+-+---+--+
| id | name |
+-+---+--+
| 3 | ccc |
| 2 | bbb |
| 1 | aaa |
+-+---+--+
3 rows selected (1.594 seconds)
{code}
As you can see，it got wrong result.
> Beeline discard the last line of the sql file when submited to thriftserver
> via beeline
>
>
> Key: SPARK-31955
> URL: https://issues.apache.org/jira/browse/SPARK-31955
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.4
>Reporter: Lin Gang Deng
>Priority: Blocker
>
> I submitted a sql file on beeline and the result returned is wrong. After
> many tests, it was found that the sql executed by Spark would discard the
> last line.This should be beeline's bug parsing sql file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31955) Beeline discard the last line of the sql file when submited to thriftserver via beeline
[ https://issues.apache.org/jira/browse/SPARK-31955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130750#comment-17130750 ] YoungGyu Chun commented on SPARK-31955: --- [~denglg] can you add a screenshot that discards the last line and the sql file you are testing? > Beeline discard the last line of the sql file when submited to thriftserver > via beeline > > > Key: SPARK-31955 > URL: https://issues.apache.org/jira/browse/SPARK-31955 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4 >Reporter: Lin Gang Deng >Priority: Blocker > > I submitted a sql file on beeline and the result returned is wrong. After > many tests, it was found that the sql executed by Spark would discard the > last line.This should be beeline's bug parsing sql file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
