philipse created SPARK-31710:
--------------------------------
Summary: result is the not the same when query and execute jobs
Key: SPARK-31710
URL: https://issues.apache.org/jira/browse/SPARK-31710
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.5
Environment: hdp:2.7.7
spark:2.4.5
Reporter: philipse
Hi Team
Steps to reproduce.
{code:java}
create table test(id bigint);
insert into test select 1586318188000;
create table test1(id bigint) partitioned by (year string);
insert overwrite table test1 partition(year) select 234,cast(id as TIMESTAMP)
from test;
{code}
let's check the result.
Case 1:
*select * from test1;*
234 | 52238-06-04 13:06:400.0
Case 2:
*select 234,cast(id as TIMESTAMP) from test;*
java.lang.IllegalArgumentException: Timestamp format must be yyyy-mm-dd
hh:mm:ss[.fffffffff]
at java.sql.Timestamp.valueOf(Timestamp.java:237)
at org.apache.hive.jdbc.HiveBaseResultSet.evaluate(HiveBaseResultSet.java:441)
at
org.apache.hive.jdbc.HiveBaseResultSet.getColumnValue(HiveBaseResultSet.java:421)
at org.apache.hive.jdbc.HiveBaseResultSet.getString(HiveBaseResultSet.java:530)
at org.apache.hive.beeline.Rows$Row.<init>(Rows.java:166)
at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:43)
at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1756)
at org.apache.hive.beeline.Commands.execute(Commands.java:826)
at org.apache.hive.beeline.Commands.sql(Commands.java:670)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:974)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:810)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:767)
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:480)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:463)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Error: Unrecognized column type:TIMESTAMP_TYPE (state=,code=0)
I try hive,it works well,and the convert is correct
Two questions:
q1:
if we forbid this convert,should we keep all cases the same?
q2:
if we allow the convert in some cases, should we decide the long length, for
the code seems to force to convert to ns with times*1000000 nomatter how long
the data is,if it convert to timestamp with incorrect length, we can raise the
error.
{code:java}
// // converting seconds to us
private[this] def longToTimestamp(t: Long): Long = t * 1000000L{code}
Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
