miaowang created SPARK-39578:
--------------------------------
Summary: The driver cannot parse the SQL statement and the job is
not executed
Key: SPARK-39578
URL: https://issues.apache.org/jira/browse/SPARK-39578
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.8
Reporter: miaowang
We use Spark's API to submit SQL execution tasks. Occasionally, SQL cannot be
executed and the task does not exit. Please give us some advice on how to solve
this problem.
Operation execution mode:
!image-2022-06-24-18-41-50-864.png!
Read SQL script and submit jobs in sequence。
Abnormal operation information：
{code:java}
//22/06/23 20:55:36 WARN HiveConf: HiveConf of name
hive.strict.checks.type.safety does not exist
22/06/23 20:55:36 WARN HiveConf: HiveConf of name
hive.strict.checks.cartesian.product does not exist
22/06/23 20:55:37 INFO metastore: Trying to connect to metastore with URI
thrift://bdp-datalake-hive-metastore-01-10-8-49-114:9083
22/06/23 20:55:37 INFO metastore: Connected to metastore.
22/06/23 20:55:38 INFO SessionState: Created local directory:
/hdfsdata/subdata10/yarn/nmcgroup/usercache/hive/appcache/application_1643023142753_3753253/container_e16_1643023142753_3753253_01_000001/tmp/nobody
22/06/23 20:55:38 INFO SessionState: Created local directory:
/hdfsdata/subdata10/yarn/nmcgroup/usercache/hive/appcache/application_1643023142753_3753253/container_e16_1643023142753_3753253_01_000001/tmp/944b1102-4564-4879-8305-4df390b26669_resources
22/06/23 20:55:38 INFO SessionState: Created HDFS directory:
/tmp/hive/hive/944b1102-4564-4879-8305-4df390b26669
22/06/23 20:55:38 INFO SessionState: Created local directory:
/hdfsdata/subdata10/yarn/nmcgroup/usercache/hive/appcache/application_1643023142753_3753253/container_e16_1643023142753_3753253_01_000001/tmp/nobody/944b1102-4564-4879-8305-4df390b26669
22/06/23 20:55:38 INFO SessionState: Created HDFS directory:
/tmp/hive/hive/944b1102-4564-4879-8305-4df390b26669/_tmp_space.db
22/06/23 20:55:38 INFO HiveClientImpl: Warehouse location for Hive client
(version 1.2.2) is /user/hive/warehouse
22/06/23 20:55:47 INFO ContextCleaner: Cleaned accumulator 7
22/06/23 20:55:47 INFO ContextCleaner: Cleaned accumulator 12
22/06/23 20:55:47 INFO ContextCleaner: Cleaned accumulator 14
22/06/23 20:55:47 INFO BlockManagerInfo: Removed broadcast_0_piece0 on
bdp-hdfs-36-10-8-33-65:39014 in memory (size: 31.1 KB, free: 3.0 GB)
22/06/23 20:55:48 INFO BlockManagerInfo: Removed broadcast_0_piece0 on
bdp-hdfs-16-10-8-33-45:33394 in memory (size: 31.1 KB, free: 8.4 GB)
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 10
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 2
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 18
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 24
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 19
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 6
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 5
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 17
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 25
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 13
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 29
22/06/23 20:55:48 INFO BlockManagerInfo: Removed broadcast_1_piece0 on
bdp-hdfs-36-10-8-33-65:39014 in memory (size: 4.0 KB, free: 3.0 GB)
22/06/23 20:55:48 INFO BlockManagerInfo: Removed broadcast_1_piece0 on
bdp-hdfs-16-10-8-33-45:33394 in memory (size: 4.0 KB, free: 8.4 GB)
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 28
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 4
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 16
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 21
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 20
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 9
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 23
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 26
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 0
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 3
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 11
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 15
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 27
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 8
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 1
22/06/23 20:55:48 INFO ContextCleaner: Cleaned accumulator 22
22/06/23 20:55:49 INFO InMemoryFileIndex: It took 66 ms to list leaf files for
1 paths.
22/06/23 20:55:49 INFO InMemoryFileIndex: It took 8 ms to list leaf files for 1
paths.
====正在执行====>drop table if exists
dc.cowell_omni_channel_jingli_day03_operator_info_org_20220622
====执行结束====>drop table if exists
dc.cowell_omni_channel_jingli_day03_operator_info_org_20220622
====正在执行====> create table
dc.cowell_omni_channel_jingli_day03_operator_info_org_20220622 as select
zone_new ,data_from ,ent_name ,business_id ,org_name ,org_no ,store_id
,chnl ,coalesce(day_paid,0) as day_paid {code}
Driver log output
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
