lklong commented on issue #9447:
URL:
https://github.com/apache/dolphinscheduler/issues/9447#issuecomment-1095342931
full log
[INFO] 2021-12-22 05:22:08.373 - [taskAppId=TASK-59-979115-175415]:[92] -
shell task params {"rawScript":"sh
/data/home/luohao/wam/rt_insert_t_user_login.sh","localParams":[],"resourceList":[]}
[INFO] 2021-12-22 05:22:08.374 - [taskAppId=TASK-59-979115-175415]:[150] -
raw script : sh /data/home/luohao/wam/rt_insert_t_user_login.sh
[INFO] 2021-12-22 05:22:08.374 - [taskAppId=TASK-59-979115-175415]:[151] -
task dir : /data/software/dolphinscheduler-dest/exec/process/16/59/979115/175415
[INFO] 2021-12-22 05:22:08.374 - [taskAppId=TASK-59-979115-175415]:[99] -
tenantCode user:hadoop, task dir:59_979115_175415
[INFO] 2021-12-22 05:22:08.374 - [taskAppId=TASK-59-979115-175415]:[103] -
create command
file:/data/software/dolphinscheduler-dest/exec/process/16/59/979115/175415/59_979115_175415.command
[INFO] 2021-12-22 05:22:08.374 - [taskAppId=TASK-59-979115-175415]:[116] -
command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /data/software/dolphinscheduler-dest/conf/env/.dolphinscheduler_env.sh
/data/software/dolphinscheduler-dest/exec/process/16/59/979115/175415/59_979115_175415_node.sh
[INFO] 2021-12-22 05:22:08.375 - [taskAppId=TASK-59-979115-175415]:[329] -
task run command:
sudo -u hadoop sh
/data/software/dolphinscheduler-dest/exec/process/16/59/979115/175415/59_979115_175415.command
[INFO] 2021-12-22 05:22:08.379 - [taskAppId=TASK-59-979115-175415]:[158] -
process start, process id is: 382105
[INFO] 2021-12-22 05:22:10.210 - [taskAppId=TASK-59-979115-175415]:[106] -
-> minute:52
day_hour_start:22-00
day_hour_end:22-12
store_day:2021-12-22
cmd:add jar /data/home/luohao/wam/hive-hcatalog-core.jar;
insert overwrite table rt_wam.t_user_login partition
(store_day='2021-12-22')
select
'',tt.userid,tt.gameCode,'',tt.appplatform,tt.gpid,tt.thirdplate,tt.logindate,tt.logindate,
tt.total,tt.useripaddress,tt.loginip,tt.advertising_id,tt.advertising_id,'','',''
from
(select
t.userid,t.gameCode,t.gpid,t.appplatform,t.thirdplate,t.logindate,
t.useripaddress,t.loginip,t.advertising_id,
row_number() over (partition by t.userid,t.gamecode order by
t.logindate) as rank,
row_number() over (partition by t.userid,t.gamecode order by
t.logindate desc) as total
from rt_wam.rt_wam_login_log t
where t.store_day_hour between '22-00' and '22-12') tt
where tt.rank=1;
which: no hbase in
(/usr/local/service/hadoop/bin:/usr/local/service/spark/bin:/usr/local/service/spark/bin:/usr/local/python3:/usr/local/jdk/bin:/usr/local/service/hive/bin:/sbin:/bin:/usr/sbin:/usr/bin:/bin:/sbin:/bin:/usr/sbin:/usr/bin)
[INFO] 2021-12-22 05:22:11.270 - [taskAppId=TASK-59-979115-175415]:[106] -
-> SLF4J: Class path contains multiple SLF4J bindings.
[INFO] 2021-12-22 05:22:13.554 - [taskAppId=TASK-59-979115-175415]:[106] -
-> SLF4J: Found binding in
[jar:file:/usr/local/service/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/service/hive/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/service/tez/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/service/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = b1f9812d-2bb0-4617-9028-b3b23015a1a2
[INFO] 2021-12-22 05:22:15.318 - [taskAppId=TASK-59-979115-175415]:[106] -
->
Logging initialized using configuration in
file:/usr/local/service/hive/conf/hive-log4j2.properties Async: true
Hive Session ID = 6284bfc2-644f-4bac-bf31-d9a9ba9b1206
[INFO] 2021-12-22 05:22:18.439 - [taskAppId=TASK-59-979115-175415]:[106] -
-> Added [/data/home/luohao/wam/hive-hcatalog-core.jar] to class path
Added resources: [/data/home/luohao/wam/hive-hcatalog-core.jar]
Query ID = hadoop_20211222052215_accff17c-4002-478f-8f72-a486315d5a8e
[INFO] 2021-12-22 05:22:20.243 - [taskAppId=TASK-59-979115-175415]:[106] -
-> Total jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1639971392921_12251, Tracking URL =
http://10.168.0.15:5004/proxy/application_1639971392921_12251/
[INFO] 2021-12-22 05:22:25.371 - [taskAppId=TASK-59-979115-175415]:[106] -
-> Kill Command = /usr/local/service/hadoop/bin/mapred job -kill
job_1639971392921_12251
Hadoop job information for Stage-1: number of mappers: 6; number of
reducers: 1
[INFO] 2021-12-22 05:22:40.721 - [taskAppId=TASK-59-979115-175415]:[106] -
-> 2021-12-22 05:22:25,403 Stage-1 map = 0%, reduce = 0%
2021-12-22 05:22:40,721 Stage-1 map = 100%, reduce = 100%
[INFO] 2021-12-22 05:22:41.761 - [taskAppId=TASK-59-979115-175415]:[106] -
-> Ended Job = job_1639971392921_12251 with errors
[INFO] 2021-12-22 05:22:41.800 - [taskAppId=TASK-59-979115-175415]:[106] -
-> Error during job, obtaining debugging information...
Examining task ID: task_1639971392921_12251_m_000000 (and more) from
job job_1639971392921_12251
Examining task ID: task_1639971392921_12251_m_000001 (and more) from
job job_1639971392921_12251
Task with the most failures(4):
-----
Task ID:
task_1639971392921_12251_m_000005
URL:
http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1639971392921_12251&tipid=task_1639971392921_12251_m_000005
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:113)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:157)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:462)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:110)
... 10 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:113)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:157)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
at
org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 15 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:110)
... 19 more
Caused by: java.lang.RuntimeException: Map operator initialization
failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:137)
... 24 more
Caused by: java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/serde2/SerDe
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
[INFO] 2021-12-22 05:22:42.227 - [taskAppId=TASK-59-979115-175415]:[106] -
-> at
org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2575)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2540)
at
org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:182)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator$RowDeserializePartitionContext.init(VectorMapOperator.java:359)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.createAndInitPartitionContext(VectorMapOperator.java:423)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.internalSetChildren(VectorMapOperator.java:570)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.setChildren(VectorMapOperator.java:482)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:106)
... 24 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.serde2.SerDe
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 46 more
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 6 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
input params:2
wahaha
[INFO] 2021-12-22 05:22:42.227 - [taskAppId=TASK-59-979115-175415]:[168] -
process has exited, work
dir:/data/software/dolphinscheduler-dest/exec/process/16/59/979115/175415,
pid:382105 ,exitStatusCode:2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
