[
https://issues.apache.org/jira/browse/HIVE-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146880#comment-16146880
]
Zoltan Haindrich commented on HIVE-15467:
-----------------------------------------
the same issue happened again for me...I've looked into it a bit more...and it
seems like there is some issues with the nodemanagers...they report that the
local dirs are bad
resourcemanager ui shows this info:
{code}
NodeHealthReport 4/4 local-dirs are bad:
/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-localDir-nm-1_0,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-localDir-nm-1_2,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-localDir-nm-1_1,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-localDir-nm-1_3;
4/4 log-dirs are bad:
/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-logDir-nm-1_1,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-logDir-nm-1_0,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-logDir-nm-1_3,/home/kirk/hw/asf-hive/itests/qtest/target/hive/hive-logDir-nm-1_2
{code}
nodemanagers are in an unworkable state...and because of this the tez AM stucks
in initializing state
the resourcemanager ui seems to be not available...and I've not found any other
usefull info...
I've switched to a dfferent cli driver which didn't get stuck...
> escape1.q hangs in TestMiniLlapLocalCliDriver
> ---------------------------------------------
>
> Key: HIVE-15467
> URL: https://issues.apache.org/jira/browse/HIVE-15467
> Project: Hive
> Issue Type: Bug
> Reporter: Pengcheng Xiong
> Assignee: Prasanth Jayachandran
>
> here is part of the log before it hangs
> {code}
> 2016-12-19T15:21:05,779 INFO [LlapScheduler]
> tezplugins.LlapTaskSchedulerService: ScheduleResult for Task:
> TaskInfo{task=attempt_1482189645956_0001_33_00_000000_1, priority=1,
> startTime=0, containerId=null, assignedNode=, uniqueId=54,
> localityDelayTimeout=0} = DELAYED_RESOURCES
> 2016-12-19T15:21:05,779 DEBUG [LlapScheduler]
> tezplugins.LlapTaskSchedulerService: Attempting to preempt on any host for
> task=attempt_1482189645956_0001_33_00_000000_1, pendingPreemptions=0
> 2016-12-19T15:21:05,779 INFO [LlapScheduler]
> tezplugins.LlapTaskSchedulerService: Preempting for
> task=attempt_1482189645956_0001_33_00_000000_1 on any available host
> 2016-12-19T15:21:05,779 DEBUG [LlapScheduler]
> tezplugins.LlapTaskSchedulerService: Unable to schedule all requests at
> priority=1. Skipping subsequent priority levels
> 2016-12-19T15:21:07,953 DEBUG [AMReporterQueueDrainer] impl.AMReporter:
> Removing am localhost:61788 with last associated dag
> QueryIdentifier{appIdentifier='application_1482189645956_0001',
> dagIdentifier=33} from heartbeat with taskCount=0, amFailed=false
> 2016-12-19T15:21:08,634 INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main]
> SessionState: Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:11,700 INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main]
> SessionState: Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:14,755 INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main]
> SessionState: Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:17,814 INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main]
> SessionState: Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:20,871 INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main]
> SessionState: Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:23,931 INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main]
> SessionState: Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:26,977 INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main]
> SessionState: Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:30,027 INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main]
> SessionState: Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:33,078 INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main]
> SessionState: Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:36,133 INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main]
> SessionState: Map 1: 0(+1,-1)/1
> 2016-12-19T15:21:39,179 INFO [86edca30-bf12-42f8-90cd-a9fbdfbcb546 main]
> SessionState: Map 1: 0(+1,-1)/1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
