Mostafa Mokhtar created TEZ-2263:
------------------------------------
Summary: Tez : Don't try to recover from a failed commit
Key: TEZ-2263
URL: https://issues.apache.org/jira/browse/TEZ-2263
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Mostafa Mokhtar
Fix For: 0.5.0
Commit fails then Tez tries to recover which fails again.
{code}
5499174247-2015-04-01 16:23:20,600 INFO [main] app.RecoveryParser: Found
summary file in attempt directory,
summaryFile=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1/summary,
path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1
5499174696-2015-04-01 16:23:20,600 INFO [main] app.RecoveryParser: Using
hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1
for recovering data from previous attempt
5499174963-2015-04-01 16:23:20,690 INFO [main] app.RecoveryParser: Parsing
summary file,
path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1/summary,
len=4024, lastModTime=1427919788998
5499175254-2015-04-01 16:23:20,786 INFO [main] app.RecoveryParser: Reached end
of summary stream
5499175340-2015-04-01 16:23:21,087 INFO [main] app.RecoveryParser: Checking if
DAG is in recoverable state, dagId=dag_1426707664723_1086_1
5499175468-2015-04-01 16:23:21,088 WARN [main] app.RecoveryParser: Found last
inProgress DAG but not recoverable: dagId=dag_1426707664723_1086_1,
dagCompleted=false
5499175622-2015-04-01 16:23:21,088 INFO [main] app.RecoveryParser: Trying to
recover dag from recovery file, dagId=dag_1426707664723_1086_1,
dataDir=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1,
intoCurrentDir=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2
5499176102-2015-04-01 16:23:21,091 INFO [main] app.RecoveryParser: Copying DAG
data into Current Attempt directory,
filePath=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2/dag_1426707664723_1086_1.recovery
5499176413-2015-04-01 16:23:21,211 INFO [main] app.RecoveryParser: Recovering
from event, eventType=DAG_SUBMITTED, event=dagID=dag_1426707664723_1086_1,
submitTime=1427917169723
5499176580-2015-04-01 16:23:21,309 INFO [main] app.DAGAppMaster: Generating DAG
graphviz file, dagId=dag_1426707664723_1086_1,
filePath=/grid/0/cluster/yarn/log/application_1426707664723_1086/container_1426707664723_1086_02_000001/dag_1426707664723_1086_1.dot
5499176829-2015-04-01 16:23:21,347 INFO [main] app.DAGAppMaster: Writing DAG
plan to:
/grid/0/cluster/yarn/log/application_1426707664723_1086/container_1426707664723_1086_02_000001/dag_1426707664723_1086_1-tez-dag.pb.txt
5499177039-2015-04-01 16:23:22,576 INFO [main] app.RecoveryParser: Finished
copying data from previous attempt into current attempt
5499177160-2015-04-01 16:23:22,576 INFO [main] app.RecoveryParser: Trying to
create data recovered flag file,
filePath=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2/dataRecovered
5499177445-2015-04-01 16:23:22,601 INFO [main] app.DAGAppMaster: In Session
mode. Waiting for DAG over RPC
5499177541-2015-04-01 16:23:22,601 INFO [main] app.DAGAppMaster: Found previous
DAG in completed or non-recoverable state, dagId=dag_1426707664723_1086_1,
isCompleted=false, isNonRecoverable=true, state=null, failureReason=DAG Commit
was in progress, not recoverable, dagId=dag_1426707664723_1086_1
5499177829-2015-04-01 16:23:22,601 INFO [main] common.TezUtilsInternal:
Redirecting log file based on addend: dag_1426707664723_1086_1
5499177953-
5499177954-LogType:syslog_dag_1426707664723_1086_1
5499177994-Log Upload Time:1-Apr-2015 16:24:30
5499178030-LogLength:521
5499178044-Log Contents:
5499178058-2015-04-01 16:23:22,604 INFO [main] impl.DAGImpl: Recovered DAG:
dag_1426707664723_1086_1 finished with state: FAILED
5499178176-2015-04-01 16:23:22,604 INFO [main] impl.DAGImpl:
dag_1426707664723_1086_1 transitioned from NEW to FAILED
5499178283-2015-04-01 16:23:22,604 INFO [AsyncDispatcher event handler]
app.DAGAppMaster: DAG completed, dagId=dag_1426707664723_1086_1, dagState=FAILED
5499178425-2015-04-01 16:23:22,605 INFO [AsyncDispatcher event handler]
common.TezUtilsInternal: Redirecting log file based on addend:
dag_1426707664723_1086_1_post
5499178579-
5499178580-LogType:syslog_dag_1426707664723_1086_1_post
5499178625-Log Upload Time:1-Apr-2015 16:24:30
5499178661-LogLength:4021
5499178676-Log Contents:
5499178690-2015-04-01 16:23:22,605 INFO [AsyncDispatcher event handler]
app.DAGAppMaster: Waiting for next DAG to be submitted.
5499178807-2015-04-01 16:24:01,681 INFO [IPC Server handler 0 on 53890]
client.DAGClientHandler: Received message to shutdown AM
5499178925-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890]
rm.TaskSchedulerEventHandler: TaskScheduler notified that it should unregister
from RM
5499179073-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890]
app.DAGAppMaster: No current running DAG, shutting down the AM
5499179197-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890]
app.DAGAppMaster: DAGAppMasterShutdownHandler invoked
5499179312-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890]
app.DAGAppMaster: Handling DAGAppMaster shutdown
5499179422-2015-04-01 16:24:01,683 INFO [AMShutdownThread] app.DAGAppMaster:
Sleeping for 5 seconds before shutting down
5499179532-2015-04-01 16:24:04,151 INFO [HistoryEventHandlingThread]
ats.ATSHistoryLoggingService: Event queue stats,
eventsProcessedSinceLastUpdate=2, eventQueueSize=0
5499179690-2015-04-01 16:24:06,683 INFO [AMShutdownThread] app.DAGAppMaster:
Calling stop for all the services
5499179790-2015-04-01 16:24:06,686 INFO [AMShutdownThread]
history.HistoryEventHandler: Stopping HistoryEventHandler
5499179896-2015-04-01 16:24:06,686 INFO [AMShutdownThread]
recovery.RecoveryService: Stopping RecoveryService
5499179995-2015-04-01 16:24:06,686 INFO [AMShutdownThread]
ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=0
5499180114-2015-04-01 16:24:06,686 INFO [RecoveryEventHandlingThread]
recovery.RecoveryService: EventQueue take interrupted. Returning
5499180238-2015-04-01 16:24:06,692 INFO [DelayedContainerManager]
rm.YarnTaskSchedulerService: AllocatedContainerManager Thread interrupted
5499180367-2015-04-01 16:24:06,697 INFO [AMShutdownThread]
rm.YarnTaskSchedulerService: Unregistering application from RM,
exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=0,
successfulDAGs=0, failedDAGs=1, killedDAGs=0
5499180589-, trackingURL=
5499180604-2015-04-01 16:24:06,713 INFO [AMShutdownThread] impl.AMRMClientImpl:
Waiting for application to be successfully unregistered.
5499180730-2015-04-01 16:24:06,819 INFO [AMShutdownThread]
rm.YarnTaskSchedulerService: Successfully unregistered application from RM
5499180853-2015-04-01 16:24:06,821 INFO [AMShutdownThread] ipc.Server: Stopping
server on 50998
5499180938-2015-04-01 16:24:06,821 INFO [AMRM Callback Handler Thread]
impl.AMRMClientAsyncImpl: Interrupted while waiting for queue
5499181060:java.lang.InterruptedException
5499181091- at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
5499181228- at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
5499181346- at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
5499181426- at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
5499181551-2015-04-01 16:24:06,823 INFO [IPC Server Responder] ipc.Server:
Stopping IPC Server Responder
5499181645-2015-04-01 16:24:06,822 INFO [IPC Server listener on 50998]
ipc.Server: Stopping IPC Server listener on 50998
5499181755-2015-04-01 16:24:06,822 INFO [AMShutdownThread] ipc.Server: Stopping
server on 53890
5499181840-2015-04-01 16:24:06,826 INFO [IPC Server listener on 53890]
ipc.Server: Stopping IPC Server listener on 53890
5499181950-2015-04-01 16:24:06,826 INFO [IPC Server Responder] ipc.Server:
Stopping IPC Server Responder
5499182044-2015-04-01 16:24:06,828 INFO [Thread-1] app.DAGAppMaster:
DAGAppMasterShutdownHook invoked
5499182135-2015-04-01 16:24:06,828 INFO [Thread-1] app.DAGAppMaster: The
shutdown handler is still running, waiting for it to complete
5499182259-2015-04-01 16:24:06,839 WARN [AMShutdownThread] app.DAGAppMaster:
Failed to delete tez scratch data dir,
path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086
5499182521-2015-04-01 16:24:06,839 INFO [AMShutdownThread] app.DAGAppMaster:
Exiting DAGAppMaster..GoodBye!
5499182618-2015-04-01 16:24:06,839 INFO [Thread-1] app.DAGAppMaster: The
shutdown handler has completed
5499182711-
5499182712-
5499182713-
5499182714-Container: container_1426707664723_1086_01_000304 on
cn110-10.l42scl.hortonworks.com_45454
5499182805-============================================================================================
5499182898-LogType:stderr
5499182913-Log Upload Time:1-Apr-2015 16:24:30
5499182949-LogLength:0
5499182961-Log Contents:
5499182975-
5499182976-LogType:stdout
5499182991-Log Upload Time:1-Apr-2015 16:24:30
5499183027-LogLength:81042
5499183043-Log Contents:
5499183057-9.076: [GC [PSYoungGen: 415194K->108031K(756736K)]
1463770K->1182454K(7674880K), 0.1355580 secs] [Times: user=0.59 sys=0.13,
real=0.13 secs]
5499183199-16.885: [GC [PSYoungGen: 446979K->70470K(756736K)]
1521402K->1144901K(7674880K), 0.1141880 secs] [Times: user=0.41 sys=0.08,
real=0.11 secs]
5499183341-32.610: [GC [PSYoungGen: 401811K->23054K(756736K)]
1476242K->1097493K(7674880K), 0.0198940 secs] [Times: user=0.26 sys=0.00,
real=0.02 secs]
5499183483-37.397: [GC [PSYoungGen: 354557K->108009K(756736K)]
2477572K->2246623K(7674880K), 0.0591560 secs] [Times: user=0.82 sys=0.06,
real=0.06 secs]
5499183626-42.607: [GC [PSYoungGen: 434535K->15928K(649216K)]
2573148K->2154549K(7567360K), 0.0639210 secs] [Times: user=0.45 sys=0.03,
real=0.06 secs]
5499183768-47.641: [GC [PSYoungGen: 553804K->22893K(689152K)]
3741001K->3210098K(7607296K), 0.0727270 secs] [Times: user=0.55 sys=0.06,
real=0.07 secs]
5499183910-53.107: [GC [PSYoungGen: 343874K->137457K(646656K)]
4579655K->4384105K(7564800K), 0.1012020 secs] [Times: user=0.50 sys=0.14,
real=0.10 secs]
5499184053-62.586: [GC [PSYoungGen: 484074K->89878K(676864K)]
4730722K->4336778K(7595008K), 0.0985300 secs] [Times: user=0.56 sys=0.08,
real=0.10 secs]
5499184195-76.005: [GC [PSYoungGen: 449294K->16827K(668672K)]
4696194K->4274117K(7586816K), 0.0315610 secs] [Times: user=0.36 sys=0.03,
real=0.03 secs]
5499184337-79.777: [GC [PSYoungGen: 211179K->19599K(680448K)]
5517045K->5332144K(7598592K), 0.0304100 secs] [Times: user=0.37 sys=0.02,
real=0.03 secs]
5499184479-81.315: [GC [PSYoungGen: 150008K->78090K(677888K)]
5462554K->5399597K(7596032K), 0.0293570 secs] [Times: user=0.39 sys=0.02,
real=0.03 secs]
5499184621-82.455: [GC [PSYoungGen: 237557K->512K(687616K)]
5559064K->5324779K(7605760K), 0.0384990 secs] [Times: user=0.34 sys=0.02,
real=0.04 secs]
5499184761-84.067: [GC [PSYoungGen: 210827K->14117K(687616K)]
6583671K->6387049K(7605760K), 0.0517180 secs] [Times: user=0.66 sys=0.01,
real=0.05 secs]
5499184903-88.416: [GC [PSYoungGen: 268920K->15351K(696320K)]
6641851K->6394989K(7614464K), 0.0787950 secs] [Times: user=0.51 sys=0.02,
real=0.08 secs]
5499185045-101.043: [GC [PSYoungGen: 376721K->448K(691200K)]
6756359K->6387846K(7609344K), 0.0282280 secs] [Times: user=0.44 sys=0.02,
real=0.03 secs]
5499185186-103.105: [GC [PSYoungGen: 82965K->13643K(705536K)]
6470363K->6401129K(7623680K), 0.1482160 secs] [Times: user=0.53 sys=0.00,
real=0.15 secs]
5499185328-103.253: [GC [PSYoungGen: 13643K->0K(699392K)]
6401129K->6400834K(7617536K), 0.0805200 secs] [Times: user=0.56 sys=0.02,
real=0.08 secs]
5499185466-103.334: [Full GC [PSYoungGen: 0K->0K(699392K)] [ParOldGen:
6400834K->26007K(6918144K)] 6400834K->26007K(7617536K) [PSPermGen:
33081K->33057K(66560K)], 0.4079400 secs] [Times: user=1.08 sys=0.17, real=0.41
secs]
5499185679-108.478: [GC [PSYoungGen: 279515K->23409K(718848K)]
1354098K->1097992K(7636992K), 0.0143960 secs] [Times: user=0.06 sys=0.00,
real=0.01 secs]
5499185822-121.280: [GC [PSYoungGen: 322641K->279K(709632K)]
1397224K->1082530K(7627776K), 0.0187460 secs] [Times: user=0.10 sys=0.00,
real=0.01 secs]
5499185963-126.595: [GC [PSYoungGen: 345663K->20073K(731648K)]
2476491K->2150964K(7649792K), 0.0305890 secs] [Times: user=0.19 sys=0.00,
real=0.03 secs]
5499186106-141.120: [GC [PSYoungGen: 596191K->14241K(723456K)]
3775658K->3200292K(7641600K), 0.0189990 secs] [Times: user=0.27 sys=0.00,
real=0.02 secs]
--
18043554745-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 49
18043554860-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 44
18043554975-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 45
18043555090-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 50
18043555205-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 25
18043555320-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 17
18043555435-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 23
18043555550-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 24
18043555665-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 21
18043555780-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 22
18043555895-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 38
18043556014-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 20
18043556129-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 31
18043556248-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 37
18043556367-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 29
18043556482-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 30
18043556601-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 27
18043556716-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 28
18043556831-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 3
18043556949-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 26
18043557064-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 2
18043557182-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 32
18043557297-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 36
18043557416-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 35
18043557535-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 34
18043557654-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 33
18043557769-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 1
18043557883-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 39
18043557998-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 5
18043558112-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 4
18043558226-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 43
18043558341-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 53
18043558456-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 42
18043558571-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 54
18043558686-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 41
18043558801-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 51
18043558916-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 14
18043559031-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 7
18043559149-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 13
18043559264-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 6
18043559382-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 16
18043559497-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 19
18043559612-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 18
18043559727-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Map 15
18043559842-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No exclusive output committers for vertex: Reducer 12
18043559971-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 10
18043560090-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 11
18043560209-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 8
18043560327-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler]
impl.DAGImpl: No output committers for vertex: Reducer 9
18043560445-2015-04-01 16:23:08,857 FATAL [AsyncDispatcher event handler]
event.AsyncDispatcher: Error in dispatcher thread
18043560557:org.apache.tez.common.counters.LimitExceededException: Too many
counters: 1201 max=1200
18043560645- at
org.apache.tez.common.counters.Limits.checkCounters(Limits.java:87)
18043560717- at
org.apache.tez.common.counters.Limits.incrCounters(Limits.java:94)
18043560788- at
org.apache.tez.common.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:75)
18043560885- at
org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:92)
18043560986- at
org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:103)
18043561085- at
org.apache.tez.common.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:198)
18043561188- at
org.apache.tez.common.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:363)
18043561283- at
org.apache.tez.dag.app.dag.impl.DAGImpl.incrTaskCounters(DAGImpl.java:598)
18043561362- at
org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:588)
18043561439- at
org.apache.tez.dag.app.dag.impl.DAGImpl.logJobHistoryFinishedEvent(DAGImpl.java:994)
18043561528- at
org.apache.tez.dag.app.dag.impl.DAGImpl.finished(DAGImpl.java:1135)
18043561600- at
org.apache.tez.dag.app.dag.impl.DAGImpl.checkDAGForCompletion(DAGImpl.java:1048)
18043561685- at
org.apache.tez.dag.app.dag.impl.DAGImpl$VertexCompletedTransition.transition(DAGImpl.java:1708)
18043561785- at
org.apache.tez.dag.app.dag.impl.DAGImpl$VertexCompletedTransition.transition(DAGImpl.java:1665)
18043561885- at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
18043562001- at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
18043562097- at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
18043562190- at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
18043562307- at
org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:944)
18043562376- at
org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:126)
18043562445- at
org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:1686)
18043562535- at
org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:1677)
18043562625- at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
18043562709- at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
18043562790- at java.lang.Thread.run(Thread.java:745)
18043562832-2015-04-01 16:23:08,882 INFO [AsyncDispatcher event handler]
event.AsyncDispatcher: Exiting, bbye..
18043562932-2015-04-01 16:23:08,885 INFO [Thread-1] app.DAGAppMaster:
DAGAppMasterShutdownHook invoked
18043563023-2015-04-01 16:23:08,885 INFO [Thread-1] app.DAGAppMaster:
DAGAppMaster received a signal. Signaling TaskScheduler
18043563137-2015-04-01 16:23:08,885 INFO [Thread-1]
rm.TaskSchedulerEventHandler: TaskScheduler notified that iSignalled was : true
18043563257-2015-04-01 16:23:08,899 INFO [Thread-1]
history.HistoryEventHandler: Stopping HistoryEventHandler
18043563355-2015-04-01 16:23:08,900 INFO [Thread-1] recovery.RecoveryService:
Stopping RecoveryService
18043563446-2015-04-01 16:23:08,900 INFO [Thread-1] recovery.RecoveryService:
Closing Summary Stream
18043563535-2015-04-01 16:23:08,900 INFO [RecoveryEventHandlingThread]
recovery.RecoveryService: EventQueue take interrupted. Returning
18043563659-2015-04-01 16:23:09,033 INFO [Thread-1] recovery.RecoveryService:
Closing Output Stream for DAG dag_1426707664723_1086_1
18043563780-2015-04-01 16:23:09,062 INFO [Thread-1]
ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=0
18043563891-2015-04-01 16:23:09,064 INFO [Thread-1]
launcher.ContainerLauncherImpl: Sending a stop request to the NM for
ContainerId: container_1426707664723_1086_01_000319
18043564052-2015-04-01 16:23:09,064 INFO [Thread-1]
impl.ContainerManagementProtocolProxy: Opening proxy :
cn113-10.l42scl.hortonworks.com:45454
18043564185-2015-04-01 16:23:09,097 INFO [Thread-1]
launcher.ContainerLauncherImpl: Sending a stop request to the NM for
ContainerId: container_1426707664723_1086_01_000047
18043564346-2015-04-01 16:23:09,097 INFO [Thread-1]
impl.ContainerManagementProtocolProxy: Opening proxy :
cn122-10.l42scl.hortonworks.com:45454
18043564479-2015-04-01 16:23:09,114 INFO [Thread-1]
launcher.ContainerLauncherImpl: Sending a stop request to the NM for
ContainerId: container_1426707664723_1086_01_000306
18043564640-2015-04-01 16:23:09,114 INFO [Thread-1]
impl.ContainerManagementProtocolProxy: Opening proxy :
cn113-10.l42scl.hortonworks.com:45454
18043564773-2015-04-01 16:23:09,120 INFO [Thread-1]
launcher.ContainerLauncherImpl: Sending a stop request to the NM for
ContainerId: container_1426707664723_1086_01_000104
18043564934-2015-04-01 16:23:09,120 INFO [Thread-1]
impl.ContainerManagementProtocolProxy: Opening proxy :
cn111-10.l42scl.hortonworks.com:45454
18043565067-2015-04-01 16:23:09,145 INFO [Thread-1]
launcher.ContainerLauncherImpl: Sending a stop request to the NM for
ContainerId: container_1426707664723_1086_01_000140
18043565228-2015-04-01 16:23:09,145 INFO [Thread-1]
impl.ContainerManagementProtocolProxy: Opening proxy :
cn120-10.l42scl.hortonworks.com:45454
18043565361-2015-04-01 16:23:09,152 INFO [Thread-1]
launcher.ContainerLauncherImpl: Sending a stop request to the NM for
ContainerId: container_1426707664723_1086_01_000236
18043565522-2015-04-01 16:23:09,152 INFO [Thread-1]
impl.ContainerManagementProtocolProxy: Opening proxy :
cn107-10.l42scl.hortonworks.com:45454
18043565655-2015-04-01 16:23:09,159 INFO [Thread-1]
launcher.ContainerLauncherImpl: Sending a stop request to the NM for
ContainerId: container_1426707664723_1086_01_000255
18043565816-2015-04-01 16:23:09,159 INFO [Thread-1]
impl.ContainerManagementProtocolProxy: Opening proxy :
cn116-10.l42scl.hortonworks.com:45454
18043565949-2015-04-01 16:23:09,182 INFO [Thread-1]
launcher.ContainerLauncherImpl: Sending a stop request to the NM for
ContainerId: container_1426707664723_1086_01_000074
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
