[
https://issues.apache.org/jira/browse/YARN-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15185261#comment-15185261
]
Steve Loughran commented on YARN-4696:
--------------------------------------
BTW, I managed to create an exception in scale tests; when the file:// fs did
eventually flush, a checksum exception was raised on the read
{code}
2016-03-08 12:01:21,991 [EntityLogPluginWorker #2] INFO
org.apache.hadoop.fs.FSInputChecker
(FSInputChecker.java:readChecksumChunk(308)) - Found checksum error: b[1802,
1802]=
org.apache.hadoop.fs.ChecksumException: Checksum error:
file:/Users/stevel/Projects/Hortonworks/Projects/sparkwork/spark-timeline-integration/target/tmp/integration/ats/active/application_1111_0000/appattempt_1111_0000_000000/summarylog-appattempt_1111_0000_000000_1457438447660
at 13106176
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:258)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:299)
at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:251)
at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.codehaus.jackson.impl.Utf8StreamParser.loadMore(Utf8StreamParser.java:172)
at
org.codehaus.jackson.impl.Utf8StreamParser._decodeEscaped(Utf8StreamParser.java:2515)
at
org.codehaus.jackson.impl.Utf8StreamParser._finishString2(Utf8StreamParser.java:1949)
at
org.codehaus.jackson.impl.Utf8StreamParser._finishString(Utf8StreamParser.java:1905)
at
org.codehaus.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276)
at
org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:59)
at
org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.mapObject(UntypedObjectDeserializer.java:218)
at
org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:47)
at
org.codehaus.jackson.map.deser.std.MapDeserializer._readAndBind(MapDeserializer.java:319)
at
org.codehaus.jackson.map.deser.std.MapDeserializer.deserialize(MapDeserializer.java:263)
at
org.codehaus.jackson.map.deser.std.MapDeserializer.deserialize(MapDeserializer.java:33)
at
org.codehaus.jackson.map.deser.SettableBeanProperty$SetterlessProperty.deserializeAndSet(SettableBeanProperty.java:511)
at
org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:697)
at
org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)
at
org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:217)
at
org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:194)
at
org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:30)
at
org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:299)
at
org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:414)
at
org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:697)
at
org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)
at
org.codehaus.jackson.map.MappingIterator.nextValue(MappingIterator.java:178)
at
org.codehaus.jackson.map.MappingIterator.next(MappingIterator.java:111)
at
org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:189)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:153)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:112)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$AppLogs.parseSummaryLogs(EntityGroupFSTimelineStore.java:589)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$AppLogs.parseSummaryLogs(EntityGroupFSTimelineStore.java:565)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$ActiveLogParser.run(EntityGroupFSTimelineStore.java:752)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
> EntityGroupFSTimelineStore to work in the absence of an RM
> ----------------------------------------------------------
>
> Key: YARN-4696
> URL: https://issues.apache.org/jira/browse/YARN-4696
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: YARN-4696-001.patch, YARN-4696-002.patch,
> YARN-4696-003.patch, YARN-4696-005.patch, YARN-4696-006.patch,
> YARN-4696-007.patch, YARN-4696-008.patch, YARN-4696-009.patch,
> YARN-4696-010.patch, YARN-4696-012.patch
>
>
> {{EntityGroupFSTimelineStore}} now depends on an RM being up and running; the
> configuration pointing to it. This is a new change, and impacts testing where
> you have historically been able to test without an RM running.
> The sole purpose of the probe is to automatically determine if an app is
> running; it falls back to "unknown" if not. If the RM connection was
> optional, the "unknown" codepath could be called directly, relying on age of
> file as a metric of completion
> Options
> # add a flag to disable RM connect
> # skip automatically if RM not defined/set to 0.0.0.0
> # disable retries on yarn client IPC; if it fails, tag app as unknown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
