[
https://issues.apache.org/jira/browse/HADOOP-17657?focusedWorklogId=588161&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-588161
]
ASF GitHub Bot logged work on HADOOP-17657:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 23/Apr/21 21:14
Start Date: 23/Apr/21 21:14
Worklog Time Spent: 10m
Work Description: kishendas opened a new pull request #2949:
URL: https://github.com/apache/hadoop/pull/2949
Following exception is thrown whenever we invoke ProtoMessageWriter.hflush
on S3 from Tez, which internally calls
org.apache.hadoop.io.SequenceFile$Writer.hflush -> org.apache.hadoop.fs.FS
DataOutputStream.hflush -> S3ABlockOutputStream.hflush which is not implemented
and throws java.lang.UnsupportedOperationException.
bdffe22d96ae [mdc@18060 class="yarn.YarnUncaughtExceptionHandler"
level="ERROR" thread="HistoryEventHandlingThread"] Thread
Thread[HistoryEventHandlingThread, 5,main] threw an
Exception.^Mjava.lang.UnsupportedOperationException: S3A streams are not
Syncable^M at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.hflush(S3ABlockOutputStream.java:657)^M
at org.apache.hadoop.fs.FS
DataOutputStream.hflush(FSDataOutputStream.java:136)^M at
org.apache.hadoop.io.SequenceFile$Writer.hflush(SequenceFile.java:1367)^M at
org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(ProtoMessageWr
iter.java:64)^M at
org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.finishCurrentDag(ProtoHistoryLoggingService.java:239)^M
at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.han
dleEvent(ProtoHistoryLoggingService.java:198)^M at
org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.loop(ProtoHistoryLoggingService.java:153)^M
at java.lang.Thread.run(Thread.java:748)^M
In order to fix this issue we should implement StreamCapabilities in
SequenceFile.Writer. Also, we should fall back to flush(), if hflush() is not
supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 588161)
Remaining Estimate: 0h
Time Spent: 10m
> SequeneFile.Writer should implement StreamCapabilities
> ------------------------------------------------------
>
> Key: HADOOP-17657
> URL: https://issues.apache.org/jira/browse/HADOOP-17657
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Kishen Das
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Following exception is thrown whenever we invoke ProtoMessageWriter.hflush on
> S3 from Tez, which internally calls
> org.apache.hadoop.io.SequenceFile$Writer.hflush -> org.apache.hadoop.fs.FS
> DataOutputStream.hflush -> S3ABlockOutputStream.hflush which is not
> implemented and throws java.lang.UnsupportedOperationException.
> bdffe22d96ae [mdc@18060 class="yarn.YarnUncaughtExceptionHandler"
> level="ERROR" thread="HistoryEventHandlingThread"] Thread
> Thread[HistoryEventHandlingThread, 5,main] threw an
> Exception.^Mjava.lang.UnsupportedOperationException: S3A streams are not
> Syncable^M at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.hflush(S3ABlockOutputStream.java:657)^M
> at org.apache.hadoop.fs.FS
> DataOutputStream.hflush(FSDataOutputStream.java:136)^M at
> org.apache.hadoop.io.SequenceFile$Writer.hflush(SequenceFile.java:1367)^M at
> org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(ProtoMessageWr
> iter.java:64)^M at
> org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.finishCurrentDag(ProtoHistoryLoggingService.java:239)^M
> at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.han
> dleEvent(ProtoHistoryLoggingService.java:198)^M at
> org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.loop(ProtoHistoryLoggingService.java:153)^M
> at java.lang.Thread.run(Thread.java:748)^M
> In order to fix this issue we should implement StreamCapabilities in
> SequenceFile.Writer. Also, we should fall back to flush(), if hflush() is not
> supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
