[
https://issues.apache.org/jira/browse/HADOOP-17847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17408221#comment-17408221
]
Steve Loughran commented on HADOOP-17847:
-----------------------------------------
That error there is to force halt apps which think that s3a is a filesystem
with flush/sync semantics they can use (hbase, audit logging in ranger etc).
It does fail quite dramatically, but that's a way of letting you know it's
broken. And yes, you can downgrade
The event log writer thinks it is writing to something it can flush.
if it's expecting incremental writes without a need to call close, yes things
are broken. And that first error probably means that the FS was shut down
before close() was called.
if the logging is just writing to the FS and rolling files every few minutes,
well, it'll only be the last few minutes of log lost, with some errors. If it's
actually expected to collect over time (or: other processes to read it as it is
written), not going to happen.
That said, spark log is trying to rename a file which doesn't exist; it doesn't
exist because the block upload is still in progress.
Overall then
* probably some ordering/race condition in shutdown, e.g. s3a FS being shut
down before the event logger has finished
* probably exists everywhere, but surfaces here because hflush doesn't flush
and close() is slow.
Write to some "real" FS, or have some event logger which works with S3.
> S3AInstrumentation Closing output stream statistics while data is still
> marked as pending upload in OutputStreamStatistics
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-17847
> URL: https://issues.apache.org/jira/browse/HADOOP-17847
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.2.1
> Environment: hadoop: 3.2.1
> spark: 3.0.2
> k8s server version: 1.18
> aws.java.sdk.bundle.version:1.11.1033
> Reporter: Li Rong
> Priority: Major
> Attachments: logs.txt
>
>
> When using hadoop s3a file upload for spark event Logs, the logs were queued
> up and not uploaded before the process is shut down:
> {code:java}
> // 21/08/13 12:22:39 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client
> has been closed (this is expected if the application is shutting down.)
> 21/08/13 12:22:39 WARN S3AInstrumentation: Closing output stream statistics
> while data is still marked as pending upload in
> OutputStreamStatistics{blocksSubmitted=1, blocksInQueue=1, blocksActive=0,
> blockUploadsCompleted=0, blockUploadsFailed=0, bytesPendingUpload=106716,
> bytesUploaded=0, blocksAllocated=1, blocksReleased=1,
> blocksActivelyAllocated=0, exceptionsInMultipartFinalize=0,
> transferDuration=0 ms, queueDuration=0 ms, averageQueueTime=0 ms,
> totalUploadDuration=0 ms, effectiveBandwidth=0.0 bytes/s}{code}
> details see logs attached
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
