[
https://issues.apache.org/jira/browse/HADOOP-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17621073#comment-17621073
]
Zbigniew Kostrzewa commented on HADOOP-15915:
---------------------------------------------
I've recently stumbled upon this with {{{}3.2.2{}}}. For me the problem was
that I did not change {{hadoop.tmp.dir}} and so the {{s3ablock-0001-}} were
created in {{/tmp/hadoop-<HADOOP_USER_NAME>/s3a}} directory. At the same time,
on CentOS 7 in my case, there is a systemd service
{{systemd-tmpfiles-clean.service}} run once a day which cleans up {{/tmp}} of
files and directories older than 10 days. However, Node Manager after it caches
that {{/tmp/hadoop-<HADOOP_USER_NAME>/s3a}} exists it does not re-check it and
does not re-create that directory if it no longer exists, I believe the code
responsible for this is:
{code:java}
/** This method gets called everytime before any read/write to make sure
* that any change to localDirs is reflected immediately.
*/
private Context confChanged(Configuration conf)
throws IOException {
...
if (!newLocalDirs.equals(ctx.savedLocalDirs)) { {code}
and when the directory is missing log aggregation fails with this
{{DiskChecker}} error.
> Report problems w/ local S3A buffer directory meaningfully
> ----------------------------------------------------------
>
> Key: HADOOP-15915
> URL: https://issues.apache.org/jira/browse/HADOOP-15915
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.1
> Reporter: Steve Loughran
> Priority: Major
>
> When there's a problem working with the temp directory used for block output
> and the staging committers the actual path (and indeed config option) aren't
> printed.
> Improvements: tell the user which directory isn't writeable
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
