[
https://issues.apache.org/jira/browse/HADOOP-19573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17952954#comment-17952954
]
ASF GitHub Bot commented on HADOOP-19573:
-----------------------------------------
steveloughran commented on PR #7699:
URL: https://github.com/apache/hadoop/pull/7699#issuecomment-2895627960
me having set a per-bucket override for fs.s3a.buffer.dir meant that the
fallback to hadoop.tmp.dir never got tested.
but, because of how dir allocators were in a static conf -> instance
mapping, this only surfaced depending on what the config in earlier runs was.
the stuff with the space is so that you can explicitly set a value for the
fs.s3a.bucket.BUCKET.buffer.dir and so not have the per-bucket mapping code
pick up the one in the auth-keys.xml config instead.
And it looks like a space is needed for that mapping, rather than "". But
that's OK as if you look at the local dir allocator code it does
`StringUtils.getTrimmedStrings` on the conf value, so we should fall back to
hadoop.tmp.dir if the buffer dir is a space.
Too me a while to work out WTF was wrong. I was worried the local dir
allocator patch had actually triggered a regression -but really it was me
changing my test setup surfaced a latent bug.
> S3A: ITestS3AConfiguration.testDirectoryAllocatorDefval() failing
> -----------------------------------------------------------------
>
> Key: HADOOP-19573
> URL: https://issues.apache.org/jira/browse/HADOOP-19573
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
> Affects Versions: 3.5.0, 3.4.2, 3.4.3
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
>
> while working on HADOOP-19554 I added a per-bucket setting for
> fs.s3a.buffer.dir
> after this, ITestS3AConfiguration.testDirectoryAllocatorDefval() would fail
> in a test run of the entire class, but not if run alone.
> Causes
> * dir allocator map of config key to allocator is static; previous uses
> tainted outcome
> * per-bucket settings were't being overridden. This is complicated by the
> fact that "unset" isn't a setting, therefore can't be forced in. Instead some
> whitespace needs to be set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
