[
https://issues.apache.org/jira/browse/HADOOP-18168?focusedWorklogId=763726&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-763726
]
ASF GitHub Bot logged work on HADOOP-18168:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 28/Apr/22 17:17
Start Date: 28/Apr/22 17:17
Worklog Time Spent: 10m
Work Description: steveloughran commented on code in PR #4140:
URL: https://github.com/apache/hadoop/pull/4140#discussion_r861131956
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/PublicDatasetTestUtils.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
Review Comment:
can you put in the sub package .test
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java:
##########
@@ -18,6 +18,8 @@
package org.apache.hadoop.fs.s3a;
+import org.apache.hadoop.conf.Configuration;
Review Comment:
nit: import should go below the java. one
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARequesterPays.java:
##########
@@ -102,14 +88,30 @@ public void testRequesterPaysDisabledFails() throws
Throwable {
}
}
- private Path getRequesterPaysPath(Configuration conf) {
- String requesterPaysFile =
- conf.getTrimmed(KEY_REQUESTER_PAYS_FILE, DEFAULT_REQUESTER_PAYS_FILE);
- S3ATestUtils.assume(
- "Empty test property: " + KEY_REQUESTER_PAYS_FILE,
- !requesterPaysFile.isEmpty()
- );
- return new Path(requesterPaysFile);
+ /**
+ * Use this after creating the file system, as this is when bucket-specific
+ * overrides are applied.
+ * @param conf Hadoop configuration from FS to mutate
+ * @param requesterPaysEnabled Indicate if requester pays be on or off
+ */
+ private static void updateConf(
+ Configuration conf,
+ boolean requesterPaysEnabled
Review Comment:
nit: we tend to pull that ) onto the same line as the last paraem
Issue Time Tracking
-------------------
Worklog Id: (was: 763726)
Time Spent: 2.5h (was: 2h 20m)
> ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket
> content purged
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-18168
> URL: https://issues.apache.org/jira/browse/HADOOP-18168
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.3.4
> Reporter: Steve Loughran
> Assignee: Daniel Carl Jones
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> {{ITestMarkerTool.testRunLimitedLandsatAudit}} is failing -a scan which was
> meant to stop after the first page of results is finishing because there
> aren't so many objects there.
>
> first visible sign of the landsat-pds cleanup
> now we have requester pays, we could do this against another store with
> stability promises, e.g common crawl.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
