Steve Jacobs created HADOOP-15628:
-------------------------------------
Summary: S3A Filesystem does not check return from AmazonS3Client
deleteObjects
Key: HADOOP-15628
URL: https://issues.apache.org/jira/browse/HADOOP-15628
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.0.3, 2.8.4, 2.9.1, 3.1.1
Environment: Hadoop 3.0.2 / Hadoop 2.8.3
Hive 2.3.2 / Hive 2.3.3 / Hive 3.0.0
Reporter: Steve Jacobs
Deletes in S3A that use the Multi-Delete functionality in the Amazon S3 api do
not check to see if all objects have been succesfully delete. In the event of a
failure, the api will still return a 200 OK (which isn't checked currently):
[Current Delete
Code|https://github.com/apache/hadoop/blob/a0da1ec01051108b77f86799dd5e97563b2a3962/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L574]
{code:java}
if (keysToDelete.size() == MAX_ENTRIES_TO_DELETE) {
DeleteObjectsRequest deleteRequest =
new DeleteObjectsRequest(bucket).withKeys(keysToDelete);
s3.deleteObjects(deleteRequest);
statistics.incrementWriteOps(1);
keysToDelete.clear();
}
{code}
This should be converted to use the DeleteObjectsResult class from the
S3Client:
[Amazon Code
Example|https://docs.aws.amazon.com/AmazonS3/latest/dev/DeletingMultipleObjectsUsingJava.htm]
{code:java}
// Verify that the objects were deleted successfully.
DeleteObjectsResult delObjRes =
s3Client.deleteObjects(multiObjectDeleteRequest); int successfulDeletes =
delObjRes.getDeletedObjects().size();
System.out.println(successfulDeletes + " objects successfully deleted.");
{code}
Bucket policies can be misconfigured, and deletes will fail without warning by
S3A clients.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
