[
https://issues.apache.org/jira/browse/HADOOP-19654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18032517#comment-18032517
]
ASF GitHub Bot commented on HADOOP-19654:
-----------------------------------------
steveloughran commented on PR #7882:
URL: https://github.com/apache/hadoop/pull/7882#issuecomment-3437736945
fun test run today, against s3 london. Most of the multipart upload/commit
tests were failing "missing part", from cli or IDE. Testing with S3 express was
happy. (`-Dparallel-tests -DtestsThreadCount=8 -Panalytics -Dscale`)
```
[ERROR] ITestS3AHugeMagicCommits.test_030_postCreationAssertions:192 »
AWSBadRequest Completing multipart upload on
job-00/test/tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin:
software.amazon.awssdk.services.s3.model.S3Exception: One or more of the
specified parts could not be found. The part may not have been uploaded, or
the specified entity tag may not match the part's entity tag. (Service: S3,
Status Code: 400, Request ID: JAEYPCZ4P3JYGMTD, Extended Request ID:
O/135mw9Xd2aEuFUh0ICWYc8DLXSpBUWaVGkEgEFGf0xO8o+XlZXY0hI+mvennOGt+C/UI7mNrQ=)
(SDK Attempt Count: 1):InvalidPart: One or more of the specified parts could
not be found. The part may not have been uploaded, or the specified entity tag
may not match the part's entity tag. (Service: S3, Status Code: 400, Request
ID: JAEYPCZ4P3JYGMTD, Extended Request ID:
O/135mw9Xd2aEuFUh0ICWYc8DLXSpBUWaVGkEgEFGf0xO8o+XlZXY0hI+mvennOGt+C/UI7mNrQ=)
(SDK Attempt Count: 1)
[ERROR]
ITestS3AHugeMagicCommits>AbstractSTestS3AHugeFiles.test_045_vectoredIOHugeFile:538->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin
in
s3a://stevel-london/job-00/test/tests3ascale/ITestS3AHugeMagicCommits/commit
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.test_010_CreateHugeFile:276
» AWSBadRequest Completing multipart upload on
job-00/test/tests3ascale/array/src/hugefile:
software.amazon.awssdk.services.s3.model.S3Exception: One or more of the
specified parts could not be found. The part may not have been uploaded, or
the specified entity tag may not match the part's entity tag. (Service: S3,
Status Code: 400, Request ID: 1NNBCSX4NCDN7G9X, Extended Request ID:
8vMmeyt1GfjGrf3UL9AN8vlwWSn9860f1gdeIBC3drmcjeQwC6wOPinMD8MSO6ggGw9ywwdcXroGTdVSFLYq0S0VdM/5bYfanDXJ43Eb4QU=)
(SDK Attempt Count: 1):InvalidPart: One or more of the specified parts could
not be found. The part may not have been uploaded, or the specified entity tag
may not match the part's entity tag. (Service: S3, Status Code: 400, Request
ID: 1NNBCSX4NCDN7G9X, Extended Request ID:
8vMmeyt1GfjGrf3UL9AN8vlwWSn9860f1gdeIBC3drmcjeQwC6wOPinMD8MSO6ggGw9ywwdcXroGTdVSFLYq0S0VdM/5bYfanDXJ43Eb4QU=)
(SDK Attempt Count: 1)
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.test_030_postCreationAssertions:433
» FileNotFound Huge file: not found
s3a://stevel-london/job-00/test/tests3ascale/array/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/array/src
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.test_040_PositionedReadHugeFile:478->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/array/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/array/src
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.test_045_vectoredIOHugeFile:538->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/array/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/array/src
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.test_050_readHugeFile:624->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/array/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/array/src
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.test_100_renameHugeFile:679->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/array/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/array/src
[ERROR]
ITestS3AHugeFilesByteBufferBlocks>AbstractSTestS3AHugeFiles.test_010_CreateHugeFile:276
» AWSBadRequest Completing multipart upload on
job-00/test/tests3ascale/bytebuffer/src/hugefile:
software.amazon.awssdk.services.s3.model.S3Exception: One or more of the
specified parts could not be found. The part may not have been uploaded, or
the specified entity tag may not match the part's entity tag. (Service: S3,
Status Code: 400, Request ID: K0K75V8AH7SVBHS3, Extended Request ID:
kDosbp+Z2PLZn9tVtRF9QfOqh1MgLbIKYaYFn2JeIptXlBV4v1a/wFukoXnaF7fCp6zx3vR8feE0fScUJEw+WhNW9lzu9dBxssOA62UA2kg=)
(SDK Attempt Count: 1):InvalidPart: One or more of the specified parts could
not be found. The part may not have been uploaded, or the specified entity tag
may not match the part's entity tag. (Service: S3, Status Code: 400, Request
ID: K0K75V8AH7SVBHS3, Extended Request ID:
kDosbp+Z2PLZn9tVtRF9QfOqh1MgLbIKYaYFn2JeIptXlBV4v1a/wFukoXnaF7fCp6zx3vR8feE0fScUJEw+WhNW9lzu9dBxssOA62UA2kg=)
(SDK Attempt Count: 1)
[ERROR]
ITestS3AHugeFilesByteBufferBlocks>AbstractSTestS3AHugeFiles.test_030_postCreationAssertions:433
» FileNotFound Huge file: not found
s3a://stevel-london/job-00/test/tests3ascale/bytebuffer/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/bytebuffer/src
[ERROR]
ITestS3AHugeFilesByteBufferBlocks>AbstractSTestS3AHugeFiles.test_040_PositionedReadHugeFile:478->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/bytebuffer/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/bytebuffer/src
[ERROR]
ITestS3AHugeFilesByteBufferBlocks>AbstractSTestS3AHugeFiles.test_045_vectoredIOHugeFile:538->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/bytebuffer/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/bytebuffer/src
[ERROR]
ITestS3AHugeFilesByteBufferBlocks>AbstractSTestS3AHugeFiles.test_050_readHugeFile:624->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/bytebuffer/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/bytebuffer/src
[ERROR]
ITestS3AHugeFilesByteBufferBlocks>AbstractSTestS3AHugeFiles.test_100_renameHugeFile:679->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/bytebuffer/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/bytebuffer/src
[ERROR]
ITestS3AHugeFilesDiskBlocks>AbstractSTestS3AHugeFiles.test_010_CreateHugeFile:276
» AWSBadRequest Completing multipart upload on
job-00/test/tests3ascale/disk/src/hugefile:
software.amazon.awssdk.services.s3.model.S3Exception: One or more of the
specified parts could not be found. The part may not have been uploaded, or
the specified entity tag may not match the part's entity tag. (Service: S3,
Status Code: 400, Request ID: 73T4YAYRWE63WAW5, Extended Request ID:
6ucEY2heh2NsxE8dBrlZp9AE4Tb+hbvnyxea1/yp5H85BEvkQdYsfNlRH5XZM1g4hHPDSoGMVtM=)
(SDK Attempt Count: 1):InvalidPart: One or more of the specified parts could
not be found. The part may not have been uploaded, or the specified entity tag
may not match the part's entity tag. (Service: S3, Status Code: 400, Request
ID: 73T4YAYRWE63WAW5, Extended Request ID:
6ucEY2heh2NsxE8dBrlZp9AE4Tb+hbvnyxea1/yp5H85BEvkQdYsfNlRH5XZM1g4hHPDSoGMVtM=)
(SDK Attempt Count: 1)
[ERROR]
ITestS3AHugeFilesDiskBlocks>AbstractSTestS3AHugeFiles.test_030_postCreationAssertions:433
» FileNotFound Huge file: not found
s3a://stevel-london/job-00/test/tests3ascale/disk/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/disk/src
[ERROR]
ITestS3AHugeFilesDiskBlocks>AbstractSTestS3AHugeFiles.test_040_PositionedReadHugeFile:478->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/disk/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/disk/src
[ERROR]
ITestS3AHugeFilesDiskBlocks>AbstractSTestS3AHugeFiles.test_045_vectoredIOHugeFile:538->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/disk/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/disk/src
[ERROR]
ITestS3AHugeFilesDiskBlocks>AbstractSTestS3AHugeFiles.test_050_readHugeFile:624->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/disk/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/disk/src
[ERROR]
ITestS3AHugeFilesDiskBlocks>AbstractSTestS3AHugeFiles.test_100_renameHugeFile:679->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/disk/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/disk/src
[ERROR]
ITestS3AHugeFilesSSECDiskBlocks>AbstractSTestS3AHugeFiles.test_010_CreateHugeFile:276
» AWSBadRequest Completing multipart upload on
job-00/test/tests3ascale/disk/src/hugefile:
software.amazon.awssdk.services.s3.model.S3Exception: One or more of the
specified parts could not be found. The part may not have been uploaded, or
the specified entity tag may not match the part's entity tag. (Service: S3,
Status Code: 400, Request ID: ZSY181YB49GQFR83, Extended Request ID:
FrPEfsXO3Gbhxi3m4ZmyYSiyfscQ1QSm/1lKjRPLHEbLWH5vtGked+fHvZl281Dm6u013/5VP6pj42h4XISftk7p9uEIDGw31E7Ymcoviq4=)
(SDK Attempt Count: 1):InvalidPart: One or more of the specified parts could
not be found. The part may not have been uploaded, or the specified entity tag
may not match the part's entity tag. (Service: S3, Status Code: 400, Request
ID: ZSY181YB49GQFR83, Extended Request ID:
FrPEfsXO3Gbhxi3m4ZmyYSiyfscQ1QSm/1lKjRPLHEbLWH5vtGked+fHvZl281Dm6u013/5VP6pj42h4XISftk7p9uEIDGw31E7Ymcoviq4=)
(SDK Attempt Count: 1)
[ERROR]
ITestS3AHugeFilesSSECDiskBlocks>AbstractSTestS3AHugeFiles.test_030_postCreationAssertions:433
» FileNotFound Huge file: not found
s3a://stevel-london/job-00/test/tests3ascale/disk/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/disk/src
[ERROR]
ITestS3AHugeFilesSSECDiskBlocks>AbstractSTestS3AHugeFiles.test_040_PositionedReadHugeFile:478->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/disk/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/disk/src
[ERROR]
ITestS3AHugeFilesSSECDiskBlocks>AbstractSTestS3AHugeFiles.test_045_vectoredIOHugeFile:538->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/disk/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/disk/src
[ERROR]
ITestS3AHugeFilesSSECDiskBlocks>AbstractSTestS3AHugeFiles.test_050_readHugeFile:624->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/disk/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/disk/src
[ERROR]
ITestS3AHugeFilesSSECDiskBlocks>AbstractSTestS3AHugeFiles.test_100_renameHugeFile:679->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/disk/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/disk/src
[ERROR]
ITestS3AHugeFilesStorageClass.test_010_CreateHugeFile:74->AbstractSTestS3AHugeFiles.test_010_CreateHugeFile:276
» AWSBadRequest Completing multipart upload on
job-00/test/tests3ascale/array/src/hugefile:
software.amazon.awssdk.services.s3.model.S3Exception: One or more of the
specified parts could not be found. The part may not have been uploaded, or
the specified entity tag may not match the part's entity tag. (Service: S3,
Status Code: 400, Request ID: APYCQNP1GY02DGDE, Extended Request ID:
lE0hQJ67sSwCYSMmO7tDEAvEIOCcpwIbLdfqqrNTpWT0bHIaacaIEzZusajj79rnFQlWudxsMHBIUXdS9ELiKR0T923lcULZy4Essx1LoTs=)
(SDK Attempt Count: 1):InvalidPart: One or more of the specified parts could
not be found. The part may not have been uploaded, or the specified entity tag
may not match the part's entity tag. (Service: S3, Status Code: 400, Request
ID: APYCQNP1GY02DGDE, Extended Request ID:
lE0hQJ67sSwCYSMmO7tDEAvEIOCcpwIbLdfqqrNTpWT0bHIaacaIEzZusajj79rnFQlWudxsMHBIUXdS9ELiKR0T923lcULZy4Essx1LoTs=)
(SDK Attempt Count: 1)
[ERROR]
ITestS3AHugeFilesStorageClass.test_030_postCreationAssertions:81->AbstractSTestS3AHugeFiles.test_030_postCreationAssertions:433
» FileNotFound Huge file: not found
s3a://stevel-london/job-00/test/tests3ascale/array/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/array/src
[ERROR]
ITestS3AHugeFilesStorageClass>AbstractSTestS3AHugeFiles.test_045_vectoredIOHugeFile:538->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/array/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/array/src
[ERROR]
ITestS3AHugeFilesStorageClass.test_100_renameHugeFile:108->AbstractSTestS3AHugeFiles.assumeHugeFileExists:404->AbstractSTestS3AHugeFiles.assumeFileExists:414
» FileNotFound huge file not created: not found
s3a://stevel-london/job-00/test/tests3ascale/array/src/hugefile in
s3a://stevel-london/job-00/test/tests3ascale/array/src
[INFO]
[ERROR] Tests run: 124, Failures: 1, Errors: 30, Skipped: 13
[INFO]
```
This has to be some transient issue with my s3 london bucket, as if in
progress upload parts were not being retained. Never seen this before; the
expiry time is set to 24h
When these uploads fail we do leave incomplete uploads in progress:
```
Listing uploads under path ""
job-00-fork-0005/test/testCommitOperations
141OKG11JHhWF1GOnunHUd9ZzBJ8cUG9z0LsW_4wUGgCXCvDMQM3kRi5IOCUV8FdCHtg_w8SlipfubRtzCQoT5yEpOLv.cWOiOwjEaBzUjnuJORppfXuKy1piHpLnu98
job-00-fork-0005/test/testIfMatchTwoMultipartUploadsRaceConditionOneClosesFirst
yBJpm3zh4DjNQIDtyWgEmWVCk5sehVz5Vzn3QGr_tQT2iOonRp5ErXsQy24yIvnzRxBCZqVapy5VepLeu2udZBT5EXLnKRA3bchvzjtKDlipywSzYlL2N_xLUDCT359I
job-00-fork-0005/test/testIfNoneMatchTwoConcurrentMultipartUploads
AnspJPHUoPJqg61t28OvLfAogi6G9ocyx1Dm6XY2C.a_H_onklM0Nr0LIXaPiYlQjZIiH0fTsQ1e2KhEjS9pGxvSKOXq_4YibiGZmFC6rBolmfACMqIRpoeaqYDgzYW4
job-00-fork-0005/test/testMagicWriteRecovery/file.txt
KpvoTuVh85Wzm9XuU1EuxbATjb6D.Zv8vEj3z2S6AvJBHCBssy4iphxNhTkLDs7ceEwak4IPtdXED1vRf3geXT7MRMJn8d6feafvHVEgzbD31odpzTLmOaPrU_mFQXGV
job-00-fork-0005/test/testMagicWriteRecovery/file.txt
CnrbWU3pzgEGvjRuDuaP43Xcv1eBF5aLknqYaZA1vwO3b1QUIu9QJSiZjuLMYKT9GKw1QXwqoKo4iuxTY1a18bARx4XMEiL98kZBv0TPMaAfXE.70Olh8Q2kTyDlUCSh
job-00-fork-0005/test/testMagicWriteRecovery/file.txt
dEVGPBRsuOAzL5pGA02ve9qJhAlNK8lb8khF6laKjo9U0j_aG1xLkHEfPLrmcrcsLxC3R755Yv_uKbzY_Vnoc.nXCprvutM1TZmLLN_7LHrQ0tY0IjYSS6hVzDVlHbvC
job-00-fork-0006/test/restricted/testCommitEmptyFile/empty-commit.txt
NOCjVJqycZhkalrvU26F5oIaJP51q055et2N6b74.2JVjiKL8KwrhOhdrtumOrZ2tZWNqaK4iKZ_iosqgehJOiPbWJwxvrfvA5V.dAUTLNqjtEf5tfWh0UXu.vahDy_S5SSgNLFXK.VB82i5MZtOcw--
job-00/test/tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin
lsYNpdn_oiWLwEVvvM621hCvIwDVaL4y_bbwVpQouW1OBThA.P9cR8fZtxvBjGdMY41UH0dTjxGHtF3BXEY8WXqmcnO9QHs_Jy.os781pE3MGzqgzFyxmd0yN6LFcTbq
test/restricted/testCommitEmptyFile/empty-commit.txt
T3W9V56Bv_FMhKpgcBgJ1H2wOBkPKk23T0JomesBzZyqiIAu3NiROibAgoZUhWSdoTKSJoOgcn3UWYGOvGBbsHteS_N_c1QoTEp0GE7PNlzDfs1GheJ5SOpUgaEY6MaYdNe0mn0gY48FDXpVB2nqiA--
test/restricted/testCommitEmptyFile/empty-commit.txt
.cr4b3xkfze4N24Bj3PAm_ACIyIVuTU4DueDktU1abNu2LJWXH2HKnUu1oOjfnnQwnUXp4VmXBVbZ5aq8E8gVCxN.Oyb7hmGVtESmRjpqIXSW80JrB_0_dqXe.uAT.JH7kEWywAlb4NIqJ5Xz99tvA--
Total 10 uploads found.
```
Most interesting here is `testIfNoneMatchTwoConcurrentMultipartUploads`,
because this initiates then completes an MPU, so as to create a zero byte file.
It doesn't upload any parts.
The attempt to complete failed.
```
[ERROR]
ITestS3APutIfMatchAndIfNoneMatch.testIfNoneMatchTwoConcurrentMultipartUploads:380->createFileWithFlags:190
» AWSBadRequest Completing multipart upload on
job-00-fork-0005/test/testIfNoneMatchTwoConcurrentMultipartUploads:
software.amazon.awssdk.services.s3.model.S3Exception: One or more of the
specified parts could not be found. The part may not have been uploaded, or
the specified entity tag may not match the part's entity tag. (Service: S3,
Status Code: 400, Request ID: 9JCJ6M5QRDGJNYYS, Extended Request ID:
Z7Q7+LA0o/5B4xoIGhgo+tVppawZ0UBj7X4RNb+0m9RbOAOwD/Apv1o+KmnW0aypjwmfFlarxjo=)
(SDK Attempt Count: 1):InvalidPart: One or more of the specified parts could
not be found. The part may not have been uploaded, or the specified entity tag
may not match the part's entity tag. (Service: S3, Status Code: 400, Request
ID: 9JCJ6M5QRDGJNYYS, Extended Request ID:
Z7Q7+LA0o/5B4xoIGhgo+tVppawZ0UBj7X4RNb+0m9RbOAOwD/Apv1o+KmnW0aypjwmfFlarxjo=)
(SDK Attempt Count: 1)
```
Yet the uploads list afterwards finds it
```
job-00-fork-0005/test/testIfNoneMatchTwoConcurrentMultipartUploads
AnspJPHUoPJqg61t28OvLfAogi6G9ocyx1Dm6XY2C.a_H_onklM0Nr0LIXaPiYlQjZIiH0fTsQ1e2KhEjS9pGxvSKOXq_4YibiGZmFC6rBolmfACMqIRpoeaqYDgzYW4
```
I have to conclude that the list of pending uploads was briefly
offline/inconsistent.
This is presumably so, so rare that there's almost no point retrying here.
With no retries, every active write/job would have failed, even though the
system had recovered within a minute.
Maybe we should retry here? I remember a long long time ago the v1 sdk
didn't retry on failures of the final POST to commit an upload, and how that
sporadically caused problems. Retrying on MPU failures will allow for recovery
in the presence of a transient failure here, and the cost of "deletion of all
pending uploads will take longer to fail all active uploads".
> Upgrade AWS SDK to 2.33.x
> -------------------------
>
> Key: HADOOP-19654
> URL: https://issues.apache.org/jira/browse/HADOOP-19654
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, fs/s3
> Affects Versions: 3.5.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Upgrade to a recent version of 2.33.x or later while off the critical path of
> things.
> HADOOP-19485 froze the sdk at a version which worked with third party stores.
> Apparently the new version works; early tests show that Bulk Delete calls
> with third party stores complain about lack of md5 headers, so some tuning is
> clearly going to be needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
