[
https://issues.apache.org/jira/browse/HADOOP-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18085571#comment-18085571
]
ASF GitHub Bot commented on HADOOP-19569:
-----------------------------------------
steveloughran opened a new pull request, #8528:
URL: https://github.com/apache/hadoop/pull/8528
HADOOP-19569.
Executors in hadoop-common to
- pick up shutdown of inner executor and shut themselves down.
- semaphore executor to decrement counters in this process so that queue
state is updated This stops callers being able to submit work when the inner
executor has shut down.
S3A code
- All S3 write operations moved from S3AFS down to
`org.apache.hadoop.fs.s3a.impl.write.StoreWriter` implementation. This is a
Service with its lifecycle managed in StoreImpl.
-
- StoreImpl to IllegalStateException on method invocation whene the service
isn't running. Some methods are kept open as they do seem needed.
- WriteOperationHelper callbacks to raise IllegalStateException when invoked
when FS is closed.
S3AStore has a new interface
```java
<S extends Service> S lookupService(String name, Class<S> serviceClass);
```
This looks up a service by name, with well known names for various managed
services, here StoreWriter. This a clear re-implementation of COM `IUnknown`
which itself is probably from Apollo RPC. Stops Store interface needing to wrap
and forward interfaces of hosted services, instead just serves up access to the
impl. Should have done this earlier, maybe even with the Service API.
### Copilot review
This PR addresses HADOOP-19569 by enhancing the S3A filesystem’s behavior
when closed, ensuring that write/close operations fail gracefully, and improves
internal service registration and executor shutdown behavior. Key changes
include enforcing FS state checks via added checkRunning() calls, refactoring
helper and callback methods (e.g. renaming getWriteOperationHelper to
createWriteOperationHelperWithinActiveSpan and replacing direct FS calls with
getStore() invocations), and cleaning up legacy utilities such as
MultipartUtils.
### How was this patch tested?
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
PR predates my AI code use.
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> S3A: stream write/close fails badly once FS is closed
> -----------------------------------------------------
>
> Key: HADOOP-19569
> URL: https://issues.apache.org/jira/browse/HADOOP-19569
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.5.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> when closing a process during a large upload, and NPE is triggered in the
> abort call. This is because the S3 client has already been released.
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$abortMultipartUpload$41(S3AFileSystem.java:5337)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.abortMultipartUpload(S3AFileSystem.java:5336)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$abortMultipartUpload$4(WriteOperationHelper.java:392)
> {code}
> * close() in small writes also fails, just with a different exception
> * and on some large writes, the output stream hangs as it awaits the end of
> the queued writes. This is a problem inside the semaphore executor
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
