[ https://issues.apache.org/jira/browse/HDDS-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bharat Viswanadham resolved HDDS-3995. -------------------------------------- Fix Version/s: 1.1.0 Resolution: Fixed > s3g met NPE exception while write file by multiPartUpload > --------------------------------------------------------- > > Key: HDDS-3995 > URL: https://issues.apache.org/jira/browse/HDDS-3995 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: S3 > Affects Versions: 0.5.0 > Reporter: maobaolong > Assignee: maobaolong > Priority: Major > Labels: pull-request-available, s3g > Fix For: 1.1.0 > > > *strong text*The following is the exception > > 2020-07-20 17:26:17,692 > [java.util.concurrent.ThreadPoolExecutor$Worker@2b70f372[State = -1, empty > queue]] ERROR org.apache.hadoop.hdds.scm.storage.BlockOutputStream: writing > chunk failed 104545451305649559_chunk_1 blockID conID: 284442 locID: > 104545451305649559 bcsId: 0 with exception > org.apache.ratis.protocol.AlreadyClosedException: > SlidingWindow$Client:client-96F1E0530C66->RAFT is closed. > 2020-07-20 17:26:17,700 > [java.util.concurrent.ThreadPoolExecutor$Worker@2b70f372[State = -1, empty > queue]] ERROR org.apache.hadoop.hdds.scm.storage.BlockOutputStream: writing > chunk failed 104545451305649559_chunk_2 blockID conID: 284442 locID: > 104545451305649559 bcsId: 0 with exception > org.apache.ratis.protocol.AlreadyClosedException: > SlidingWindow$Client:client-96F1E0530C66->RAFT is closed. > 2020-07-20 17:26:17,700 > [java.util.concurrent.ThreadPoolExecutor$Worker@2b70f372[State = -1, empty > queue]] ERROR org.apache.hadoop.hdds.scm.storage.BlockOutputStream: writing > chunk failed 104545451305649559_chunk_3 blockID conID: 284442 locID: > 104545451305649559 bcsId: 0 with exception > org.apache.ratis.protocol.AlreadyClosedException: > SlidingWindow$Client:client-96F1E0530C66->RAFT is closed. > 2020-07-20 17:26:17,726 [qtp2131952342-276] WARN > org.apache.hadoop.ozone.client.io.KeyOutputStream: Encountered exception > java.io.IOException: Unexpected Storage Container Exception: > java.util.concurrent.CompletionException: > java.util.concurrent.CompletionException: > org.apache.ratis.protocol.AlreadyClosedException: > SlidingWindow$Client:client-96F1E0530C66->RAFT is closed. on the pipeline > Pipeline[ Id: 58e4eef0-d2b1-4246-a152-83a28012848f, Nodes: > 38103930-07b9-4434-a657-621276c65683\{ip: 9.180.20.47, host: > host-9-180-20-47, networkLocation: /rack1, certSerialId: > null}207b98d9-ad64-45a8-940f-504b514feff5\{ip: 9.180.21.88, host: > host-9-180-21-88, networkLocation: /rack2, certSerialId: > null}d3336357-8920-4a4e-a12f-e57da1640c4d\{ip: 9.180.20.94, host: > host-9-180-20-94, networkLocation: /rack1, certSerialId: null}, Type:RATIS, > Factor:THREE, State:OPEN, leaderId:207b98d9-ad64-45a8-940f-504b514feff5, > CreationTimestamp2020-07-07T15:11:10.887Z]. The last committed block length > is 0, uncommitted data length is 10485760 retry count 0 > 2020-07-20 17:26:17,726 [qtp2131952342-276] INFO > org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool: Allocating > block with ExcludeList \{datanodes = [], containerIds = [], pipelineIds = > [PipelineID=58e4eef0-d2b1-4246-a152-83a28012848f]} > 2020-07-20 17:26:18,238 [qtp2131952342-276] WARN > org.eclipse.jetty.server.HttpChannel: /bucketmbl/114.dat > javax.servlet.ServletException: javax.servlet.ServletException: > java.lang.NullPointerException > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at org.eclipse.jetty.server.Server.handle(Server.java:500) > at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) > at java.lang.Thread.run(Thread.java:748) > Caused by: javax.servlet.ServletException: java.lang.NullPointerException > at > org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) > at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) > at > org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617) > at > org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596) > at > org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1645) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) > at > org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) > ... 17 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.createMultipartKey(ObjectEndpoint.java:565) > at > org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:136) > at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) > at > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) > at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) > at org.glassfish.jersey.internal.Errors.process(Errors.java:316) > at org.glassfish.jersey.internal.Errors.process(Errors.java:298) > at org.glassfish.jersey.internal.Errors.process(Errors.java:268) > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) > at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) > at > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) > at > org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) > ... 45 more > After switch the log level to TRACE, i see the following error > 2020-07-21 15:31:03,212 [qtp1150963491-534] ERROR exception.S3ErrorTable: > <?xml version="1.0" encoding="UTF-8"?> > <Error> > <Code>NoSuchUpload</Code> > <Message>The specified multipart upload does not exist. The upload ID might > be invalid, or the multipart upload might have been aborted or > completed.</Message> > <Resource>828e5250-d3e5-42dd-a273-e2c687293e2d-104550574711057985</Resource> > <RequestId/> > </Error> -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org