[jira] [Commented] (HDDS-1936) ozonesecure s3 test fails intermittently

Xu Jiang (Jira) Sun, 03 Jan 2021 02:32:05 -0800


    [ 
https://issues.apache.org/jira/browse/HDDS-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17257712#comment-17257712
 ]


Xu Jiang commented on HDDS-1936:
--------------------------------

Hi I encountered the same problem when testing presto to query Ozone，I will use 
S3 protocol，environmental information is as follows：
 * Deploy model: Single node by official docker image
 * Ozone version: 1.0.0 
 * Hive Metastore version: 3.0.0
 * Presto version: 0.246-SNAPSHOT

Error message: 

 

PrestoDB stack trace:

 

```

com.facebook.presto.spi.PrestoException: Error committing write to Hive at 
com.facebook.presto.hive.RecordFileWriter.commit(RecordFileWriter.java:187) at 
com.facebook.presto.hive.HiveWriter.commit(HiveWriter.java:102) at 
com.facebook.presto.hive.HivePageSink.doFinish(HivePageSink.java:229) at 
com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23)
 at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:81) at 
com.facebook.presto.hive.HivePageSink.finish(HivePageSink.java:220) at 
com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSink.finish(ClassLoaderSafeConnectorPageSink.java:74)
 at 
com.facebook.presto.operator.TableWriterOperator.finish(TableWriterOperator.java:252)
 at com.facebook.presto.operator.Driver.processInternal(Driver.java:440) at 
com.facebook.presto.operator.Driver.lambda$processFor$8(Driver.java:300) at 
com.facebook.presto.operator.Driver.tryWithLock(Driver.java:721) at 
com.facebook.presto.operator.Driver.processFor(Driver.java:293) at 
com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077)
 at 
com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
 at 
com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:545)
 at 
com.facebook.presto.$gen.Presto_null__testversion____20210103_092858_1.run(Unknown
 Source) at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: 
com.amazonaws.services.s3.model.AmazonS3Exception: Server Error (Service: 
Amazon S3; Status Code: 500; Error Code: 500 Server Error; Request ID: null; S3 
Extended Request ID: null), S3 Extended Request ID: null at 
com.facebook.presto.hive.s3.PrestoS3FileSystem$PrestoS3OutputStream.uploadObject(PrestoS3FileSystem.java:1255)
 at 
com.facebook.presto.hive.s3.PrestoS3FileSystem$PrestoS3OutputStream.close(PrestoS3FileSystem.java:1204)
 at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
 at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) 
at parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:400) at 
parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:117)
 at parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:112) at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.close(ParquetRecordWriterWrapper.java:102)
 at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.close(ParquetRecordWriterWrapper.java:119)
 at 
com.facebook.presto.hive.ParquetRecordWriterUtil$1.close(ParquetRecordWriterUtil.java:95)
 at com.facebook.presto.hive.RecordFileWriter.commit(RecordFileWriter.java:182) 
... 18 more Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: 
Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Server 
Error; Request ID: null; S3 Extended Request ID: null) at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1726)
 at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1381)
 at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127)
 at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784)
 at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:745)
 at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
 at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
 at 
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052) at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998) at 
com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394) at 
com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5940)
 at 
com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808) 
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768) 
at 
com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:131)
 at 
com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:123)
 at 
com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:143)
 at 
com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:48)
 at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at 
java.util.concurrent.FutureTask.run(FutureTask.java) ... 3 more

```

 

Ozone ozone-docker_s3g_1 log:

 

```

2021-01-03 10:20:09 WARN  HttpChannel:587 - handleException 
/tiny/customer/20210103_102003_00013_sdm72_91ce2645-c4ae-45cf-805a-417d1bce52b4 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks2021-01-03 10:20:09 WARN  HttpChannel:587 - 
handleException 
/tiny/customer/20210103_102003_00013_sdm72_91ce2645-c4ae-45cf-805a-417d1bce52b4 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks2021-01-03 10:20:09 WARN  HttpChannelState:768 - 
unhandled due to prior sendErrorjavax.servlet.ServletException: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)Caused by: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 ... 17 moreCaused by: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:278)
 at 
org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:260)
 at 
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:460)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:285) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) 
... 45 moreCaused by: INTERNAL_ERROR 
org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks. 
Requested 1 blocks at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
 at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:584)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:688) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) ... 
54 more2021-01-03 10:20:09 ERROR ObjectEndpoint:181 - Exception occurred in 
PutObjectINTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: 
Allocated 0 blocks. Requested 1 blocks at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
 at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:584)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:688) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)2021-01-03 10:20:09 WARN  
HttpChannel:587 - handleException 
/tiny/customer/20210103_102003_00013_sdm72_91ce2645-c4ae-45cf-805a-417d1bce52b4 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks2021-01-03 10:20:09 WARN  HttpChannelState:768 - 
unhandled due to prior sendErrorjavax.servlet.ServletException: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)Caused by: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 ... 17 moreCaused by: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:278)
 at 
org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:260)
 at 
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:460)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:285) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) 
... 45 moreCaused by: INTERNAL_ERROR 
org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks. 
Requested 1 blocks at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
 at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:584)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:688) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) ... 
54 more2021-01-03 10:20:11 ERROR ObjectEndpoint:181 - Exception occurred in 
PutObjectINTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: 
Allocated 0 blocks. Requested 1 blocks at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
 at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:584)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:688) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)2021-01-03 10:20:11 WARN  
HttpChannel:587 - handleException 
/tiny/customer/20210103_102003_00013_sdm72_91ce2645-c4ae-45cf-805a-417d1bce52b4 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks2021-01-03 10:20:11 WARN  HttpChannelState:768 - 
unhandled due to prior sendErrorjavax.servlet.ServletException: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)Caused by: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 ... 17 moreCaused by: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:278)
 at 
org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:260)
 at 
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:460)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:285) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) 
... 45 moreCaused by: INTERNAL_ERROR 
org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks. 
Requested 1 blocks at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
 at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:584)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:688) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) ... 
54 more2021-01-03 10:20:22 ERROR ObjectEndpoint:181 - Exception occurred in 
PutObjectINTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: 
Allocated 0 blocks. Requested 1 blocks at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
 at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:584)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:688) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)2021-01-03 10:20:23 WARN  
HttpChannel:587 - handleException 
/tiny/customer/20210103_102003_00013_sdm72_91ce2645-c4ae-45cf-805a-417d1bce52b4 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks2021-01-03 10:20:23 WARN  HttpChannelState:768 - 
unhandled due to prior sendErrorjavax.servlet.ServletException: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)Caused by: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 ... 17 moreCaused by: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:278)
 at 
org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:260)
 at 
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:460)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:285) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) 
... 45 moreCaused by: INTERNAL_ERROR 
org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks. 
Requested 1 blocks at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
 at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:584)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:688) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) ... 
54 more2021-01-03 10:20:36 WARN  ShellBasedUnixGroupsMapping:210 - unable to 
return groups for user ozonePartialGroupNameException The user name 'ozone' is 
not found. id: ozone: no such userid: ozone: no such user
 at 
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294)
 at 
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:207)
 at 
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
 at 
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51)
 at 
org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:387)
 at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:321) at 
org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:270) at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
 at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) 
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
 at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) at 
com.google.common.cache.LocalCache.get(LocalCache.java:3951) at 
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) 
at org.apache.hadoop.security.Groups.getGroups(Groups.java:228) at 
org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1588)
 at 
org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1576)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.getAclList(RpcClient.java:489) 
at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:671) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)2021-01-03 10:20:36 ERROR 
ObjectEndpoint:181 - Exception occurred in PutObjectINTERNAL_ERROR 
org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks. 
Requested 1 blocks at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
 at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:584)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:688) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)2021-01-03 10:20:36 WARN  
HttpChannel:587 - handleException 
/tiny/customer/20210103_102003_00013_sdm72_91ce2645-c4ae-45cf-805a-417d1bce52b4 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks2021-01-03 10:20:36 WARN  HttpChannelState:768 - 
unhandled due to prior sendErrorjavax.servlet.ServletException: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)Caused by: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 ... 17 moreCaused by: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:278)
 at 
org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:260)
 at 
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:460)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:285) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) 
... 45 moreCaused by: INTERNAL_ERROR 
org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks. 
Requested 1 blocks at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
 at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:584)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:688) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) ... 
54 more2021-01-03 10:20:56 ERROR ObjectEndpoint:181 - Exception occurred in 
PutObjectINTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: 
Allocated 0 blocks. Requested 1 blocks at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
 at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:584)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:688) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)2021-01-03 10:20:56 WARN  
HttpChannel:587 - handleException 
/tiny/customer/20210103_102003_00013_sdm72_91ce2645-c4ae-45cf-805a-417d1bce52b4 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks2021-01-03 10:20:56 WARN  HttpChannelState:768 - 
unhandled due to prior sendErrorjavax.servlet.ServletException: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at org.eclipse.jetty.server.Server.handle(Server.java:500) at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
 at java.base/java.lang.Thread.run(Thread.java:834)Caused by: 
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) at 
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
 at 
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
 at 
org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
 at 
org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
 at 
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at 
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 ... 17 moreCaused by: org.glassfish.jersey.server.ContainerException: 
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 
blocks. Requested 1 blocks at 
org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:278)
 at 
org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:260)
 at 
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:460)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:285) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at 
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at 
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) 
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
 at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) 
... 45 moreCaused by: INTERNAL_ERROR 
org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks. 
Requested 1 blocks at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
 at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:584)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:688) 
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:396) 
at 
org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:168) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) ... 
54 more

```

I want to know if S3 function is not fully supported yet, can we get around 
this problem?

 

> ozonesecure s3 test fails intermittently
> ----------------------------------------
>
>                 Key: HDDS-1936
>                 URL: https://issues.apache.org/jira/browse/HDDS-1936
>             Project: Hadoop Distributed Data Store
>          Issue Type: Sub-task
>          Components: test
>            Reporter: Attila Doroszlai
>            Priority: Major
>              Labels: TriagePending
>
> Sometimes acceptance tests fail at ozonesecure s3 test, starting with:
> {code:title=https://ci.anzix.net/job/ozone/17607/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s18-s1-t1-k3-k1-k2}
> Completed 29 Bytes/29 Bytes (6 Bytes/s) with 1 file(s) remaining
> upload failed: ../../tmp/testfile to s3://bucket-07853/testfile An error 
> occurred (500) when calling the PutObject operation (reached max retries: 4): 
> Internal Server Error
> {code}
> followed by:
> {code:title=https://ci.anzix.net/job/ozone/17607/artifact/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/log.html#s1-s18-s5-t1}
> ('Connection aborted.', error(32, 'Broken pipe'))
> {code}
> in subsequent test cases.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

[jira] [Commented] (HDDS-1936) ozonesecure s3 test fails intermittently

Reply via email to