[
https://issues.apache.org/jira/browse/HDDS-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17730151#comment-17730151
]
Attila Doroszlai commented on HDDS-8772:
----------------------------------------
[~sadanand_shenoy], [~szetszwo],
Steps to reproduce:
# start secure cluster ({{ozonesecure}} compose environment)
# create volume and bucket with quota (no keys)
# put file with streaming enabled (and above threshold)
* If the container does not exist and quota is set on the bucket: OM will
eventually reject the request with quota exceeded.
* If the container already exists (by having created other keys), then write
eventually succeeds (despite the missing block token).
{code}
cd hadoop-ozone/dist/target/ozone-1.4.0-SNAPSHOT/compose/ozonesecure
docker-compose up -d --scale datanode=3
sleep 10
docker-compose exec scm bash
# steps in container:
kinit -k -t /etc/security/keytabs/testuser.keytab 'testuser/[email protected]'
ozone admin safemode wait
ozone sh volume create vol1
ozone sh bucket create --space-quota 10GB --layout FILE_SYSTEM_OPTIMIZED
vol1/bucket1
OZONE_LOGLEVEL=INFO ozone fs -Dozone.fs.datastream.auto.threshold=1KB
-Dozone.fs.datastream.enabled=true -put NOTICE.txt ofs://om/vol1/bucket1/
{code}
FS client output:
{code}
...
2023-06-07 14:42:20,373 [main] WARN storage.BlockDataStreamOutput: Failed to
write all chunks through stream: java.util.concurrent.ExecutionException:
java.io.IOException: result is not success
...
2023-06-07 14:42:20,892 [main] WARN storage.BlockDataStreamOutput: Failed to
write all chunks through stream: java.util.concurrent.ExecutionException:
java.io.IOException: result is not success
put: QUOTA_EXCEEDED org.apache.hadoop.ozone.om.exceptions.OMException: The
DiskSpace quota of bucket:bucket1 exceeded quotaInBytes: 10737418240 Bytes but
diskspace consumed: 11274289152 Bytes.
{code}
Datanode log (exception repeated few times):
{code}
2023-06-07 14:42:20,263 [ContainerOp-08c04320-609f-4eb3-9e65-fbce10b10143-0]
WARN impl.HddsDispatcher: Operation: StreamInit , Trace ID: , Message: Block
token verification failed. Failed to find any token (empty or null.) , Result:
BLOCK_TOKEN_VERIFICATION_FAILED , StorageContainerException Occurred.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
Block token verification failed. Failed to find any token (empty or null.)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:217)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:173)
at
org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:172)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:439)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:449)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.getStreamDataChannel(ContainerStateMachine.java:562)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$stream$4(ContainerStateMachine.java:577)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.hadoop.hdds.security.token.BlockTokenException: Failed to
find any token (empty or null.)
at
org.apache.hadoop.hdds.security.token.TokenVerifier.verify(TokenVerifier.java:60)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateToken(HddsDispatcher.java:462)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:214)
... 11 more
2023-06-07 14:42:20,363 [NettyClientStreamRpc-workerGroup--thread1] WARN
server.DataStreamManagement: Failed to process
DataStreamRequestByteBuf:clientId=client-C4FE60D39A2D,type=STREAM_DATA,id=0,offset=0,length=17289
java.util.concurrent.CompletionException: Failed to create data stream
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$stream$4(ContainerStateMachine.java:582)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by:
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
ContainerID 1 does not exist
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.getStreamDataChannel(HddsDispatcher.java:734)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.getStreamDataChannel(ContainerStateMachine.java:563)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$stream$4(ContainerStateMachine.java:577)
... 4 more
{code}
Client output if container already exists:
{code}
2023-06-07 14:48:34,058 [timer6] ERROR impl.OrderedStreamAsync: Failed to send
request,
header=DataStreamRequestHeader:clientId=client-C67FF2CE3664,type=STREAM_DATA,id=0,offset=17948,length=4
org.apache.ratis.protocol.exceptions.TimeoutIOException: Timeout 20000ms:
Failed to send
DataStreamWindowRequest:seqNum=4,DataStreamRequestHeader:clientId=client-C67FF2CE3664,type=STREAM_DATA,id=0,offset=17948,length=4
at
org.apache.ratis.client.impl.OrderedStreamAsync.lambda$scheduleWithTimeout$7(OrderedStreamAsync.java:172)
at
org.apache.ratis.util.TimeoutTimer.lambda$onTimeout$2(TimeoutTimer.java:101)
at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:38)
at org.apache.ratis.util.LogUtils$1.run(LogUtils.java:79)
at org.apache.ratis.util.TimeoutTimer$Task.run(TimeoutTimer.java:55)
at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
at java.base/java.util.TimerThread.run(Timer.java:506)
2023-06-07 14:48:34,061 [timer6] WARN storage.BlockDataStreamOutput: Failed
executePutBlockClose, reply=null
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.exceptions.TimeoutIOException: Timeout 20000ms:
Failed to send
DataStreamWindowRequest:seqNum=4,DataStreamRequestHeader:clientId=client-C67FF2CE3664,type=STREAM_DATA,id=0,offset=17948,length=4
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at
org.apache.ratis.client.impl.OrderedStreamAsync.lambda$scheduleWithTimeout$7(OrderedStreamAsync.java:172)
at
org.apache.ratis.util.TimeoutTimer.lambda$onTimeout$2(TimeoutTimer.java:101)
at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:38)
at org.apache.ratis.util.LogUtils$1.run(LogUtils.java:79)
at org.apache.ratis.util.TimeoutTimer$Task.run(TimeoutTimer.java:55)
at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
at java.base/java.util.TimerThread.run(Timer.java:506)
Caused by: org.apache.ratis.protocol.exceptions.TimeoutIOException: Timeout
20000ms: Failed to send
DataStreamWindowRequest:seqNum=4,DataStreamRequestHeader:clientId=client-C67FF2CE3664,type=STREAM_DATA,id=0,offset=17948,length=4
... 7 more
{code}
> Failed to create data stream:
> hdds.scm.container.common.helpers.StorageContainerException: ContainerID #
> does not exist
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-8772
> URL: https://issues.apache.org/jira/browse/HDDS-8772
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: George Huang
> Priority: Major
>
> {code:java}
> HddsDispatcher
> Operation: StreamInit , Trace ID: , Message: Block token verification
> failed. Failed to find any token (empty or null.) , Result:
> BLOCK_TOKEN_VERIFICATION_FAILED , StorageContainerException Occurred.
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Block token verification failed. Failed to find any token (empty or null.)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:215)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:171)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:170)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:439)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:449)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.getStreamDataChannel(ContainerStateMachine.java:562)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$stream$4(ContainerStateMachine.java:577)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.hadoop.hdds.security.token.BlockTokenException: Failed
> to find any token (empty or null.)
> at
> org.apache.hadoop.hdds.security.token.TokenVerifier.verify(TokenVerifier.java:60)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateToken(HddsDispatcher.java:460)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:212)
> ... 11 more
> |DataStreamManagement|{code:java}
> Failed to process
> DataStreamRequestByteBuf:clientId=client-6F7180804FBC,type=STREAM_DATA,id=2499,offset=0,length=112
> java.util.concurrent.CompletionException: Failed to create data stream
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$stream$4(ContainerStateMachine.java:582)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by:
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> ContainerID 392 does not exist
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.getStreamDataChannel(HddsDispatcher.java:715)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.getStreamDataChannel(ContainerStateMachine.java:563)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$stream$4(ContainerStateMachine.java:577)
> ... 4 more{code}|
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
