Also, as you can see from the SSVM, the qcow2 file is there, see the bottom.
root@s-230-VM:/var/log# tail -f cloud.log
2024-01-05 16:12:45,395 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-3:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:12:55,755 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:13:45,639 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-4:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:13:56,099 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-5:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:14:45,879 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-1:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:14:56,345 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-3:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:15:46,167 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:15:56,611 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-4:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:16:46,363 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-5:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:16:56,951 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-1:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:17:46,610 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-3:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:17:46,928 INFO [handler.logging.LoggingHandler]
(nioEventLoopGroup-2-1:null) [id: 0xc2744c38, L:/0.0.0.0:8210] READ: [id:
0x1bc34b72, L:/127.0.0.1:8210 - R:/127.0.0.1:33898]
2024-01-05 16:17:46,930 INFO [handler.logging.LoggingHandler]
(nioEventLoopGroup-2-1:null) [id: 0xc2744c38, L:/0.0.0.0:8210] READ COMPLETE
2024-01-05 16:17:46,936 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-7:null) HEADER: SIGNATURE=xRbt2tIf27eRHe/H9MqxowS9QMI=
2024-01-05 16:17:46,937 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-7:null) HEADER:
METADATA=JvY7F8jeDr1zGrJiWz4RAx6XllfhiaRie1t/mp30Kri+I9rYaBaQieJQ4LUshDFPdQ+UsOmecgrc9Udeh5mLPU0VK+sJLhbmAVQFT6wcdMTgycHI/tvIPwQZYARuKsL2tjc8TwsiXkIuK6BeAro9+MHWvh8Z0H1XTrasyU4UAOvPL4hEOBVoGqalnz/XcC++4l00AjtDuoJwuhVn9sv20zV/4x7UiwVPyBAZCnyGvTh+whf5+hpj9eUKBAljxOetN00bW3Mcp8XNlwEnGAcr33n/CjdGeiMrN5+cMIKiBocfTQXOgeN7koDn7Aqvu0MOFXUOMG5F8AjaFb+D7I+YC9JEjlTippGKmRipKJ2/NCHV7DozS8vwb2O19u6VtvIxcpxsvrO8NVppAOojpFY4/ZrpEd0fpazrLAd3aJTI1jtxOQLtFjLE4iMVvvAGFPEntm7PqFKEjKCMBXuTESnz5QXCh4kg3bXKtSMxRKOm6cDDYLiXw++aWYK3w/bSke0sw/PBYfKGkhnyqLUPxqo+Drqacl+dnQErX/bTUFa9oAXH4LWftvqtqX8B7pwJ5bqz9NuJKXDxJzUSFORpC117CADegTo2B9eYLTw=
2024-01-05 16:17:46,938 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-7:null) HEADER: EXPIRES=2024-01-05T17:17:46.775Z
2024-01-05 16:17:46,938 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-7:null) HEADER: HOST=216-120-178-249.aigcloud.io
2024-01-05 16:17:46,938 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-7:null) HEADER: CONTENT_LENGTH=5509021929
2024-01-05 16:17:46,938 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-7:null) URI: uuid=a69107bb-fc76-4439-b553-c270f792945a
2024-01-05 16:17:46,941 INFO [storage.resource.NfsSecondaryStorageResource]
(nioEventLoopGroup-3-7:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:17:46,964 INFO [storage.resource.NfsSecondaryStorageResource]
(nioEventLoopGroup-3-7:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:17:49,141 INFO [storage.resource.NfsSecondaryStorageResource]
(nioEventLoopGroup-3-7:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:17:49,310 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-7:null) base directory:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239
2024-01-05 16:17:57,260 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:18:46,821 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-4:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:18:57,878 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-5:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:19:47,679 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-1:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:19:59,514 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-3:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:20:49,108 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:21:00,692 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-4:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:21:50,637 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-5:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:22:02,156 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-1:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:22:51,242 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-3:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:23:03,209 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-2:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:23:52,487 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-4:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:24:04,091 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-7:null) BODY FileUpload: FileUpload: content-disposition:
form-data; name="files[]"; filename="1b4f2ce0-0999-4314-ab69-348953432514.qcow2"
content-type: application/octet-stream; charset=UTF-8
content-length: 5509021696
Completed: true
IsInMemory: false
RealFile:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239/FUp_9513061883467712232_upload
DefaultDeleteAfter: true
2024-01-05 16:24:04,597 INFO [storage.resource.NfsSecondaryStorageResource]
(agentRequest-Handler-5:null) Determined host 10.22.1.10 corresponds to IP
10.22.1.10
2024-01-05 16:24:05,335 INFO [storage.template.OVAProcessor]
(nioEventLoopGroup-3-7:null) Template processing. templatePath:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239,
templateName: a9ba3e9f-16dc-375b-9781-1edb79159336
2024-01-05 16:24:05,360 INFO [storage.template.OVAProcessor]
(nioEventLoopGroup-3-7:null) Unable to find the vmware template file:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239/a9ba3e9f-16dc-375b-9781-1edb79159336.ova
2024-01-05 16:24:05,361 INFO [storage.template.VmdkProcessor]
(nioEventLoopGroup-3-7:null) Template processing. templatePath:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239,
templateName: a9ba3e9f-16dc-375b-9781-1edb79159336
2024-01-05 16:24:05,385 INFO [storage.template.VmdkProcessor]
(nioEventLoopGroup-3-7:null) Unable to find the vmware template file:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239/a9ba3e9f-16dc-375b-9781-1edb79159336.vmdk
2024-01-05 16:24:05,538 WARN [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-7:null)
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at
io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
at
io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
at
io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
at
io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935)
at
io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-01-05 16:24:05,540 WARN [netty.channel.AbstractChannelHandlerContext]
(nioEventLoopGroup-3-7:null) An exception 'java.lang.NullPointerException'
[enable DEBUG level for full stacktrace] was thrown by a user handler's
exceptionCaught() method while handling the following exception:
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at
io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
at
io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
at
io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
at
io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935)
at
io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
^Z
[1]+ Stopped tail -f cloud.log
root@s-230-VM:/var/log#
root@s-230-VM:/var/log# cd /mnt
root@s-230-VM:/mnt# ls
SecStorage
root@s-230-VM:/mnt# cd SecStorage/
root@s-230-VM:/mnt/SecStorage# ls
c7d010a7-ec39-344f-afab-f7bff31a309f extractmnt
root@s-230-VM:/mnt/SecStorage# cd c7d010a7-ec39-344f-afab-f7bff31a309f/
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f# ls
snapshots template volumes
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f# cd template
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template# ls
tmpl
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template# cd
templ
-bash: cd: templ: No such file or directory
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template# ls
-la
total 0
drwxr-xr-x 3 root root 0 Dec 4 17:11 .
drwxrwxrwt 5 root root 0 Dec 27 18:09 ..
drwxr-xr-x 5 root root 0 Dec 27 16:25 tmpl
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template# cd
tmpl
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl#
ls
1 2 7
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl#
cd 2
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2#
ls
205 208 218 221 225 232 233 234 238 239
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2#
cd 239
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239#
ls
a9ba3e9f-16dc-375b-9781-1edb79159336.qcow2 template.properties
root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239#
Thanks,
Brian
From: Brian Fossmeyer <[email protected]>
Date: Friday, January 5, 2024 at 5:41 AM
To: [email protected] <[email protected]>
Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload.
Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload
as SSVM is either destroyed or SSVM agent not in 'Up' state
.
No, that was the entire upload process. There are no other logs in that log
file and no errors. Any place else I can look?
________________________________
From: Jithin Raju
Sent: Thursday, January 4, 2024 11:44:30 PM
To: [email protected]
Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload.
Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload
as SSVM is either destroyed or SSVM agent not in 'Up' state
Hi Brian,
These stack traces are not expected and reported recently in github:
https://github.com/apache/cloudstack/issues/8376
But as per this issue ticket, this stack trace is not the one causing the
upload error mostly. Do you see anything else in cloud.log with an ERROR in the
upload process?
-Jithin
From: Brian Fossmeyer
Date: Friday, 5 January 2024 at 12:09 AM
To: [email protected]
Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload.
Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload
as SSVM is either destroyed or SSVM agent not in 'Up' state
Thanks for the heads up on the cloud.log. I have added the log portion from the
upload below. The thing that stands out to me is that the error states that
there is no vmware template and we are not using vmnware?
2024-01-04 18:32:08,970 INFO [handler.logging.LoggingHandler]
(nioEventLoopGroup-2-1:null) [id: 0xc2744c38, L:/0.0.0.0:8210] READ: [id:
0x9ea99be8, L:/127.0.0.1:8210 - R:/127.0.0.1:60612]
2024-01-04 18:32:08,971 INFO [handler.logging.LoggingHandler]
(nioEventLoopGroup-2-1:null) [id: 0xc2744c38, L:/0.0.0.0:8210] READ COMPLETE
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) HEADER: SIGNATURE=8ehXP50nEe++Y8kK9uLWFS4ez5s=
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) HEADER:
METADATA=zgZ2nuZ5Ml9E+OYDsKzLoZw0YIBzqwfH7i/wlz6vG3If1P4CLO/+MTvgAfYwBtUj+MUmfrgX6nKaQJJ9V92H0qyPXCfwu7e2JLOOIvTcrXOUYOQOApZDWLNGcqlSVtcNoWN0qlYhN1Ot/JHL6UnU3oGaGvuA03bTlAHIajTpLopXSJC0QAND5J7pdMSve5GeR+yDpN2mQMdDQZ25swM0zUd1SnAPuR3sprQO/MrE+eh7OOFbp6M86v7cWhq1Z6TYdaaD58woNaeyxr8kVCJBuLbG7mfMLbtFQ840nrZ9SC5ejATbMKUQvxYbHWjpyheePRHLhC9LR+nPI7TY05F3PMjkw1Ipt2Vh8Io1Q6Bb62D5ixFM5fWHn++dVi1onlg0oJL/F1BO2mX0+WB+46t8vfvZgJxf/00eoq7TdTaGgzoanj7JYNqLznVjCD+gzAVgoIjjMyVjDsIPuvXr40yraX7Cb7/1KbHLJtqybup5++euF7qbJ9hzqAUWVtVD+8VfseVMjnTHB2/c4WB2NhGxJ55sr82sIBN6as6WpveLTCDR//r2Hl+j8BfvwVbIyix5rhiCT4vc/Gk4/g0FWd/xjxXIEPsYw/izjA==
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) HEADER: EXPIRES=2024-01-04T19:32:08.683Z
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) HEADER: HOST=216-120-178-249.aigcloud.io
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) HEADER: CONTENT_LENGTH=2133391587
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) URI: uuid=330b70a5-258e-4e7c-a698-ad8ccfe2246f
2024-01-04 18:32:10,895 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) base directory:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234
2024-01-04 18:34:07,578 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) BODY FileUpload: FileUpload: content-disposition:
form-data; name="files[]"; filename="ubuntu-22.04.3-live-server-amd64.iso"
content-type: application/octet-stream; charset=UTF-8
content-length: 2133391360
Completed: true
IsInMemory: false
RealFile:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234/FUp_12970119428300643144_upload
DefaultDeleteAfter: true
2024-01-04 18:34:08,645 INFO [storage.template.OVAProcessor]
(nioEventLoopGroup-3-5:null) Template processing. templatePath:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234,
templateName: 330b70a5-258e-4e7c-a698-ad8ccfe2246f
2024-01-04 18:34:08,671 INFO [storage.template.OVAProcessor]
(nioEventLoopGroup-3-5:null) Unable to find the vmware template file:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234/330b70a5-258e-4e7c-a698-ad8ccfe2246f.ova
2024-01-04 18:34:08,671 INFO [storage.template.VmdkProcessor]
(nioEventLoopGroup-3-5:null) Template processing. templatePath:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234,
templateName: 330b70a5-258e-4e7c-a698-ad8ccfe2246f
2024-01-04 18:34:08,695 INFO [storage.template.VmdkProcessor]
(nioEventLoopGroup-3-5:null) Unable to find the vmware template file:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234/330b70a5-258e-4e7c-a698-ad8ccfe2246f.vmdk
2024-01-04 18:34:08,883 WARN [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null)
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at
io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
at
io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
at
io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
at
io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935)
at
io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-01-04 18:34:08,885 WARN [netty.channel.AbstractChannelHandlerContext]
(nioEventLoopGroup-3-5:null) An exception 'java.lang.NullPointerException'
[enable DEBUG level for full stacktrace] was thrown by a user handler's
exceptionCaught() method while handling the following exception:
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at
io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
at
io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
at
io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
at
io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935)
at
io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
Thanks,
Brian
From: Jithin Raju
Date: Wednesday, January 3, 2024 at 11:14 PM
To: [email protected]
Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload.
Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload
as SSVM is either destroyed or SSVM agent not in 'Up' state
Hi Brian,
Does the SSVM agent state shows ‘Up’? you should be able to find more
information in the cloud.log of the SSVM.
-Jithin
From: Brian Fossmeyer
Date: Thursday, 4 January 2024 at 12:39 AM
To: [email protected]
Subject: FW: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload.
Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload
as SSVM is either destroyed or SSVM agent not in 'Up' state
Hello,
I am trying to upload ISO’s from my desktop and having this issue. The ISO will
upload to 100%, but then will generate this error and then will not ever get
into a ready state. Both of the System VMs are in a running and in an up state.
Template failed to upload. Error details: Template failed to upload as SSVM is
either destroyed or SSVM agent not in 'Up' state
Thanks
Brian
Brian Fossmeyer
Senior Systems Engineer, Technology
[Answers in Genesis] [Creation Museum] [Ark Encounter]
