Hi Brian,
These stack traces are not expected and reported recently in github:
https://github.com/apache/cloudstack/issues/8376
But as per this issue ticket, this stack trace is not the one causing the
upload error mostly. Do you see anything else in cloud.log with an ERROR in the
upload process?
-Jithin
From: Brian Fossmeyer <bfossme...@answersingenesis.org.INVALID>
Date: Friday, 5 January 2024 at 12:09 AM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload.
Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload
as SSVM is either destroyed or SSVM agent not in 'Up' state
Thanks for the heads up on the cloud.log. I have added the log portion from the
upload below. The thing that stands out to me is that the error states that
there is no vmware template and we are not using vmnware?
2024-01-04 18:32:08,970 INFO [handler.logging.LoggingHandler]
(nioEventLoopGroup-2-1:null) [id: 0xc2744c38, L:/0.0.0.0:8210] READ: [id:
0x9ea99be8, L:/127.0.0.1:8210 - R:/127.0.0.1:60612]
2024-01-04 18:32:08,971 INFO [handler.logging.LoggingHandler]
(nioEventLoopGroup-2-1:null) [id: 0xc2744c38, L:/0.0.0.0:8210] READ COMPLETE
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) HEADER: SIGNATURE=8ehXP50nEe++Y8kK9uLWFS4ez5s=
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) HEADER:
METADATA=zgZ2nuZ5Ml9E+OYDsKzLoZw0YIBzqwfH7i/wlz6vG3If1P4CLO/+MTvgAfYwBtUj+MUmfrgX6nKaQJJ9V92H0qyPXCfwu7e2JLOOIvTcrXOUYOQOApZDWLNGcqlSVtcNoWN0qlYhN1Ot/JHL6UnU3oGaGvuA03bTlAHIajTpLopXSJC0QAND5J7pdMSve5GeR+yDpN2mQMdDQZ25swM0zUd1SnAPuR3sprQO/MrE+eh7OOFbp6M86v7cWhq1Z6TYdaaD58woNaeyxr8kVCJBuLbG7mfMLbtFQ840nrZ9SC5ejATbMKUQvxYbHWjpyheePRHLhC9LR+nPI7TY05F3PMjkw1Ipt2Vh8Io1Q6Bb62D5ixFM5fWHn++dVi1onlg0oJL/F1BO2mX0+WB+46t8vfvZgJxf/00eoq7TdTaGgzoanj7JYNqLznVjCD+gzAVgoIjjMyVjDsIPuvXr40yraX7Cb7/1KbHLJtqybup5++euF7qbJ9hzqAUWVtVD+8VfseVMjnTHB2/c4WB2NhGxJ55sr82sIBN6as6WpveLTCDR//r2Hl+j8BfvwVbIyix5rhiCT4vc/Gk4/g0FWd/xjxXIEPsYw/izjA==
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) HEADER: EXPIRES=2024-01-04T19:32:08.683Z
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) HEADER: HOST=216-120-178-249.aigcloud.io
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) HEADER: CONTENT_LENGTH=2133391587
2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) URI: uuid=330b70a5-258e-4e7c-a698-ad8ccfe2246f
2024-01-04 18:32:10,895 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) base directory:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234
2024-01-04 18:34:07,578 INFO [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null) BODY FileUpload: FileUpload: content-disposition:
form-data; name="files[]"; filename="ubuntu-22.04.3-live-server-amd64.iso"
content-type: application/octet-stream; charset=UTF-8
content-length: 2133391360
Completed: true
IsInMemory: false
RealFile:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234/FUp_12970119428300643144_upload
DefaultDeleteAfter: true
2024-01-04 18:34:08,645 INFO [storage.template.OVAProcessor]
(nioEventLoopGroup-3-5:null) Template processing. templatePath:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234,
templateName: 330b70a5-258e-4e7c-a698-ad8ccfe2246f
2024-01-04 18:34:08,671 INFO [storage.template.OVAProcessor]
(nioEventLoopGroup-3-5:null) Unable to find the vmware template file:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234/330b70a5-258e-4e7c-a698-ad8ccfe2246f.ova
2024-01-04 18:34:08,671 INFO [storage.template.VmdkProcessor]
(nioEventLoopGroup-3-5:null) Template processing. templatePath:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234,
templateName: 330b70a5-258e-4e7c-a698-ad8ccfe2246f
2024-01-04 18:34:08,695 INFO [storage.template.VmdkProcessor]
(nioEventLoopGroup-3-5:null) Unable to find the vmware template file:
/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234/330b70a5-258e-4e7c-a698-ad8ccfe2246f.vmdk
2024-01-04 18:34:08,883 WARN [storage.resource.HttpUploadServerHandler]
(nioEventLoopGroup-3-5:null)
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at
io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
at
io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
at
io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
at
io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935)
at
io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-01-04 18:34:08,885 WARN [netty.channel.AbstractChannelHandlerContext]
(nioEventLoopGroup-3-5:null) An exception 'java.lang.NullPointerException'
[enable DEBUG level for full stacktrace] was thrown by a user handler's
exceptionCaught() method while handling the following exception:
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at
io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
at
io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
at
io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
at
io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952)
at
io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935)
at
io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222)
at
org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
Thanks,
Brian
From: Jithin Raju <jithin.r...@shapeblue.com>
Date: Wednesday, January 3, 2024 at 11:14 PM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload.
Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload
as SSVM is either destroyed or SSVM agent not in 'Up' state
Hi Brian,
Does the SSVM agent state shows ‘Up’? you should be able to find more
information in the cloud.log of the SSVM.
-Jithin
From: Brian Fossmeyer <bfossme...@answersingenesis.org.INVALID>
Date: Thursday, 4 January 2024 at 12:39 AM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: FW: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload.
Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload
as SSVM is either destroyed or SSVM agent not in 'Up' state
Hello,
I am trying to upload ISO’s from my desktop and having this issue. The ISO will
upload to 100%, but then will generate this error and then will not ever get
into a ready state. Both of the System VMs are in a running and in an up state.
Template <I REMOVED TEMPLATE ID> failed to upload. Error details: Template <I
REMOVED TEMPLATE ID> failed to upload as SSVM is either destroyed or SSVM agent
not in 'Up' state
Thanks
Brian
Brian Fossmeyer
Senior Systems Engineer, Technology
[Answers in Genesis]<https://answersingenesis.org/> [Creation Museum]
<https://creationmuseum.org/> [Ark Encounter] <https://arkencounter.com/>
