RosiKyu opened a new issue, #12489: URL: https://github.com/apache/cloudstack/issues/12489
### problem **Related Issue** Related to https://github.com/apache/cloudstack/issues/12486 **Summary** When uploading templates/ISOs from local file via the UI, the upload completes successfully on the SSVM (file is written to disk, SSVM returns HTTP 200 "upload successful"), but a Netty IllegalReferenceCountException occurs during response handling. This causes the UI to display an error to the user even though the upload actually succeeded. The template eventually becomes "Ready" after background processing completes. ### versions _No response_ ### The steps to reproduce the bug 1. Navigate to Images → Templates 2. Click "Upload Template from local" 4. Fill in the form and submit 5. Watch browser Network tab - upload returns 200 OK 6. Check browser Console - may show error despite 200 OK 7. Check SSVM logs - see Netty IllegalReferenceCountException 8. Wait - template eventually becomes Ready Note: This issue is intermittent. **Expected Behavior** When the upload succeeds (HTTP 200, file written to disk), the UI should show success to the user. **Actual Behavior** - Upload succeeds (HTTP 200, "upload successful") - File is written to disk correctly - Netty throws IllegalReferenceCountException during cleanup - UI may show error to user (HTTP 432 in some cases) - Template eventually becomes Ready after background processing **Impact** - User confusion: User sees error but upload actually worked - Duplicate uploads: User may retry, creating duplicate templates - Wasted time: User troubleshoots non-existent problem **Root Cause Analysis** The IllegalReferenceCountException occurs in `HttpUploadServerHandler.channelUnregistered()` when trying to destroy the `HttpPostRequestDecoder`. The decoder's reference count has already reached 0, indicating it was already released elsewhere. This is a reference counting bug in the Netty upload handler - the decoder is being released twice: - Once during normal processing - Again in `channelUnregistered()` cleanup **Evidence** Browser Network Tab - Upload Succeeded (HTTP 200) Request Headers (all correct): ``` X-Expires: 2026-01-21T16:46:06.391Z X-Metadata: 7d9tasmSFbD/i/a+7cIPVHUKLB5oZUTfHOf6nIwEo6gdgJ1IHcgXfSHe1TouKFGRARpVRJJyyfVX5fqGcsv14oYurocRB4KGOkuW6zfW3GZC3hxhM2vA5OjK5Q497cxlc/TkgD/BOYoyx3VS32w4jFFxfVty9YSrc2tx1/bo9kE4MFXbFiuGNHI799xK8WhEXiTuR/7iyxRqLM17+ql/JYxlPLSP7DWbVOMFwm7HACbmSSaUNyJhi2OiaOmmU0TC5jpezvLxXhVmDsl9Fe9/uhTG2PDtmd/d2L+QtQz8+zNqxwgrS8EJqJKsk7q7tPzoDFKWc7Qp0uYpPvMqpyd377kXA2qhKnphEKrB6rBvIHV0OaX071UMNSysfrYC+GOIi3smUK6p5x939GlNlVFI6kYM5pkVZ9AsuNzGii4SMDZc8UGYrYmhazT10+ye0IlAVw5cFu5hcwVmeGnK8O2xDQkHo1z5wFYxiqCzZSuVdrhk0jdV0ktZ0CAcQeZzjBblu4DWFCcdqRDHl+8SkeAwtUjV3ZDht0u4D1lQdgV4G/NeDPCPhkm61WhKcR4AHvq5pb1+bs9h/8+b2BkuhfNbdNCd/gjW2kMsl4KcbhQwB+3uQsxZ9srg+hKacPB2Cyd8kgdLzuC4zYBq8mv8pRg4gpj5fbVC5Vs7pYbzpDlfoeIAuCZS7pDJM4Jontlae/QoGGo5 X-Signature: zw+vnaAEiYNsjUfHov44KNZ/GBw= Content-Length: 400621794 ``` - Response: ``` Status Code: 200 OK Body: upload successful. ``` - SSVM Logs (/var/log/cloud.log) - Upload Received and Processed ``` 2026-01-21T15:46:06,551 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: SIGNATURE=zw+vnaAEiYNsjUfHov44KNZ/GBw= 2026-01-21T15:46:06,552 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: METADATA=7d9tasmSFbD/i/a+7cIPVHUKLB5oZUT... 2026-01-21T15:46:06,552 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: EXPIRES=2026-01-21T16:46:06.391Z 2026-01-21T15:46:06,552 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: HOST=10.0.56.181 2026-01-21T15:46:06,553 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: CONTENT_LENGTH=400621794 2026-01-21T15:46:06,554 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) URI: uuid=6c80606a-8826-460f-88a8-c15ed3606a6c 2026-01-21T15:46:06,555 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) base directory: /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203 2026-01-21T15:46:06,611 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) BODY: FileUpload: name="file"; filename="linux-debian-11-x86_64-gen2-v1.qcow2" ``` - SSVM Logs - Netty Exception During Cleanup ``` 2026-01-21T15:47:30,572 WARN [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) Decoder already destroyed io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83) at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:140) at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:79) at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:293) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelUnregistered(HttpUploadServerHandler.java:169) ``` - File Successfully Written to SSVM ``` root@s-1-VM:~# ls -la /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203/ total 391236 drwxrwxrwx 2 root root 95 Jan 21 2026 . drwxr-xr-x 3 root root 25 Jan 21 2026 .. -rw-rw-rw- 1 root root 400621568 Jan 21 2026 c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2 -rw-rw-rw- 1 root root 340 Jan 21 2026 template.properties ``` - Template Properties File Created ``` root@s-1-VM:~# cat /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203/template.properties # #Wed Jan 21 15:47:30 UTC 2026 uniquename=203-2-0cb72ddc-d73a-3f2c-9f1f-24af1d033391 qcow2.filename=c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2 qcow2.virtualsize=5368709120 description=test qcow2.size=400621568 filename=c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2 size=400621568 public=true id=203 hvm=true virtualsize=5368709120 qcow2=true ``` - Despite the error shown to the user, the template eventually became Ready: ``` { "id": "6c80606a-8826-460f-88a8-c15ed3606a6c", "name": "Debian 11", "isready": true, "status": "Download Complete", "downloaddetails": [ { "datastore": "NFS://10.0.32.4/acs/secondary/ref-trl-10677-k-Mr8-rositsa-kyuchukova/ref-trl-10677-k-Mr8-rositsa-kyuchukova-sec1", "downloadPercent": "100", "downloadState": "DOWNLOADED" } ], "physicalsize": 400621568, "size": 5368709120 } ``` ### What to do about it? In H`ttpUploadServerHandler.java`, add a null check or try-catch around the decoder destruction in `channelUnregistered()` **Workaround** Ignore the error in the UI and wait - template will eventually become Ready Refresh the template list after a minute to see actual status -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
