Hey Midhun, This error is you see is coming from the XenServer, looks like it’s not able to mount the NFS on the destination directory, I know you have verified the mount point but can you try to manually mount your secondary storage on XenServer, also try checking the global settings - secstorage.nfs.version to check and enforce your nfs version.
Regards, Vivek Kumar > On 05-Apr-2022, at 11:25 AM, Midhun Jose <[email protected]> wrote: > > > Hi @all, > > I am using Apache cloudstack 4.16.1 integrated with XCP-ng 8.2.1. I am not > able to take VM volume snapshot and sometimes vm creation is also not > working. I am getting the error as attached screenshot and while checking the > logs we are getting below error, > > P.S: I have veriifed that nfs server is working fine and secondary storage is > mounted in xen server. Can someome piont out the reason on this.. > > ~~~~~~ > 2022-04-04 13:13:44,163 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-17:ctx-f6381170) (logid:4835671f) Task failed! Task record: > uuid: e907d695-825f-24e4-5ce3-afd74eb0ece4 > nameLabel: Async.host.call_plugin > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon Apr 04 13:13:43 IST 2022 > finished: Mon Apr 04 13:13:43 IST 2022 > status: failure > residentOn: com.xensource.xenapi.Host@5fa0255a > progress: 1.0 > type: <none/> > result: > errorInfo: [XENAPI_PLUGIN_FAILURE, mountNfsSecondaryStorage, > SROSError, Error reporting error, unknown key Unexpected error while trying > to mount 10.31.9.226:/storage/cloudstack/secondary/ to > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > 2022-04-04 13:13:44,173 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-17:ctx-f6381170) (logid:4835671f) callHostPlugin failed for cmd: > mountNfsSecondaryStorage with args remoteDir: > 10.31.9.226:/storage/cloudstack/secondary/, localDir: > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3, nfsVersion: null, due > to Task failed! Task record: uuid: > e907d695-825f-24e4-5ce3-afd74eb0ece4 > nameLabel: Async.host.call_plugin > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon Apr 04 13:13:43 IST 2022 > finished: Mon Apr 04 13:13:43 IST 2022 > status: failure > residentOn: com.xensource.xenapi.Host@5fa0255a > progress: 1.0 > type: <none/> > result: > errorInfo: [XENAPI_PLUGIN_FAILURE, mountNfsSecondaryStorage, > SROSError, Error reporting error, unknown key Unexpected error while trying > to mount 10.31.9.226:/storage/cloudstack/secondary/ to > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > Task failed! Task record: uuid: > e907d695-825f-24e4-5ce3-afd74eb0ece4 > nameLabel: Async.host.call_plugin > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon Apr 04 13:13:43 IST 2022 > finished: Mon Apr 04 13:13:43 IST 2022 > status: failure > residentOn: com.xensource.xenapi.Host@5fa0255a > progress: 1.0 > type: <none/> > result: > errorInfo: [XENAPI_PLUGIN_FAILURE, mountNfsSecondaryStorage, > SROSError, Error reporting error, unknown key Unexpected error while trying > to mount 10.31.9.226:/storage/cloudstack/secondary/ to > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > Task failed! Task record: uuid: > e907d695-825f-24e4-5ce3-afd74eb0ece4 > nameLabel: Async.host.call_plugin > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon Apr 04 13:13:43 IST 2022 > finished: Mon Apr 04 13:13:43 IST 2022 > status: failure > residentOn: com.xensource.xenapi.Host@5fa0255a > progress: 1.0 > type: <none/> > result: > errorInfo: [XENAPI_PLUGIN_FAILURE, mountNfsSecondaryStorage, > SROSError, Error reporting error, unknown key Unexpected error while trying > to mount 10.31.9.226:/storage/cloudstack/secondary/ to > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:500) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:422) > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:80) > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:249) > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:286) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:95) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:59) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:36) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:30) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1758) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-04-04 13:13:44,180 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-17:ctx-f6381170) (logid:4835671f) unable to destroy > task(com.xensource.xenapi.Task@3cd54231) on > host(bc164e35-7efe-4ff7-9460-1c92cdb44396) due to You gave an invalid object > reference. The object may have recently been deleted. The class parameter > gives the type of reference given, and the handle parameter echoes the bad > value given. > 2022-04-04 13:13:44,180 WARN [c.c.h.x.r.XenServerStorageProcessor] > (DirectAgent-17:ctx-f6381170) (logid:4835671f) Could not mount secondary > storage 10.31.9.226:/storage/cloudstack/secondary/ on host > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3 > 2022-04-04 13:13:44,181 WARN [c.c.h.x.r.XenServerStorageProcessor] > (DirectAgent-17:ctx-f6381170) (logid:4835671f) Catch Exception > com.cloud.utils.exception.CloudRuntimeException for template due to > com.cloud.utils.exception.CloudRuntimeException: Could not mount secondary > storage 10.31.9.226:/storage/cloudstack/secondary/ on host > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3 > com.cloud.utils.exception.CloudRuntimeException: Could not mount secondary > storage 10.31.9.226:/storage/cloudstack/secondary/ on host > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3 > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:84) > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:249) > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:286) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:95) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:59) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:36) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:30) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1758) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttach > 2022-04-04 13:13:44,163 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-17:ctx-f6381170) (logid:4835671f) Task failed! Task record: > uuid: e907d695-825f-24e4-5ce3-afd74eb0ece4 > nameLabel: Async.host.call_plugin > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon Apr 04 13:13:43 IST 2022 > finished: Mon Apr 04 13:13:43 IST 2022 > status: failure > residentOn: com.xensource.xenapi.Host@5fa0255a > progress: 1.0 > type: <none/> > result: > errorInfo: [XENAPI_PLUGIN_FAILURE, mountNfsSecondaryStorage, > SROSError, Error reporting error, unknown key Unexpected error while trying > to mount 10.31.9.226:/storage/cloudstack/secondary/ to > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > 2022-04-04 13:13:44,173 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-17:ctx-f6381170) (logid:4835671f) callHostPlugin failed for cmd: > mountNfsSecondaryStorage with args remoteDir: > 10.31.9.226:/storage/cloudstack/secondary/, localDir: > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3, nfsVersion: null, due > to Task failed! Task record: uuid: > e907d695-825f-24e4-5ce3-afd74eb0ece4 > nameLabel: Async.host.call_plugin > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon Apr 04 13:13:43 IST 2022 > finished: Mon Apr 04 13:13:43 IST 2022 > status: failure > residentOn: com.xensource.xenapi.Host@5fa0255a > progress: 1.0 > type: <none/> > result: > errorInfo: [XENAPI_PLUGIN_FAILURE, mountNfsSecondaryStorage, > SROSError, Error reporting error, unknown key Unexpected error while trying > to mount 10.31.9.226:/storage/cloudstack/secondary/ to > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > Task failed! Task record: uuid: > e907d695-825f-24e4-5ce3-afd74eb0ece4 > nameLabel: Async.host.call_plugin > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon Apr 04 13:13:43 IST 2022 > finished: Mon Apr 04 13:13:43 IST 2022 > status: failure > residentOn: com.xensource.xenapi.Host@5fa0255a > progress: 1.0 > type: <none/> > result: > errorInfo: [XENAPI_PLUGIN_FAILURE, mountNfsSecondaryStorage, > SROSError, Error reporting error, unknown key Unexpected error while trying > to mount 10.31.9.226:/storage/cloudstack/secondary/ to > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > Task failed! Task record: uuid: > e907d695-825f-24e4-5ce3-afd74eb0ece4 > nameLabel: Async.host.call_plugin > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon Apr 04 13:13:43 IST 2022 > finished: Mon Apr 04 13:13:43 IST 2022 > status: failure > residentOn: com.xensource.xenapi.Host@5fa0255a > progress: 1.0 > type: <none/> > result: > errorInfo: [XENAPI_PLUGIN_FAILURE, mountNfsSecondaryStorage, > SROSError, Error reporting error, unknown key Unexpected error while trying > to mount 10.31.9.226:/storage/cloudstack/secondary/ to > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:500) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:422) > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:80) > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:249) > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:286) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:95) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:59) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:36) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:30) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1758) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-04-04 13:13:44,180 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-17:ctx-f6381170) (logid:4835671f) unable to destroy > task(com.xensource.xenapi.Task@3cd54231) on > host(bc164e35-7efe-4ff7-9460-1c92cdb44396) due to You gave an invalid object > reference. The object may have recently been deleted. The class parameter > gives the type of reference given, and the handle parameter echoes the bad > value given. > 2022-04-04 13:13:44,180 WARN [c.c.h.x.r.XenServerStorageProcessor] > (DirectAgent-17:ctx-f6381170) (logid:4835671f) Could not mount secondary > storage 10.31.9.226:/storage/cloudstack/secondary/ on host > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3 > 2022-04-04 13:13:44,181 WARN [c.c.h.x.r.XenServerStorageProcessor] > (DirectAgent-17:ctx-f6381170) (logid:4835671f) Catch Exception > com.cloud.utils.exception.CloudRuntimeException for template due to > com.cloud.utils.exception.CloudRuntimeException: Could not mount secondary > storage 10.31.9.226:/storage/cloudstack/secondary/ on host > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3 > com.cloud.utils.exception.CloudRuntimeException: Could not mount secondary > storage 10.31.9.226:/storage/cloudstack/secondary/ on host > /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3 > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:84) > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:249) > at > com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:286) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:95) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:59) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:36) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:30) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1758) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttach > > ~~~~~~ > > > Best Regards, > Midhun > -- This message is intended only for the use of the individual or entity to which it is addressed and may contain confidential and/or privileged information. If you are not the intended recipient, please delete the original message and any copy of it from your computer system. You are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited unless proper authorization has been obtained for such action. If you have received this communication in error, please notify the sender immediately. Although IndiQus attempts to sweep e-mail and attachments for viruses, it does not guarantee that both are virus-free and accepts no liability for any damage sustained as a result of viruses.
