Hi Vivek, The error you are getting is self explanatory. You are trying to SSH into a storage resource with a ssh key that is not available in the storage's authorized_keys file. I can see that you are trying to create a new ssh credential. token = generateAndRegisterSSHKeys(authzToken, ""); Issue is, the public key of this credential is not copied to the authorized_keys file of the target storage resource. You need to fetch the SSH credential from the storage preference as we are doing in Helix [1] [2]. These keys are already registered in the storage endpoint.
[1] https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/staging/DataStagingTask.java#L83 [2] https://github.com/apache/airavata/blob/7a9b27aac395883afcf9f5bf08a3eb4c6ff577a5/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/TaskContext.java#L477 Thanks Dimuthu On Wed, Aug 19, 2020 at 4:06 AM Bandaru, Vivek Shresta <vivb...@iu.edu> wrote: > Hi All, > > > > Though the previous approach worked, it had a problem. Every gateway would > need to implement their own user storage tracking mechanism. As discussed > earlier in my previous mail, the better approach would be to establish an > SSH connection with the Storage Resource and track the storage size in > Airavata. After a little debugging and discussions, I found that > StorageResourceAdaptor and AgentAdaptor’s (which uses pooling) are the way > to establish a connection and execute commands on remote machines instead > of creating a new SSHConnection, so my new implementation uses > StorageResourceAdaptor to establish a connection and execute the du > <https://www.geeksforgeeks.org/du-command-linux-examples/#:~:text=du%20command%2C%20short%20for%20disk,space%20on%20hard%20disk%20drive.> > commands > to get the size of the user directory. > > > > Questions: > > > > 1. Currently, in case of Airavata-Django-Portal, experimentDataDir is > always inside the project directiory which in turn is inside the user > directory. Will the same behavior be followed across all the gateways? > 2. I was trying to establish a connection with the gateway using > StorageResourceAdaptor (I believe AgentAdaptor’s are used for compute > resources) but I’m stuck at this step. I was successfully able to create a > connection using AgentAdaptor (only for testing purposes) and execute > commands on it. But I’m not able to do the same with > StorageResourceAdaptor. > The only difference I can think of is, here I’m creating a new > authToken for establishing a connection with the Storage resource (and of > course deleting it at the end of execution), and I’m assuming this might be > the issue. While testing with AgentAdaptor, I hardcoded the authToken value > that’s already generated for the Jetstream compute resources, so I’m > guessing that might be the reason why it was working in this case. > > This is the error stacktrace that I’ve encountered while executing the > ‘executeCommand()’ on StorageResourceAdaptor after initiating it. > > > 2020-08-18 19:31:07,152 [SSH-Pool-Monitor-airavata.host-22222] INFO > o.a.a.h.a.PoolingSSHJClient Current active connections for null @ > airavata.host : 22222 are 0 [] > > 2020-08-18 19:31:07,173 [pool-24-thread-6] INFO > n.s.s.transport.TransportImpl Client identity string: SSH-2.0-SSHJ_0.23.0 [] > > 2020-08-18 19:31:07,180 [pool-24-thread-6] INFO > n.s.s.transport.TransportImpl Server identity string: SSH-2.0-OpenSSH_7.2p2 > Ubuntu-4ubuntu2.7 [] > > 2020-08-18 19:31:07,488 [pool-24-thread-6] ERROR > o.a.a.a.s.h.AiravataServerHandler > net.schmizz.sshj.userauth.UserAuthException: Exhausted available > authentication methods [] > > org.apache.airavata.agents.api.AgentException: > net.schmizz.sshj.userauth.UserAuthException: Exhausted available > authentication methods > > at > org.apache.airavata.helix.adaptor.SSHJAgentAdaptor.executeCommand(SSHJAgentAdaptor.java:190) > > at > org.apache.airavata.helix.adaptor.SSHJStorageAdaptor.executeCommand(SSHJStorageAdaptor.java:94) > > at > org.apache.airavata.api.server.handler.AiravataServerHandler.validateStorageLimit(AiravataServerHandler.java:726) > > at > org.apache.airavata.api.Airavata$Processor$validateStorageLimit.getResult(Airavata.java:18296) > > at > org.apache.airavata.api.Airavata$Processor$validateStorageLimit.getResult(Airavata.java:18280) > > at > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > > at > org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:834) > > Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted > available authentication methods > > at net.schmizz.sshj.SSHClient.auth(SSHClient.java:231) > > at > org.apache.airavata.helix.adaptor.PoolingSSHJClient.createNewSSHClient(PoolingSSHJClient.java:273) > > at > org.apache.airavata.helix.adaptor.PoolingSSHJClient.newClientWithSessionValidation(PoolingSSHJClient.java:101) > > at > org.apache.airavata.helix.adaptor.PoolingSSHJClient.leaseSSHClient(PoolingSSHJClient.java:141) > > at > org.apache.airavata.helix.adaptor.PoolingSSHJClient.startSessionWrapper(PoolingSSHJClient.java:281) > > at > org.apache.airavata.helix.adaptor.SSHJAgentAdaptor.executeCommand(SSHJAgentAdaptor.java:173) > > ... 10 common frames omitted > > This > <https://github.com/vivekshresta/airavata/commit/2ee51719631490a00b3e8059c136f194538987df#diff-2ee16d33a51ba0c66c57f1847f0d72aeR700> > is the new validation api in AiravataServerHandler I’ve written, where I’m > encountering the issue. How can this be mitigated? > > > > Any help is appreciated. > > > > And on a side note, apart from this, I’ve started working on MFT-portal > backend in developing a new mechanism to track transfers and fetch > registered storages. > > > > Regards, > > Vivek. > > >
