Hi All,
Though the previous approach worked, it had a problem. Every gateway would need
to implement their own user storage tracking mechanism. As discussed earlier
in my previous mail, the better approach would be to establish an SSH
connection with the Storage Resource and track the storage size in Airavata.
After a little debugging and discussions, I found that StorageResourceAdaptor
and AgentAdaptor’s (which uses pooling) are the way to establish a connection
and execute commands on remote machines instead of creating a new
SSHConnection, so my new implementation uses StorageResourceAdaptor to
establish a connection and execute the
du<https://www.geeksforgeeks.org/du-command-linux-examples/#:~:text=du%20command%2C%20short%20for%20disk,space%20on%20hard%20disk%20drive.>
commands to get the size of the user directory.
Questions:
1. Currently, in case of Airavata-Django-Portal, experimentDataDir is always
inside the project directiory which in turn is inside the user directory. Will
the same behavior be followed across all the gateways?
2. I was trying to establish a connection with the gateway using
StorageResourceAdaptor (I believe AgentAdaptor’s are used for compute
resources) but I’m stuck at this step. I was successfully able to create a
connection using AgentAdaptor (only for testing purposes) and execute commands
on it. But I’m not able to do the same with StorageResourceAdaptor.
The only difference I can think of is, here I’m creating a new authToken for
establishing a connection with the Storage resource (and of course deleting it
at the end of execution), and I’m assuming this might be the issue. While
testing with AgentAdaptor, I hardcoded the authToken value that’s already
generated for the Jetstream compute resources, so I’m guessing that might be
the reason why it was working in this case.
This is the error stacktrace that I’ve encountered while executing the
‘executeCommand()’ on StorageResourceAdaptor after initiating it.
2020-08-18 19:31:07,152 [SSH-Pool-Monitor-airavata.host-22222] INFO
o.a.a.h.a.PoolingSSHJClient Current active connections for null @
airavata.host : 22222 are 0 []
2020-08-18 19:31:07,173 [pool-24-thread-6] INFO n.s.s.transport.TransportImpl
Client identity string: SSH-2.0-SSHJ_0.23.0 []
2020-08-18 19:31:07,180 [pool-24-thread-6] INFO n.s.s.transport.TransportImpl
Server identity string: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.7 []
2020-08-18 19:31:07,488 [pool-24-thread-6] ERROR
o.a.a.a.s.h.AiravataServerHandler net.schmizz.sshj.userauth.UserAuthException:
Exhausted available authentication methods []
org.apache.airavata.agents.api.AgentException:
net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication
methods
at
org.apache.airavata.helix.adaptor.SSHJAgentAdaptor.executeCommand(SSHJAgentAdaptor.java:190)
at
org.apache.airavata.helix.adaptor.SSHJStorageAdaptor.executeCommand(SSHJStorageAdaptor.java:94)
at
org.apache.airavata.api.server.handler.AiravataServerHandler.validateStorageLimit(AiravataServerHandler.java:726)
at
org.apache.airavata.api.Airavata$Processor$validateStorageLimit.getResult(Airavata.java:18296)
at
org.apache.airavata.api.Airavata$Processor$validateStorageLimit.getResult(Airavata.java:18280)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available
authentication methods
at net.schmizz.sshj.SSHClient.auth(SSHClient.java:231)
at
org.apache.airavata.helix.adaptor.PoolingSSHJClient.createNewSSHClient(PoolingSSHJClient.java:273)
at
org.apache.airavata.helix.adaptor.PoolingSSHJClient.newClientWithSessionValidation(PoolingSSHJClient.java:101)
at
org.apache.airavata.helix.adaptor.PoolingSSHJClient.leaseSSHClient(PoolingSSHJClient.java:141)
at
org.apache.airavata.helix.adaptor.PoolingSSHJClient.startSessionWrapper(PoolingSSHJClient.java:281)
at
org.apache.airavata.helix.adaptor.SSHJAgentAdaptor.executeCommand(SSHJAgentAdaptor.java:173)
... 10 common frames omitted
This<https://github.com/vivekshresta/airavata/commit/2ee51719631490a00b3e8059c136f194538987df#diff-2ee16d33a51ba0c66c57f1847f0d72aeR700>
is the new validation api in AiravataServerHandler I’ve written, where I’m
encountering the issue. How can this be mitigated?
Any help is appreciated.
And on a side note, apart from this, I’ve started working on MFT-portal backend
in developing a new mechanism to track transfers and fetch registered storages.
Regards,
Vivek.
