Hi Dan, Since no one from the OpenAFS community has replied I will chime in.
On 5/25/2021 10:21 AM, Daniel Mezhiborsky ([email protected]) wrote:
Hello all,We currently have a relatively small (~250 users, 2TB) AFS cell that I am planning on retiring soon.
If you are willing to explain, I'm sure the community would appreciate hearing the reasons behind the migration away from OpenAFS and how the replacement was selected.
I'd like to get our data out of AFS-space, but this is proving problematic because of performance issues. Our setup is one AFS server VM with iSCSI-attached /vicepa.
It would be helpful if you could provide more details of the server configuration.
1. Is the client-fileserver traffic sharing the same network adapter as the iSCSI attached filesystem backing the vice partition?
2. How much network bandwidth is available to the client-fileserver path? 3. How much network bandwidth is available to the fileserver-iSCSI path?4. What is the network latency (Round Trip Time) between the client and fileserver?
5. What version is the OpenAFS fileserver? 6. What command line options has the fileserver been started with?7. What AFS client version and operating system (and OS version) is the client?
8. What are the configuration options in use by the client? 9. What is the destination file system that the rsync client is writing to? 10. Is the destination filesystem also being accessed via the network?11. What is the OpenAFS fileserver VM configuration? number of cores, amount of RAM, clock speed, etc.
For single large-file transfers with encryption off, I can get around 30MB/s read/write, but it seems like metadata operations are very slow, so copying our data (mostly home directories) directly from AFS with cp/rsync is taking a prohibitively long time.
Which metadata operations are slow?Are you referring to operations reading the metadata from /afs or writing it to the destination?
I see similar slowness with vos dump. We do take regular backups with vos backupsys and backup dump that take about 36 hours for full backups.Does anyone have any recommendations on problem areas to try and tune to get better performance or advice on a better way copy our data out of AFS?
You will need to provide more details about the environment before specific recommendations can be provided. However, I will mention a few things to consider about the extraction methodology.
First, each new AFS RPC behaves similarly to a new TCP connection. It starts with a minimal congestion window size and grows the window via a slow start algorithm until either the maximum window size of 32 * 1412 packets has been reached or packet loss occurs. Combined with the RTT for the link you can compute the bandwitdh delay product for a single call.
In general RPCs issued by an AFS client to read from a fileserver are going to be one disk cache chunk size at a time. Smaller if the average file size is less than the chunk size.
If authentication is required to access the directories and files, a separate FetchStatus RPC will be issued for most files prior to the first FetchData RPC. At least in an OpenAFS client.
One of the strengths of AFS is the caching client, but in this case caching is not beneficial because the data will be read once and discarded during this workflow. The workflow will also be painful for an OpenAFS cache manager because of the callback tracking. If the cache recycles before the callback expires, the cache manager will cancel the outstanding callbacks.
Likewise, if the fileserver callback table fills then it will need to expend considerable effort searching for unexpired callbacks to discard early. Discarding callbacks requires issuing RPCs to the cache manager. So an insufficiently large cache and callback pool can result in the callback lifecycle dominating the workflow.
Instead of performing one rsync at a time. You should consider executing multiple rsyncs in parallel potentially from multiple client systems.
Provide details of the environment and more specific recommendations can be provided.
Jeffrey Altman AuriStor, Inc.
<<attachment: jaltman.vcf>>
OpenPGP_signature
Description: OpenPGP digital signature
