Hi, So I've finally started thinking about adding support for file transfers via cliprdr channel. As noted by akallabeth [1], I also do not see any really portable way to implement it. There is no single standard on how to handle clipboard, even without supporting files, and MS-RDPECLIP [2] is fairly Windows-biased. However, I believe that the platform differences can be reasonably abstracted out from the implementation.
Akallabeth suggested the overall implementation strategy. To sum it up: 1) just read the requested files when cliprdr asks us for content, 2) use FUSE to provide files to local applications, backing the storage with cliprdr requests. I have given it a thought for some time, but have not delved too deep in the implementation yet. I would like to get some peer review to ensure that my understanding looks okay. Please note that I am mostly experienced with XFreeRDP on Linux as that's what I use. While I'm technically able to build and test all clients (except for DirectFB one), I have minor development experience with FreeRDP on other platforms. Please feel free to point out any possible portability issues which you may spot. [1]: https://github.com/FreeRDP/FreeRDP/issues/2424#issuecomment-281128986 [2]: https://msdn.microsoft.com/en-us/library/cc241066.aspx Background ---------- Here are the core points about cliprdr, file transfers, and related clipboard formats: 1. File content is transferred out of band. Regular clipboard contents are transferred via CLIPRDR_FORMAT_DATA_RESPONSE messages. However, for files these messages contain only metadata with the list of file names, sizes, flags, etc. The actual content of files is transferred via CLIPRDR_FILECONTENTS_RESPONSE messages. 2. File content is transferred asynchronously. CLIPRDR_FILECONTENTS_RESPONSE messages are sent in response to CLIPRDR_FILECONTENTS_REQUEST messages which are sent after format data has been received. Obviously, responses arrive separetely, with a certain delay. 3. Local applications expect clipboard data in timely manner. When a local application requests clipboard contents it expects the data to be available in a short time frame. For Windows and Mac we are required to synchronously provide some response. Selections on Linux/X11 work asynchronously, but if the data is not provided within a certain timeout (e.g., 10 seconds for GTK applications), the paste is canceled. For files this means that we must reasonably quickly provide the whole list of files which are expected to be immediately available for interaction via the filesystem. While such timeout may seem more than enough to download a single file, it has strong impact on large files, large amout of small files, and complex directory structure. Thus it may be impractical to delay providing the file list data until the file are downloaded from the server. There are little issues with local-to-remote transfers as we simply need to fullfill cliprdr requests. However, for remote-to-local transfers we will have to ensure that the platform can provide a way to publish a list of files without actually having their content. - On Windows files are pasted as "FileGroupDescriptorW" format which uses some black COM magic to provide file content in a streamed fashion (see FreeRDP's Windows client for example). - On Mac NSPasteboard supports NSFilesPromisePboardType format which allows an application to immediately paste a list of files while promising to provide their content later. - On Linux we have freedom (i.e., lack of any standard way to transfer files via clipboard). Known conventions include MIME types text/uri-list, x-special/gnome-copied-files, and application/x-kde-cutselection which all are essentially text lists of URIs describing the locations of the pasted files. The content of files is expected to be immediately available, any deferred retrieval should be handled by the filesystem. Gnome applications use GVfs, KDE applications use KIOSlaves, POSIX applications can use FUSE. Unfortunately, GVfs and KIOSlaves are baked into respective desktop environments and are not really extendable which leaves us with only FUSE. Proposed architecture --------------------- ## Cast of the play There are three principal participants in clipboard redirection (using XFreeRDP as an example): - cliprdrPlugin (channels/cliprdr/client) * implements MS-RDPECLIP protocol * parses and serializes messages * provides a callback interface to the client - xfClipboard (client/X11/xf_cliprdr) * interacts with the local system clipboard (X Selection), implements some necessary quirks * implements and uses cliprdrPlugin callbacks for receiving and sending format data when requested by local or remote clipboard * uses wClipboard for actual format conversion - wClipboard (winpr/libwinpr/clipboard) * stores format data * provides conversion between formats Fundamental roles of these objects are unchanged with addition of file transfers, but some extensions and special cases will have to be made. ## cliprdrPlugin cliprdrPlugin will still be tasked with parsing and serializing protocol messages, there should be minor architectural changes to it. It already supports file clipping messages, only some flag definitions are missing. Another thing that is missing is support for CLIPRDR_FILELIST parsing. We should add some common functions to avoid duplicating parsers and serializers in clients. These utility functions may be exported by <freerdp/client/cliprdr.h> with the following approximate signatures: /** * Parse a packed file list. * * `fileList->fileDescriptorArray` is updated in case of successful * parsing. The pointer is not freed by this function, so the caller * must ensure that its contents can be safely overwritten. * The resulting array must be freed with the `free()` function. * * @param [in] formatData packed file list to parse. * @param [in] formatDataLength length of formatData in bytes. * @param [out] fileList parsed file list stored here. * * @returns 0 on success, otherwise a Win32 error code. */ UINT cliprdr_parse_file_list(const BYTE* formatData, UINT32 formatDataLength, CLIPRDR_FILELIST* fileList); /** * Serialize a packed file list. * * The resulting `formatData` must be freed with the `free()` * function. * * @param [in] fileList file list to serialize. * @param [out] formatData packet file list stored here. * @param [out] formatDataLength length of formatData stored here. * * @returns 0 on success, otherwise a Win32 error code. */ UINT cliprdr_serialize_file_list(const CLIPRDR_FILELIST* fileList, BYTE** formatData, UINT32* formatDataLength); ## xfClipboard xfClipboard (or any other cliprdr client) should also stay relatively unchanged. It is primarily tasked with correctly putting and extracting converted format data from and into the local clipboard. The client is a mediator between cliprdrPlugin, wClipboard, and the local system clipboard. File lists are just another format that it needs to handle. However, this format is a bit special as file lists must outlive the handling of CLIPRDR_FORMAT_DATA_{RESPONSE,REQUEST} messages. Here are the transfer sequences to clarify the idea of the implementation. ### Remote-to-local transfer sequence 1. The user copies some files in the remote session. 2. The server sends us a CLIPRDR_FORMAT_LIST with "FileGroupDescriptorW" among the available formats. 3. The client receives the format list, converts remote formats to local equivalents, and announces the available formats on the local clipboard. 4. The user pastes the files into some local application. The application requests a file format from the client. 5. The client handles the request and sends an appropriate CLIPRDR_FORMAT_DATA_REQUEST to the server. 6. The server replies with CLIPRDR_FORMAT_DATA_RESPONSE containing the list of files. The client receives and parses the message. 7. The client parses the packed file list into CLIPRDR_FILELIST and stores it in the wClipboard. Note that wClipboard resides in WinPR which should not know about FreeRDP's cliprdr, but it may know about Windows clipboard format structures. Thus we will need to pass the files as new FILEDESCRIPTOR structures (identical to CLIPRDR_FILEDESCRIPTOR). 8. wClipboard looks through the file list and starts providing it to the local file system. (It does not have any content yet.) 9. The client asks wClipboard to convert the file list into some local clipboard format (e.g., text/uri-list). 10. The client provides the resulting format data to that local application which has been waiting for it since step 4. 11. The local application tries to open and read a file which it has received. The filesystem magically transfers control to the client's wClipboard, requesting a chunk of data. 12. wClipboard handles the request and forwards it to the client. 13. The client sends a CLIPRDR_FILECONTENTS_REQUEST to the server. 14. The server fetches the data and replies with a CLIPRDR_FILECONTENTS_RESPONSE. 15. The client receives the response and forwards the data to wClipboard. 16. wClipboard forwards the data to local application via some filesystem magic. 17. Steps 11--16 are repeated until the local application is satisified, or until the server sends a new format list which means that the clipboard content is cleared and file content is no more available (unless file locking is used). ### Local-to-remote transfer sequence 1. The user copies some files in a local application. The application announces a local file format on the local clipboard. 2. The client receives a notification, converts the local formats to remote equivalents (with "FileGroupDescriptorW" among them), and sends a CLIPRDR_FORMAT_LIST to the server. 3. The user pastes the files in the remote session. 4. The remote desktop handles the paste and the server sends an appropriate CLIPRDR_FORMAT_DATA_REQUEST to the client. The client receives and parses the message. 5. The client requests a local file format from the local application and waits for the data to be available. 6. The client passes the local file format data to wClipboard which parses it, stores it, and prepares to serve the content requests. 7. The client asks wClipboard to convert the local file list into a list of FILEDESCRIPTOR structures. 8. The client converts the list of FILEDESCRIPTOR structures into a serialized CLIPRDR_FILELIST and sends a CLIPRDR_FORMAT_DATA_RESPONSE to the server. 9. The remote desktop needs a chunk of file data so the server sends a CLIPRDR_FILECONTENTS_REQUEST to the client. The client receives and parses the message. 10. The client asks wClipboard to provide a chunk of data from the specified file. 11. wClipboard fetches the data from the local filesystem. 12. wClipboard forwards the data to the client. 13. The client replies with CLIPRDR_FILECONTENTS_RESPONSE to the server. 14. The steps 9--13 are repeated until the remote desktop is satisified, or until the client sends a new format list to the server (which clears the file list stored in wClipboard). ## wClipboard wClipboard looks like an ideal place for the file clipping support. Effectively, it implements interface and behavior of Windows clipboard for non-Windows platforms. The way Windows handles files on clipboard seems to fit exactly into the intended purpose of this object, so I believe that we should put the portable implementation there. However, WinAPI does not provide any clipboard functions specifically for handling files. Moreover, the functions and their interface will be somewhat biased towards cliprdr needs. But I believe that their utility justifies placing them into WinPR. ### wClipboard interface extension The transfer sequences described above require wClipboard to have the following abilities: - list supported local file list formats - convert between remote file lists and local format data - publish remote files to local filesystem in an on-demand fashion - fetch content of files published on local clipboard There are multiple local file list formats which may have different dependencies, so wClipboard should be extendable to handle them. Their availability may depend on how libfreerdp and libwinpr are built, so clients should dynamically adjust the formats that they may use to interact with local clipboard, without hardcoding them. There are also multiple ways to publish remote files into local filesystem so that they are available for clipboard transfers. Every platform has its own preferred way of doing this. Platform interfaces for this are usually asynchronous. Thus it makes sense to develop some common facade interface for wClipboard and implement platform-specific parts in subsystems (like it is done, for example, with rdpsnd). The specific interface implementation will need to be able to issue cliprdr requests and receive the responses. WinPR should not depend on FreeRDP or on a specific FreeRDP client, so we will need a delegate interface to provide the ability to fetch file content via cliprdr. Finally, files on local clipboard not always refer to files located on the local disk drive. Depending on the actual clipboard format it may contain references to COM objects, URLs of remote resources, etc. They may be not accessible via simple open() or fopen() calls, requiring to use some other API instead (which may be asynchronous or synchronous). Thus it makes sense to hide this aspect behind an interface as well. ### Example of an asynchronous interface Let's take as an example the task of providing remote files in local filesystem. For this we will need to tell the local clipboard subsystem about available files and then fulfill its data requests. The file list can be published by something like this: /** * Provide remote files on local clipboard. * * This function copies the list of files so it can be freed after * the call. * * @param clipboard clipboard instance. * @param files the list of remote files to provide. * @param fileCount number of `files`. * * @returns 0 on success, otherwise a Win32 error code. */ UINT ClipboardSetRemoteFileData(wClipboard* clipboard, const FILEDESCRIPTOR* files, UINT32 fileCount); This function is a special case of ClipboardSetData() for file lists. It will copy the file list into wClipboard, increment the sequence number, and provide the list to local clipboard subsystem so that it can make arrangements for the files to become available on the local clipboard. (Sister function ClipboardGetRemoteFileData() may be added as well.) The local clipboard subsystem will need to ask the client to perform cliprdr requests. It may do so via the following delegate interface. Note that it is asynchronous, requests and replies carry on the full context as well as the sequence number to track outdated requests. /** * Backing storage of remote files. * * `clipboard` is a backlink to corrensponding wClipboard instance. * * `custom` is an opaque client context which may be used by * client callbacks. * * `Clipboard*` callbacks should be filled in by the client, they * will be used by wClipboard to request content and information * about remote files. * * `Client*` callbacks are filled by wClipboard, they should be * used by the client to report completion of matching wClipboard * requests. */ struct wClipboardRemoteFileDelegate { wClipboard* clipboard; void* custom; /** * Request the client to retrieve the size of a remote file. * * @param delegate delegate instance. * @param sequence current sequence number. * @param fileIndex index of the file. * * @returns 0 on success, otherwise a Win32 error code. */ UINT (*ClipboardRequestFileSize) (wClipboardRemoteFileDelegate* delegate, UINT32 sequence, UINT32 fileIndex); /** * Provide the size of a remote file to clipboard. * * @param delegate delegate instance. * @param sequence sequence number of the request. * @param fileIndex index of the file. * @param fileSize size of the file in bytes. * * @returns 0 on success, otherwise a Win32 error code. */ UINT (*ClientProvideFileSize) (wClipboardRemoteFileDelegate* delegate, UINT32 sequence, UINT32 fileIndex, UINT64 fileSize); UINT (*ClipboardRequestFileContent) (wClipboardRemoteFileDelegate* delegate, UINT32 sequence, UINT32 fileIndex, UINT64 position, UINT32 requestedBytes); UINT (*ClientProvideFileContent) (wClipboardRemoteFileDelegate* delegate, UINT32 sequence, UINT32 fileIndex, UINT64 position, const BYTE* data, UINT32 dataLength); /* . . . */ }; /** * Set the current remote file delegate. * * Installs the provided delegate into the clipboard and fills in * clipboard-side callbacks in it. * * The clipboard does not take ownership of the delegate. It should * be freed manually before calling ClipboardDestroy(). * * @param clipboard clipboard instance. * @param delegate delegate instance. * * @returns 0 on success, otherwise a Win32 error code. */ UINT ClipboardSetRemoteFileDelegate(wClipboard* clipboard, wClipboardRemoteFileDelegate* delegate); ### Further details on wClipboard I did not think it through too deeply as the actual implementation is likely to be different from what can be seen here, but you should get the whole idea from this. The interface has to be asynchronous as it's the most natural way to do input-output for backends (be it cliprdr requests, FUSE requests, or GIO requests). Unfortunately, asynchronicity is akin to cancer, it tends to spread out. The actual clipboard subsystems will have a similar interface for wClipboard. I suppose that we will initially need at least the following ones: - dumb remote backend: * the most portable one * initially returns a 'not available' error * downloads the files into a temporary directory in background * when download is complete it can provide the files to local clipboard * if clipboard content is changed then the download is canceled and the client gets an error response - FUSE remote backend: * requires libfuse to be available * immediately makes the files available in a temporary FUSE-mapped directory * downloads file content on demand - text/uri-list local backend: * supports file:// schema in text/uri-list MIME format * opens and reads local files using standard C or POSIX The subsystems encapsulate the platform differences. Adding a new backend in WinPR should automagically add its support to all clients which use cliprdr for file transfers without any additional changes. Issues to consider ------------------ ## Concurrency and asynchronicity wClipboard will have to have asynchronous interface to accomodate the fact that data requests can take time to complete. It is also possible for asynchronous requests to silently fail, be abandoned without further notice, be duplicated if they take too long to complete, or be completed long after useful timeframe (e.g., after clipboard content has changed). We should be ready for all this. Interactions will also most likely involve more than one thread, thus care must be taken to correctly program the concurrent parts of the implementation, using queues, locks, reference counts, etc. ## Latency and throughput Obviously, this is hardly a lightning fast approach to transferring file content. I consider this mostly a non-issue as this is not an intended use case of clipboard redirection, which is meant to be a _convenient_ way of transferring files. When speed matters and for bulky transfers one should arguably use disk drive redirection directly. If this becomes an issue then we can leverage disk drive redirection as a temporary storage for pasted files. Cliprdr protocol provides support for this in the form of CLIPRDR_TEMP_DIRECTORY message used to negotiate a temporary storage location which may be used to avoid file transfer via CLIPRDR_FILECONTENTS_{REQUEST,RESPONSE} protocol. ## Caching Both local-to-remote and remote-to-local transfers can benefit from some amount of caching. For local-to-remote transfers file content caching should be performed by the remote side, but from our local side it would be wise to avoid reopening local files for each server request, keeping them open for as long as possible. We can also benefit from the fact that the server usually issues content requests in sequential way so we may avoid unnecessary seeking. For remote-to-local transfers it would be wise to cache the received file content so as to avoid performing unnecessary duplicate requests when files are pasted or read multiple times by the local application. ## File locking Cliprdr protocol assumes that the file content is available only until a new format list is published by either client or server. Files may take a significant amount of time to transfer so it's not unreasonable to expect clipboard content to change during the transfer. To allow the transfer to complete regardless of content changes cliprdr provides a file locking facility. It allows the peer to pin requested files so that their content may be queried even after clipboard content change. Most likely we will have to implement both usage of the locking protocol for remote-to-local transfer and support for the protocol for local-to-remote transfers. Otherwise it may be extemely easy for the user to accidentally cancel an ongoing file transfer. ## Intersession transfers Aside from local-to-remote and remote-to-local transfers there also exists a case when files are transferred between two remote sessions. Currently this use case is optimized in XFreeRDP via CF_RAW format which directly transfers raw data between remote sessions without doing any intermediate format conversions, effectively proxying CLIPRDR_FORMAT_DATA_REQUESTS and CLIPRDR_FORMAT_DATA_RESPONSES. Unfortunately, raw data transfers will not work as is for files. One option is to implement some similar proxying for CLIPRDR_FILECONTENTS_{REQUEST,RESPONSE} messages. Another one is to simply disable raw transfer capability when files are available, thus treating another remote session as any other application. Not using raw transfers for files is quick to implement, but may lack performance. Proxying requests may provide better performance, but will need some proper interprocess communication mechanism to be implemented (though, X window properties may work fine). Implementation plan ------------------- Well, I'm not at my software development day job, so there obviously won't be any schedule commitment. I also dislike grand half-a-year detailed plans which never work out as expected. I believe the outline above will suffice as a roadmap. Thus I propose to move in the following small and vague steps, with testing, bugfix, and improving the code in between them: 1. Implement transferring local files to remote desktop for XFreeRDP. 2. Implement transferring remote files to local desktop for XFreeRDP. 3. The users should become happier now. 4. Later implement file locking, add support for other clients, improve intersession transfers, add drag-n-drop support, use temporary directories, add CF_HDROP support for Windows XP, and do other fancies to make the users even more happy. Parting words ------------- So, how does this all look to you? Will it fly? Please feel free to comment on the design. While this looks fairly portable and extendable to me, I would like to get a second opinion. Especially given that I intend this to be the groundwork for supporting file transfers in non-X11 clients (with which I have close to zero development experience). I am going to start writing the actual code in the nearest time. If you are interested then you can peek at it in my GitHub fork [3]. That pull request is there just to keep my notes around, I will file proper pull requests to FreeRDP repo as soon as there is something reviewable. If you have any questions, suggestions, confenssions, threats, etc., I will also available on the IRC (as ilammy). [3]: https://github.com/ilammy/FreeRDP/pull/4 ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ FreeRDP-devel mailing list FreeRDP-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freerdp-devel
