Intent to Implement: Binary Messaging Interface for Crosswalk External Extension
Description: Today's Crosswalk external extension API only provides text based messaging interface. However, coming use cases require the external extension native code to exchange binary data with JavaScript code. For example, an external extension wants to transfer the images stream from native to JavaScript in real-time (30FPS at least). With today's external extension messaging API, the extension has to encode the binary image data into text in native first, posts to JavaScript and then decodes from the text to binary data in JavaScript. The encoding, decoding (in JavaScript!) and increased message size leads to low performance. Experiment shows by using base64 to transfer VGA (640x480) images, this approach can only deliver less 10 FPS on mainstream PC. The binary messaging interface is proposed to added into Crosswalk external extension API. It allows native code to exchange binary data (a byte array) to JavaScript without unnecessary encoding/decoding. JavaScript receives an ArrayBuffer object and can access the binary data via different Typed Array View. Vice versa, JavaScript can send the binary data in an ArrayBuffer object to native, native code receives a byte array. Prototype shows 3X-8X (binary messaging vs. text messaging) performance speedup when transferring different size of data on both PC and mobile device. Initial data shows with binary messaging interface, an external extension is able to transfer VGA images at 60FPS on PC. Affected component: Crosswalk External Extension Related feature: https://crosswalk-project.org/jira/browse/XWALK-4615 Target release: Crosswalk-17 Implementation details: 1. Interface change: For Crosswalk External Extension C API, a new interface XW_BinaryMessagingInterface_1 will be added. To keep API and ABI compatibility with existing external extension, a new file (extensions/public/XW_Extension_BinaryMessage.h) will be added to contain the XW_BinaryMessagingInterface_1 interface. The XW_BinaryMessagingInterface_1 looks like: /////////////////////////////////////////////////////////////////////////////////////////// #ifdef __cplusplus extern "C" { #endif #define XW_BINARY_MESSAGING_INTERFACE_1 "XW_BinaryMessagingInterface_1" #define XW_BINARY_MESSAGING_INTERFACE XW_BINARY_MESSAGING_INTERFACE_1 typedef void (*XW_HandleBinaryMessageCallback)(XW_Instance instance, const char* message, const size_t size); struct XW_BinaryMessagingInterface_1 { void (*Register)(XW_Extension extension, XW_HandleBinaryMessageCallback handle_message); void (*PostMessage)(XW_Instance instance, const char* message, const size_t size); }; typedef struct XW_BinaryMessagingInterface_1 XW_BinaryMessagingInterface; #ifdef __cplusplus } // extern "C" #endif ////////////////////////////////////////////////////////////////////////////////////////// If the C extension wants to use binary messaging, it needs to query the "XW_BinaryMessagingInterface_1" interface and use the PostMessage method to post binary data to JavaScript. On other hand, it needs to register a XW_HandleBinaryMessageCallback to receive binary data from JavaScript. For Crosswalk external extension Android Java API, two methods will be added into XWalkExtensionClient interface: ///////////////////////////////////////////////////////////////////////////////////////// public void onBinaryMessage(int extensionInstanceID, byte[] message); public final void postBinaryMessage(int instanceID, byte[] message); //////////////////////////////////////////////////////////////////////////////////////// Java extension which inherits XWalkExtensionClient can use postBinaryMesssage to post binary data to JavaScript and implements onBinaryMessage to receive binary data from JavaScript. No new APIs will be added into Crosswalk extension JavaScript API. When native external extension posts binary message, the JavaScript message listener receives an ArrayBuffer object. When JavaScript invokes extension.postMessage with an ArrayBuffer object as argument, the native binary message listener is invoked. This proposal doesn't cover binary messaging support in SyncMessage, as there is no obvious use case. Sample code: An external extension wants to transfer image data to JavaScript. To transfer structured data, the extension can use simple custom binary message format as this sample shows. Or it can leverage more sophisticated binary format, e.g. Google's Protocol Buffer (https://developers.google.com/protocol-buffers/). C++: /////////////////////////////////////////////////////////////////////////////////////// // image binary format // uint32_t width; // uint32_t height; // char data[width * height * 4]; static const XW_BinaryMessagingInterface* g_binary_messaging = get_interface(XW_BINARY_MESSAGING_INTERFACE); ..... size_t buffer_size = sizeof(uint32_t) + sizeof(uint32_t) + width*height*4*sizeof(char); char* buffer = new char[buffer_size ]; uint32_t* uint32_array = reinterpret_cast<uint32_t*>(buffer); uint32_array[0] = image.width(); uint32_array[1] = image.height(); char* data = buffer + 2 * sizeof(uint32_t); image.CopyImageData(data); g_binary_messaging->postMessage(instance, buffer, buffer_size); delete[] buffer; /////////////////////////////////////////////////////////////////////////////////////// JavaScript: /////////////////////////////////////////////////////////////////////////////////////// extension.setMessageListener(function(msg) { if (msg instaceof ArrayBuffer) { var uint32_view = new Uint32Array(msg); var width = uint32_view[0]; var height = uint32_view[1]; var data = new Uint8Array(msg, 8, width*height*4); .... } } ////////////////////////////////////////////////////////////////////////////////////// Details: It doesn't require to modify Crosswalk Core Extension code (XWalkExtensionModule), as V8ValueConverter already supports to convert a base::BinaryValue object to V8 ArrayBuffer object and vice versa. It mainly requires to modify the external extension components, to expose the binary interface and convert the binary data to base::BinaryValue and vice versa. For C external extension, files to be modified: extensions/common/xwalk_external_adapter.cc extensions/common/xwalk_external_adapter.h extensions/common/xwalk_external_extension.cc extensions/common/xwalk_external_extension.h extensions/common/xwalk_external_instance.cc extensions/common/xwalk_external_instance.h extensions/extensions.gyp new file: extensions/public/XW_Extension_BinaryMessage.h For Java external extension, files to be modified: app/android/runtime_client/src/org/xwalk/app/runtime/extension/XWalkCoreExtensionBridge.java app/android/runtime_client/src/org/xwalk/app/runtime/extension/XWalkExtensionClient.java app/android/runtime_client/src/org/xwalk/app/runtime/extension/XWalkExtensionContextClient.java app/android/runtime_client/src/org/xwalk/app/runtime/extension/XWalkRuntimeExtensionBridge.java app/android/runtime_client/src/org/xwalk/app/runtime/extension/XWalkRuntimeExtensionManager.java extensions/android/java/src/org/xwalk/core/internal/extensions/XWalkExtensionAndroid.java extensions/common/android/xwalk_extension_android.cc extensions/common/android/xwalk_extension_android.h runtime/android/core_internal/src/org/xwalk/core/internal/XWalkExtensionInternal.java Thanks, -ningxin _______________________________________________ Crosswalk-dev mailing list [email protected] https://lists.crosswalk-project.org/mailman/listinfo/crosswalk-dev
