Hello, I'm looking for a solution to use jruby on top of spark. It looks that the work required is quite small. The only tricky point is that I need to make sure that every worker thread has a ruby interpreter initialized. Basically, I need to register a function to be called when each worker thread is created : a thread local variable must be set for the ruby interpreter so that ruby object can be deserialized.
Is there any solution to setup the worker threads before any spark call is made using this thread ? I've found a workaround using a java object that wrap any ruby function like: rdd.map( JavaFuncWrapper.new(ruby_function) ). In the java JavaFuncWrapper, the readObject method used to serialize has been overridden to perform the initialization. I must say that I'm not very confident with this workaround. I would prefer a more straightforward solution. Thanks for your help, Regards, Lionel ----------------------------------------- Moody's monitors email communications through its networks for regulatory compliance purposes and to protect its clients, employees and business and where allowed to do so by applicable law. Parties communicating with Moody's consent to such monitoring by their use of the email communication. The information contained in this e-mail message, and any attachment thereto, is confidential and may not be disclosed without our express permission. If you are not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution or copying of this message, or any attachment thereto, in whole or in part, is strictly prohibited. If you have received this message in error, please immediately notify us by telephone, fax or e-mail and delete the message and all of its attachments. Thank you. Every effort is made to keep our network free from viruses. You should, however, review this e-mail message, as well as any attachment thereto, for viruses. We take no responsibility and have no liability for any computer virus which may be transferred via this e-mail message. -----------------------------------------
