Hi Gautam, DBOutputFormat inserts records one by one, which is inherently slow. You can use open source Apache licensed hiho framework which provides MySQL's "load data infile " functionality. It may be more suited to your needs.
HIHO is available at http://code.google.com/p/hiho/ I havent tested it over ssh, please let me know if you need any help setting it up. Thanks and Regards, Sonal www.meghsoft.com On Fri, Apr 23, 2010 at 9:48 AM, Gautam Singaraju < [email protected]> wrote: > All, > > I have a use-case where I need to crunch a large amount of data and > push to the results (comparatively a smaller set) to a mysql db at a > remote location. As per security concerns, only SSH ports are open. I > tried using Java Secure Channel [1] in combination with some custom > JDBC code from the reducers. > > Can anyone comment on the performance of DBOutputFormat? Have there > been any efforts to tunnel this through SSH? This is going to be an > expensive operation; any suggestions would be welcome. > > [1] http://www.jcraft.com/jsch/ > --- > Gautam Singaraju >
