That's great to see Sequel running so lean. I haven't put Sequel under heavy
load but I know if that had been ActiveRecord you would have been drowning in
objects (and memory usage). I just remembered reading something in the Sequel
docs[1] regarding JDBC connections ...
"The following additional options are supported:
• :convert_types - If set to false, does not attempt to convert some
Java types to ruby types. Setting to false roughly doubles performance when
selecting large numbers of rows. Note that you can’t provide this option inside
the connection string (as that is passed directly to JDBC), you have to pass it
as a separate option."
I wonder what that would do to performance?
[1] http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html
On Apr 23, 2010, at 2:22 PM, Michael Lang wrote:
> Don,
>
> Thanks for the enlightening post. jconsole is pretty nifty! I ran the clone
> script again while watching from jconsole. I don't see any obvious thrashing
> of any resources going on. Here's my stats from the last run.
>
> heap size: 5k
> max heap size: 65k
> committed memory: 6k
> live threads: 12, peak: 14
> daemon threads: 7, total started: 15
> current classes loaded: 2,364
> total physical memory: 4gb
> committed virtual memory: 1.2gb
> free physical memory: 1.1gb
>
> Records being cloned: 22k
> columns: 63. of these, roughly 15 are string columns, all 25 characters or
> less, 4 are date/time, 5 are floats, and the rest are integers. four indexes
> on the table.
>
> For whatever its worth, the same data can be cloned using Delphi compiled app
> on Windows XP in about 30 seconds.
>
> My primary goal is to write a wicked fast cloner in ruby that can really
> crank through a database. I'm hoping to scale by either pushing to many
> threads or multiple servers (a la Pentaho Kettle). Of course, my better
> option may very well be to use Kettle, but I thought I'd have a little fun
> with jruby before diving into Java (which isn't something I'm all that keen
> on doing).
>
> Michael
>
>
> On Fri, Apr 23, 2010 at 4:43 PM, Don Morrison <[email protected]> wrote:
> I've been doing some work with JRuby and love it. I have found the
> performance to be fine, but the deployment options are where the differences
> really come in!
>
> That said, how many records are we talking here? 6 minutes seems like quite
> a while, so it's a fair amount. When running your code under MRI, Ruby will
> just take as many system resources as it can to get the work done, however,
> under JRuby things are a little different. JRuby includes the constraints of
> the JVM and my *guess* is that src_ds is a big list of files and that may be
> pushing on the upper limits of the JVM heap and causing some havoc there. I
> would start by checking the max heap size, if you haven't set it explicitly
> you may need to dig a bit to find the default. Give your code some headroom
> and things may look better.
>
> The last week or so I have been using jconsole to inspect my Rails
> application running under JRuby(+ Tomcat). It has been eye-opening to say the
> least. I see you're on a mac so just fire up 'jconsole' and point it at the
> running ruby process. It will tell you all kinds of things about what is
> going on including the JVM Summary where you can see the max heap size.
>
> Hope this helps somewhat.
> Don
>
> On Apr 23, 2010, at 12:57 PM, Michael Lang wrote:
>
> > One of the things I've been hearing about is jruby well outperforming ruby
> > mri.
> >
> > Today, I got jruby + jdbc + sequel working on my mac and took things for a
> > twirl on connecting Sequel to MSDE 2000 via the JDBC drivers.
> >
> > If I use dataset.all, I see practically no difference with querying,
> > copying and inserting data from one database schema when comparing the two
> > platforms. Both take roughly 6 minutes +/- 20 secs. However, if I use
> > dataset.each, I see jruby taking 9 minutes and 8 seconds to do a clone
> > while Ruby MRI stays at 6 minutes 15 seconds, both +/- 20 secs on
> > subsequent benchmarks. That's about a 1:1.44 difference with Ruby MRI well
> > outperforming JRuby! The 20 seconds margin of error is simply me testing
> > during the day on production/live data where 30 or more users are actively
> > using the server (not ideal, I know, but good enough to see basic patterns).
> >
> > Hardware: Macbook Pro 2.33 GHz Intel Core 2 Duo 4GB RAM
> > All installed from macports or latest gems
> >
> > Ruby MRI:
> > ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-darwin9]
> > sequel 3.10.0
> > ODBC Driver: Actual Technologies - Actual ODBC Driver for SQL Server
> > version 2.9e
> >
> > JRuby:
> > java version "1.5.0_22"
> > Java(TM) 2 Runtime Environment, Standard Edition (build
> > 1.5.0_22-b03-333-9M3125)
> > Java HotSpot(TM) Client VM (build 1.5.0_22-147, mixed mode, sharing)
> > jruby 1.4.0 (ruby 1.8.7 patchlevel 174) (2010-04-22 6586) (Java HotSpot(TM)
> > Client VM 1.5.0_22) [i386-java]
> > sequel 3.10.0
> > JDBC Driver: Microsoft sqljdbc_3.0.1301.101
> >
> > The code to clone a table: http://gist.github.com/377053
> >
> > The only thing outside the GIST above is using two DB connections (two
> > different servers) and making the calls to the clone_table method.
> >
> > Am I doing something horribly wrong in JRuby, or is this par for the course?
> >
> > Michael
> > --
> > http://codeconnoisseur.org
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "sequel-talk" group.
> > To post to this group, send email to [email protected].
> > To unsubscribe from this group, send email to
> > [email protected].
> > For more options, visit this group at
> > http://groups.google.com/group/sequel-talk?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "sequel-talk" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/sequel-talk?hl=en.
>
>
>
>
> --
> http://codeconnoisseur.org
>
> --
> You received this message because you are subscribed to the Google Groups
> "sequel-talk" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/sequel-talk?hl=en.
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sequel-talk?hl=en.
