Yes, I believe it would involve modifying file_destination.rb. ----- Darin Wilson <[EMAIL PROTECTED]> wrote: > Sure, I can take a look. I'm assuming that this would have to be handled at > the destination layer? > > D > > > > From: Jacob Maine <[EMAIL PROTECTED]> > > Date: Wed, 22 Oct 2008 19:05:35 -0700 (PDT) > > To: Darin Wilson <[EMAIL PROTECTED]> > > Cc: <activewarehouse-discuss@rubyforge.org> > > Subject: Re: [Activewarehouse-discuss] would the real codebase please stand > > up? :) > > > > Hi Darin - > > > > It looks like MySQL supports a few different ways of converting text into > > nulls. As you pointed out, I can't find documentation about turning empty > > strings into nulls. However, it looks like you can convert \N (or NULL, > > depending on some other settings) into a null. See > > http://dev.mysql.com/doc/refman/6.0/en/load-data.html starting with > > "Handling > > of NULL values varies ...". > > > > Similarly, by default Postgres treats \N as a null, although it can be > > configured to act differently. See > > http://www.postgresql.org/docs/8.3/interactive/sql-copy.html starting with > > "null string". > > > > Maybe the best way to handle this would be to make AW output nulls as \N > > instead of empty strings. Would you be willing to look into that? > > > > I agree it's best to have AW-ETL work the same for both adapters. > > > > Best, > > Jacob > > > > > > ----- "Darin Wilson" <[EMAIL PROTECTED]> wrote: > > > >> Hi Jacob- > >> > >> I finally got around to testing your fixes. Unfortunately, MySQL does > >> not > >> convert empty strings to nulls out of the box, nor could I find a way > >> to > >> make it support this behavior. (This was after a small amount of > >> googling > >> around; there may be an answer somewhere that I just couldn't find - > >> maybe > >> someone here knows?). > >> > >> I'll keep poking around a little bit and see if I can see a way around > >> this > >> - it would be a bummer if the ETL framework exhibited radically > >> different > >> behavior depending on which adapter it was using. > >> > >> Darin > >> > >> > >> > >>> From: Jacob Maine <[EMAIL PROTECTED]> > >>> Date: Fri, 17 Oct 2008 11:07:53 -0700 (PDT) > >>> To: Darin Wilson <[EMAIL PROTECTED]> > >>> Cc: <activewarehouse-discuss@rubyforge.org> > >>> Subject: Re: [Activewarehouse-discuss] would the real codebase > >> please stand > >>> up? :) > >>> > >>> Hi Darin - > >>> > >>> Please pull from my adapter_extensions branch > >>> (http://github.com/mainej/adapter_extensions/tree) or wait until > >> Anthony pulls > >>> my changes. Unfortunately, I'm not sure that will fix > >>> test_bulk_import_with_empties. I can't be sure because I don't have > >> MySQL > >>> installed, and don't have time to do that right now. > >>> > >>> The change I made to adapter_extensions allows Postgres to import > >> empty > >>> strings as nulls. That's what the test is about. I put the test in > >> for both > >>> adapters, hoping that MySQL would do the right thing out of the box. > >> If it's > >>> not working, I suggest either making your proposed change, or > >> looking at > >>> adapter_extensions to see if you can get MySQL to handle empty > >> strings in a > >>> more useful way. I'll look at this stuff more this weekend to make > >> sure my > >>> system is up-to-date and tests are passing for Postgres. > >>> > >>> Hopefully with two groups developing for Postgres and MySQL, we'll > >> get AW & AW > >>> ETL back into a stable state. > >>> > >>> Best, > >>> Jacob > >>> > >>> > >>> ----- Darin Wilson <[EMAIL PROTECTED]> wrote: > >>>> I just pulled the latest from your github repo and ran the ETL test > >> suite > >>>> locally against MySQL. I got 10 errors that were about SCDs, so I'm > >> assuming > >>>> those were the same ones you mentioned in your last message(?). > >>>> > >>>> There was one other error: > >>>> > >>>> 1) Failure: > >>>> test_bulk_import_with_empties(ProcessorTest) > >> [./test/processor_test.rb:19]: > >>>> <ActiveRecord::StatementInvalid> exception expected but none was > >> thrown. > >>>> > >>>> It turns out that MySQL is much less strict about what goes into > >> the bulk > >>>> loader - if you try to load a row with an empty value into a column > >> that > >>>> disallows null, MySQL just skips it without an error. Nice! :) > >>>> > >>>> I'm not sure how you'd like to handle that. I tried poking around a > >> bit to > >>>> see if there's a way to force MySQL into being more strict, but I > >> couldn't > >>>> get anything to work reliably. The simplest fix would be to do > >> something > >>>> like this in the test: > >>>> > >>>> unless ETL::Engine.connection(:data_warehouse).class == > >>>> ActiveRecord::ConnectionAdapters::MysqlAdapter > >>>> assert_raise(ActiveRecord::StatementInvalid) { > >>>> do_bulk_import('bulk_import_with_empties.txt') } > >>>> end > >>>> > >>>> which sort of invalidates the test, but it doesn't seem like > >> there's a way > >>>> to make this test actually work with MySQL. Your call. ;) > >>>> > >>>> Darin > >>>> > >>>> > >>>>> From: Anthony Eden <[EMAIL PROTECTED]> > >>>>> Date: Thu, 16 Oct 2008 15:23:04 -0400 > >>>>> To: Darin Wilson <[EMAIL PROTECTED]> > >>>>> Cc: <activewarehouse-discuss@rubyforge.org> > >>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase > >> please stand > >>>>> up? :) > >>>>> > >>>>> I think I have most of the fixes to ETL in for the 2.1 changes. > >> I've > >>>>> started changing over some of the tests to use shoulda 1.x, so if > >> you > >>>>> want to run the built in tests then you'll need the shoulda gem. > >> There > >>>>> are currently two failing tests related to the SCD > >> implementation. > >>>>> Jacob, was that an area you worked on? If so could you take a look > >> at > >>>>> the failing tests? Also, I currently have tested with Postgres > >> locally > >>>>> but not MySQL, so if someone wants to fight through the test suite > >> on > >>>>> MySQL, be my guest. :-) > >>>>> > >>>>> Thanks. > >>>>> > >>>>> -Anthony > >>>>> > >>>>> On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson > >>>>> <[EMAIL PROTECTED]> wrote: > >>>>>> Cool - that is what I was going to start working on. :) > >>>>>> > >>>>>> Please let us know when your 2.1 fixes to ETL are checked in - > >> I'd like to > >>>>>> try it out with our ETL (and I have a bunch of test cases for > >> it). > >>>>>> > >>>>>> Thanks! > >>>>>> Darin > >>>>>> > >>>>>>> From: Anthony Eden <[EMAIL PROTECTED]> > >>>>>>> Date: Thu, 16 Oct 2008 06:59:41 -0400 > >>>>>>> To: Marty Haught <[EMAIL PROTECTED]> > >>>>>>> Cc: <activewarehouse-discuss@rubyforge.org> > >>>>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase > >> please > >>>>>>> stand > >>>>>>> up? :) > >>>>>>> > >>>>>>> I've started the upgrade to 2.1. I've already modified > >> ActiveWarehouse > >>>>>>> ETL to handle the new migration process and I have it running > >> right > >>>>>>> now under 2.1. Once I get past my first ETL load on the project > >> I'm > >>>>>>> working on I'll move onto the AW plugin. > >>>>>>> > >>>>>>> -Anthony > >>>>>>> _______________________________________________ > >>>>>>> Activewarehouse-discuss mailing list > >>>>>>> Activewarehouse-discuss@rubyforge.org > >>>>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> Activewarehouse-discuss mailing list > >>>>>> Activewarehouse-discuss@rubyforge.org > >>>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > >>>>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> Activewarehouse-discuss mailing list > >>>> Activewarehouse-discuss@rubyforge.org > >>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > >>> > >> > >> > >> _______________________________________________ > >> Activewarehouse-discuss mailing list > >> Activewarehouse-discuss@rubyforge.org > >> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss@rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
_______________________________________________ Activewarehouse-discuss mailing list Activewarehouse-discuss@rubyforge.org http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
