If the mode is changed to ab/wb as the default, will the eol option be used at all? I didn't know that was even an option for the file destination.
Anthony, should I open a ticket in the tracker to document these fixes so they can get integrated? Thanks CW On 9/18/07, Tom Fakes <[EMAIL PROTECTED]> wrote: > This is close, but I think there is one more thing that needs to be done > to make a control file work on all platforms. > > Here's the problem. On Windows, with the previous patches, this now > works: > FileDestination eol => \n (the default) > Bulk Import line_separator => \r\n > > But this control file will not work on Unixes > > I'd like the defaults to work on both platforms: > FileDestination eol => \n => \r\n (windows) \n (others) > Bulk Import line_separator => \n => \n (all platforms) > > Sadly, the Ruby file output, in text mode, converts \n to \r\n. The fix > for this is to open the file in Binary mode to stop all output > translation. > > A simple change in file_destination.rb - add 'b' to the mode strings > returned: > > def mode > append ? 'ab' : 'wb' > end > > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Chris Williams > Sent: Sunday, August 19, 2007 12:46 PM > To: Anthony Eden; activewarehouse-discuss@rubyforge.org > Subject: Re: [Activewarehouse-discuss] Figured out the issues I was > havingwith the demo app > > Anthony, here are some patches I came up with for the > activewarehouse-etl and adapter_extensions. With these patches and > adding the :line_separator => '\r\n' to the bulk_import portion of the > etl scripts, it fixed the problem on windows with mysql. From what I > could tell, Postgres doesn't have a similar line_separator option and > I made an attempt at a fix for sqlserver. > > I am still working on creating some real test cases to pass on. > Either way, he is a description of the changes. > > activewarehouse-etl/lib/etl/processor/bulk_import_processor.rb > I created a new lines item in the hash. I did this since the that > option is line based not field based. > > adapter_extensions/lib/adapter_extensions/connection_adapters/mysql_adap > ter.rb > Added the additional commands to the MySQL copy command to define the > line_separator. > > adapter_extensions/lib/adapter_extensions/connection_adapters/sqlserver_ > adapter.rb > I added a -r to the bcp command to define the line_separator (not > tested) > > author_dimension.txt > Here is an example of file that has the \r\n for the end of line. > > Let me know if you have any questions. > Thanks > CW > > On 8/19/07, Anthony Eden <[EMAIL PROTECTED]> wrote: > > On 8/19/07, Chris Williams <[EMAIL PROTECTED]> wrote: > > > I think I have narrowed it down to the bulk importer but I have hit > a > > > wall. I dumped the authors table from my MySQL database and the > extra > > > character is \r. I tried changing the author_dimension.ctl to > include > > > the :line_separator => '\r\n' to the post_process item but that > didn't > > > help. Where is the conn.bulk_load defined? Is that outside the ETL > > > code base? I searched the code and didn't see bulk_load referenced. > > > > It's defined in the adapter_extensions library, which is another > > library under the ActiveWarehouse umbrella project. > > > > V/r > > Anthony > > > > -- > > Cell: 808 782-5046 > > Current Location: Melbourne, FL > > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss@rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > _______________________________________________ Activewarehouse-discuss mailing list Activewarehouse-discuss@rubyforge.org http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
