The eol option is used - the default value of \n is used, which ends up in the file as a single character on all platforms, which MySQL reads correctly on all platforms. Don't know about other bulk loaders.
I don't know if the Windows Ruby libraries convert any other character sequences in Text mode that would cause problems elsewhere. Binary mode only works in Windows, but it doesn't break on my Redhat installation at least. -----Original Message----- From: Chris Williams [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 18, 2007 4:53 PM To: Tom Fakes Cc: activewarehouse-discuss@rubyforge.org Subject: Re: [Activewarehouse-discuss] Figured out the issues I was havingwith the demo app If the mode is changed to ab/wb as the default, will the eol option be used at all? I didn't know that was even an option for the file destination. Anthony, should I open a ticket in the tracker to document these fixes so they can get integrated? Thanks CW On 9/18/07, Tom Fakes <[EMAIL PROTECTED]> wrote: > This is close, but I think there is one more thing that needs to be done > to make a control file work on all platforms. > > Here's the problem. On Windows, with the previous patches, this now > works: > FileDestination eol => \n (the default) > Bulk Import line_separator => \r\n > > But this control file will not work on Unixes > > I'd like the defaults to work on both platforms: > FileDestination eol => \n => \r\n (windows) \n (others) > Bulk Import line_separator => \n => \n (all platforms) > > Sadly, the Ruby file output, in text mode, converts \n to \r\n. The fix > for this is to open the file in Binary mode to stop all output > translation. > > A simple change in file_destination.rb - add 'b' to the mode strings > returned: > > def mode > append ? 'ab' : 'wb' > end > > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Chris Williams > Sent: Sunday, August 19, 2007 12:46 PM > To: Anthony Eden; activewarehouse-discuss@rubyforge.org > Subject: Re: [Activewarehouse-discuss] Figured out the issues I was > havingwith the demo app > > Anthony, here are some patches I came up with for the > activewarehouse-etl and adapter_extensions. With these patches and > adding the :line_separator => '\r\n' to the bulk_import portion of the > etl scripts, it fixed the problem on windows with mysql. From what I > could tell, Postgres doesn't have a similar line_separator option and > I made an attempt at a fix for sqlserver. > > I am still working on creating some real test cases to pass on. > Either way, he is a description of the changes. > > activewarehouse-etl/lib/etl/processor/bulk_import_processor.rb > I created a new lines item in the hash. I did this since the that > option is line based not field based. > > adapter_extensions/lib/adapter_extensions/connection_adapters/mysql_adap > ter.rb > Added the additional commands to the MySQL copy command to define the > line_separator. > > adapter_extensions/lib/adapter_extensions/connection_adapters/sqlserver_ > adapter.rb > I added a -r to the bcp command to define the line_separator (not > tested) > > author_dimension.txt > Here is an example of file that has the \r\n for the end of line. > > Let me know if you have any questions. > Thanks > CW > > On 8/19/07, Anthony Eden <[EMAIL PROTECTED]> wrote: > > On 8/19/07, Chris Williams <[EMAIL PROTECTED]> wrote: > > > I think I have narrowed it down to the bulk importer but I have hit > a > > > wall. I dumped the authors table from my MySQL database and the > extra > > > character is \r. I tried changing the author_dimension.ctl to > include > > > the :line_separator => '\r\n' to the post_process item but that > didn't > > > help. Where is the conn.bulk_load defined? Is that outside the ETL > > > code base? I searched the code and didn't see bulk_load referenced. > > > > It's defined in the adapter_extensions library, which is another > > library under the ActiveWarehouse umbrella project. > > > > V/r > > Anthony > > > > -- > > Cell: 808 782-5046 > > Current Location: Melbourne, FL > > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss@rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > _______________________________________________ Activewarehouse-discuss mailing list Activewarehouse-discuss@rubyforge.org http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
