This is close, but I think there is one more thing that needs to be done
to make a control file work on all platforms.
Here's the problem. On Windows, with the previous patches, this now
works:
FileDestination eol => \n (the default)
Bulk Import line_separator => \r\n
But this control file will not work on Unixes
I'd like the defaults to work on both platforms:
FileDestination eol => \n => \r\n (windows) \n (others)
Bulk Import line_separator => \n => \n (all platforms)
Sadly, the Ruby file output, in text mode, converts \n to \r\n. The fix
for this is to open the file in Binary mode to stop all output
translation.
A simple change in file_destination.rb - add 'b' to the mode strings
returned:
def mode
append ? 'ab' : 'wb'
end
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of
Chris Williams
Sent: Sunday, August 19, 2007 12:46 PM
To: Anthony Eden; activewarehouse-discuss@rubyforge.org
Subject: Re: [Activewarehouse-discuss] Figured out the issues I was
havingwith the demo app
Anthony, here are some patches I came up with for the
activewarehouse-etl and adapter_extensions. With these patches and
adding the :line_separator => '\r\n' to the bulk_import portion of the
etl scripts, it fixed the problem on windows with mysql. From what I
could tell, Postgres doesn't have a similar line_separator option and
I made an attempt at a fix for sqlserver.
I am still working on creating some real test cases to pass on.
Either way, he is a description of the changes.
activewarehouse-etl/lib/etl/processor/bulk_import_processor.rb
I created a new lines item in the hash. I did this since the that
option is line based not field based.
adapter_extensions/lib/adapter_extensions/connection_adapters/mysql_adap
ter.rb
Added the additional commands to the MySQL copy command to define the
line_separator.
adapter_extensions/lib/adapter_extensions/connection_adapters/sqlserver_
adapter.rb
I added a -r to the bcp command to define the line_separator (not
tested)
author_dimension.txt
Here is an example of file that has the \r\n for the end of line.
Let me know if you have any questions.
Thanks
CW
On 8/19/07, Anthony Eden <[EMAIL PROTECTED]> wrote:
> On 8/19/07, Chris Williams <[EMAIL PROTECTED]> wrote:
> > I think I have narrowed it down to the bulk importer but I have hit
a
> > wall. I dumped the authors table from my MySQL database and the
extra
> > character is \r. I tried changing the author_dimension.ctl to
include
> > the :line_separator => '\r\n' to the post_process item but that
didn't
> > help. Where is the conn.bulk_load defined? Is that outside the ETL
> > code base? I searched the code and didn't see bulk_load referenced.
>
> It's defined in the adapter_extensions library, which is another
> library under the ActiveWarehouse umbrella project.
>
> V/r
> Anthony
>
> --
> Cell: 808 782-5046
> Current Location: Melbourne, FL
>
_______________________________________________
Activewarehouse-discuss mailing list
Activewarehouse-discuss@rubyforge.org
http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
