I tried a couple of times to respond to this, but it seems like my
messages weren't going through. Trying again here with a different account
- my apologies to all if this lands on the list multiple times. :S
Anyway, once more with feeling...
---------------------------
Hi Bob-
The .ctl files are just plain old Ruby code, so you can insert
conditionals right into the .ctl file to control the flow. In your case,
you might try something like:
if mode == 'streaming'
destination :out {
:database => 'my_db',
etc...
},
{
}
else
destination :out, {
:file => 'outfile.txt',
:separator => ","
},
{
:order => [ ... ]
}
post_process :bulk_import, {
:file => 'outfile.txt',
:target => 'target_db',
etc...
}
end
If this logic needs to be shared across different .ctl files, you can move
the code into helper methods in a separate .rb file, then call the methods
from within the .ctl file
HTH,
Darin
> From: Bob Cotton <[EMAIL PROTECTED]>
> Date: May 7, 2008 4:06:33 AM PDT
> To: activewarehouse-discuss@rubyforge.org
> Subject: [Activewarehouse-discuss] Reusing .ctl files
>
>
> I have an ETL process that needs to run in two modes:
>
>
> - Loading files
> - Streaming
>
>
> My destination needs to be bulk loaded when reading from files, and
> writing directly to the DB when streaming.
>
> I've managed to subclass the SAX parser to read from STDIN for
> streaming.
>
> I tried splitting the source and destination into different .ctl files
> and wrapping them in a batch file, but the pipeline was not propagated
> between the control files.
>
> What's the best way to factor out and reuse portions of the ETL
> pipeline?
>
> Thanks
> -Bob
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss@rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>
>
>
_______________________________________________
Activewarehouse-discuss mailing list
Activewarehouse-discuss@rubyforge.org
http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
