Raja,

To what system are you migrating? If it is a straight DB migration, you
should be able to use ExecuteSQL -> PutDatabaseRecord. Just make sure your
Fetch Size property on ExecuteSQL is not set to the default of zero or the
PostgreSQL driver will try to fetch the whole thing into memory. For 15 GB
you could probably use 10000 as the Fetch Size, if that seems slow then
1000 might work better. You can also set Max Rows Per FlowFile to match
Fetch Size, then each "chunk" will be handled individually by
PutDatabaseRecord, rather than building a single 15 GB FlowFile to send
"downstream". The downside is that if some FlowFiles succeed and others
fail, you'll need to do something with the failed ones and eventually route
them back to PutDatabaseRecord in the hopes they will pass the second time.

Regards,
Matt

Reply via email to