Re: JDBC External Table File Target

[obaidul karim]

Hi Matthew ,

I didn't read your reply(just now reading :)), however, I have implemented
in the same way as you described :)).

In fact, I followed steps in FetchFile processor to read files directly
from an input list generated by  ListFile processor.
My flow looks like below:

 Flow: An Input Folder > [1]ListFile(Standard Processor) > [2]NzLoad(My
Custom processor) > [3]NzSql (My Custom processor)

Now need your help on below:
1. As I am reading from ListFile, it should be thread safe (a single file
should not be processed by more than 1 threads on NzLoad), right ?
2. How can I make the full flow synchronous. I want it like below:
a. [1]ListFile should list <=N files for each run and then stop until [2] &
[3]c completes
b. [2]NzLoad will process all the <=N files(using multiple thread) and only
then then trigger [3]NzSql
c. [3]NzSql will complete it processing and then trigger [1]ListFile and so
on.

Please share me if you have any alternative idea.

Regards,
Obaid

On Fri, Jan 29, 2016 at 4:22 AM, Matthew Burgess 
wrote:

> I think an approach that could work is the following:
>
> 1) Start with a ListFile processor to locate the filenames of the files
> you want to load into netezza
>
> 2) Send to a ReplaceText processor, where you replace the entire contents
> of the flow file with:
> insert into MYTABLE
> select * FROM EXTERNAL '${filename}'
> USING () ;
>
>
> Note I’m using Expression Language to refer to the “filename” attribute.
> Each incoming flow file will have that attribute set to one of the files
> found by the ListFile processor.
>
>
>
> 3) Send to a PutSQL processor. Your SQL statement(s) are complete at this
> point so you don’t need arguments, parameters, etc.
>
> I haven’t tried this yet but it seems like it should work. If you give it
> a try, please let us know how it works for you :)
>
> Regards,
> Matt
>
>
> On 1/28/16, 3:04 PM, "Joe Percivall"  wrote:
>
> >Hello Obaid,
> >
> >Sorry no one has gotten back to you sooner, many of the developers are
> working diligently to get 0.5.0 done.
> >
> >I don't know too much about loading SQL tables but when you say "local
> file system filename", is this a file that exits on the target system? If
> so you may just be able to set that command because my understanding is
> that using ExecuteSQL you have an input query which gets sent to the target
> server. Then the target server then runs that command.
> >
> >Can anyone else that has more experience with SQL and loading tables
> chime in?
> >
> >
> >Joe
> >- - - - - -
> >Joseph Percivall
> >linkedin.com/in/Percivall
> >e: joeperciv...@yahoo.com
> >
> >
> >
> >On Saturday, January 23, 2016 12:31 AM, obaidul karim <
> obaidc...@gmail.com> wrote:
> >
> >
> >
> >Hi,
> >
> >I developing a nifi processor for netezza.
> >As per by experience the best way to load data to netezza is using
> netezza JDBC external table.
> >
> >I want to run below command to load a file within NiFi processor.
> >
> >insert into MYTABLE
> >select * FROM EXTERNAL ''
> >USING () ;
> >
> >My question is which file to usefor   ?
> >- Is it the flow files ? If yes then how can I get full path of a
> flowfile ?
> >- or can I directly load the file in a spool directory ?
> >
> >
> >Thanks for your help in advance.
> >
> >-Obaid
>
>

Re: JDBC External Table File Target

[Joe Percivall]

Hello Obaid,

Sorry no one has gotten back to you sooner, many of the developers are working 
diligently to get 0.5.0 done.

I don't know too much about loading SQL tables but when you say "local file 
system filename", is this a file that exits on the target system? If so you may 
just be able to set that command because my understanding is that using 
ExecuteSQL you have an input query which gets sent to the target server. Then 
the target server then runs that command.

Can anyone else that has more experience with SQL and loading tables chime in?
 

Joe
- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com



On Saturday, January 23, 2016 12:31 AM, obaidul karim  
wrote:



Hi,

I developing a nifi processor for netezza. 
As per by experience the best way to load data to netezza is using netezza JDBC 
external table.

I want to run below command to load a file within NiFi processor.

insert into MYTABLE 
select * FROM EXTERNAL '' 
USING () ;

My question is which file to usefor   ?
- Is it the flow files ? If yes then how can I get full path of a flowfile ?
- or can I directly load the file in a spool directory ?
 

Thanks for your help in advance.

-Obaid

JDBC External Table File Target

[obaidul karim]

Hi,

I developing a nifi processor for netezza.
As per by experience the best way to load data to netezza is using netezza
JDBC external table.

I want to run below command to load a file within NiFi processor.

insert into MYTABLE
select * FROM EXTERNAL ''
USING () ;

My question is which file to usefor   ?
- Is it the flow files ? If yes then how can I get full path of a flowfile ?
- or can I directly load the file in a spool directory ?


Thanks for your help in advance.

-Obaid