Re: [EXT] Re: Primary Only Content Migration

2018-06-08 Thread Joe Witt
t;> >> > >> >> > I'm not sure what that design would look like just yet - but agreeing >> >> > this is an important step to take soon. If you want to start >> >> > sketching out design ideas that would be awesome. >> >> > >

Re: [EXT] Re: Primary Only Content Migration

2018-06-08 Thread Brandon DeVries
gt; pwi...@micron.com> > >> wrote: > >> >> > >> >> Joe, > >> >> > >> >> I agree it is a lot of work, which is why I was thinking of starting > >> with a processor that could do some of these operations before looking &

Re: [EXT] Re: Primary Only Content Migration

2018-06-08 Thread Bryan Bende
, >> >> >> >> I agree it is a lot of work, which is why I was thinking of starting >> with a processor that could do some of these operations before looking >> further. If the processor could move flowfile's between nodes in the >> cluster it would be a

Re: [EXT] Re: Primary Only Content Migration

2018-06-08 Thread Pierre Villard
to work on it, and was trying to figure out if it could be done > using only a processor, or if larger changes would be needed for sure. > >> > >> --Peter > >> > >> -Original Message- > >> From: Joe Witt [mailto:joe.w...@gmail.com] > >&

Re: [EXT] Re: Primary Only Content Migration

2018-06-07 Thread Koji Kawamura
ure. >> >> --Peter >> >> -Original Message- >> From: Joe Witt [mailto:joe.w...@gmail.com] >> Sent: Thursday, June 7, 2018 3:34 PM >> To: dev@nifi.apache.org >> Subject: Re: [EXT] Re: Primary Only Content Migration >> >> Peter,

Re: [EXT] Re: Primary Only Content Migration

2018-06-07 Thread Joe Witt
; > --Peter > > -Original Message- > From: Joe Witt [mailto:joe.w...@gmail.com] > Sent: Thursday, June 7, 2018 3:34 PM > To: dev@nifi.apache.org > Subject: Re: [EXT] Re: Primary Only Content Migration > > Peter, > > It isn't a pattern that is well suppor

RE: [EXT] Re: Primary Only Content Migration

2018-06-07 Thread Peter Wicks (pwicks)
From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Thursday, June 7, 2018 3:34 PM To: dev@nifi.apache.org Subject: Re: [EXT] Re: Primary Only Content Migration Peter, It isn't a pattern that is well supported now in a cluster context. What is needed are automatically load balanced connec

Re: [EXT] Re: Primary Only Content Migration

2018-06-07 Thread Joe Witt
Peter, It isn't a pattern that is well supported now in a cluster context. What is needed are automatically load balanced connections with partitioning. This would mean a user could select a given relationship and indicate that data should automatically distributed and they should be able to exp

RE: [EXT] Re: Primary Only Content Migration

2018-06-07 Thread Peter Wicks (pwicks)
Bryan, We see this with large files that we have split up into smaller files and distributed across the cluster using site-to-site. We then want to merge them back together, so we send them to the primary node before continuing processing. --Peter -Original Message- From: Bryan Bende [