Re: Closing in on a NiFi 1.2.0 release?

[Andre]

All,

For some reason my canvas did not refresh after a process bounce (which
generally occurs) but reloading page allows for modifications.

Cheers

On Wed, May 3, 2017 at 7:43 AM, Andre  wrote:

> folks,
>
> I was just working to debug the final thorns found reviewing NIFI-3726 and
> noticed an odd behavior and wanted to confirm.
>
> If I recall correctly in the past users could simply replace a processor
> NAR file and even if that NAR existed the flow would continue to work.
>
> I just replaced
>
> cp ~/nifi/nifi-nar-bundles/nifi-cybersecurity-bundle/nifi-
> cybersecurity-nar/target/nifi-cybersecurity-nar-1.2.0-SNAPSHOT.nar
> ~/devel/nifi-1.2.0-SNAPSHOT/lib/nifi-cybersecurity-nar-1.2.0-SNAPSHOT.nar
>
> (note the different ~/nifi ~/devel used to ensure I don't explode the rest
> of the already compiled components).
>
> When I try to make changes to the flow I am displayed with the following
> error:
>
> [image: Inline image 1]
>
> This happens even when I try to drag and drop connected processors around
> the canvas.
>
>
> Oddly enough I can still add and delete components to the canvas but
> whatever touches the tainted processor cannot be modified at all.
>
> Examples of messages:
>
> *Attempt to move*
>
> Component Position
> [5, cb0a31ac-015b-1000-7473-873a47eb702e, 
> cb0a52ab-015b-1000-e43a-f6293a9ae99d]
> is not the most up-to-date revision. This component appears to have been
> modified
>
>
> *Attempt to delete a downstream processor*
> Error
> [1, cb0a31ac-015b-1000-7473-873a47eb702e, 
> cb0b2ae4-015b-1000-35a8-9eaf6a45fc6a]
> is not the most up-to-date revision. This component appears to have been
> modified
>
>
> I don't have a 1.1.0 instance around me at the moment but I vaguely
> remember being able to do that in the past.
>
> Can someone confirm this is new and expected behavior?
>
> Cheers
>
>
> On Wed, May 3, 2017 at 5:54 AM, Andy LoPresto 
> wrote:
>
>> I’ll review & merge as soon as they are available.
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On May 2, 2017, at 3:51 PM, Bryan Bende  wrote:
>>
>> Thanks Drew. These seem like good candidates for the release.
>>
>> On Tue, May 2, 2017 at 3:42 PM, Andrew Lim 
>> wrote:
>>
>> There are three doc updates/additions that would be great to include in
>> the RC:
>>
>> https://issues.apache.org/jira/browse/NIFI-3701
>> https://issues.apache.org/jira/browse/NIFI-3773
>> https://issues.apache.org/jira/browse/NIFI-3774
>>
>> Sarah Olson and I have been working on these.  We should have PRs
>> submitted for them very soon.
>>
>> -Drew
>>
>>
>> On May 2, 2017, at 2:11 PM, Aldrin Piri  wrote:
>>
>> Haven't had much luck in getting our Docker efforts incorporated into
>> Docker Hub.  As a result I have created an issue to track that integration
>> [1] and resolved the original issue.
>>
>> We can evaluate our options and figure out the best path forward.  At this
>> time procedures are not yet well established within ASF to support
>> configuring these builds.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3772
>>
>> On Tue, May 2, 2017 at 11:13 AM, Andrew Lim 
>> wrote:
>>
>> I will be making updates to the Release Notes and Migration Guidance doc
>> regarding the TLS 1.2 version support.  Tracked by:
>>
>> https://issues.apache.org/jira/browse/NIFI-3720
>>
>>
>> -Drew
>>
>>
>> On May 2, 2017, at 11:08 AM, Joe Witt  wrote:
>>
>> Those are great updates.  I'd recommend we avoid highlighting the
>> versions of UI components though.
>>
>> Thanks
>>
>>
>> On Tue, May 2, 2017 at 11:03 AM, Scott Aslan 
>>
>> wrote:
>>
>> Hey Bryan,
>>
>> Please include the following in the release notes:
>>
>>
>> - Core UI
>>- Circular references have been removed and the code modularized.
>>- Upgraded Node version to 6.9.3.
>>- Upgraded npm version to 3.10.10.
>>- Upgraded jQuery version to 3.1.1.
>>- Upgraded D3 version to 3.5.17.
>>- Reduced download size by removing bundled dependencies.
>> - User Experience Improvements
>> - Ever wish that it was easier to align components on the canvas? Me
>>too...and now you can!
>>- We now provide deep links to any component(s) on the canvas. This
>>will help make collaborating and sharing more natural.
>>- Users will enjoy a better understanding of the scope of
>>
>> Controller
>>
>>Services through an improved experience.
>>- All actions available on the operate palette are now also
>>
>> available
>>
>>under the context menu too!
>>
>> Thanks!
>>
>> On Tue, May 2, 2017 at 10:17 AM, Bryan Bende  wrote:
>>
>> All,
>>
>> Looks like we are closer than we have ever been!
>>
>> Down to three outstanding JIRAs...
>>
>> - NIFI-1833 (Azure Processors) - Active review and discussion
>> occurring, appears to be close
>> - NIFI-3260 (Official Docker Image) - Active discussion, looks like
>> this may be moved from the release to a separate effort
>> - NIFI-3765 (Status operat

Re: Closing in on a NiFi 1.2.0 release?

[Andre]

folks,

I was just working to debug the final thorns found reviewing NIFI-3726 and
noticed an odd behavior and wanted to confirm.

If I recall correctly in the past users could simply replace a processor
NAR file and even if that NAR existed the flow would continue to work.

I just replaced

cp
~/nifi/nifi-nar-bundles/nifi-cybersecurity-bundle/nifi-cybersecurity-nar/target/nifi-cybersecurity-nar-1.2.0-SNAPSHOT.nar
~/devel/nifi-1.2.0-SNAPSHOT/lib/nifi-cybersecurity-nar-1.2.0-SNAPSHOT.nar

(note the different ~/nifi ~/devel used to ensure I don't explode the rest
of the already compiled components).

When I try to make changes to the flow I am displayed with the following
error:

[image: Inline image 1]

This happens even when I try to drag and drop connected processors around
the canvas.


Oddly enough I can still add and delete components to the canvas but
whatever touches the tainted processor cannot be modified at all.

Examples of messages:

*Attempt to move*

Component Position
[5, cb0a31ac-015b-1000-7473-873a47eb702e,
cb0a52ab-015b-1000-e43a-f6293a9ae99d] is not the most up-to-date revision.
This component appears to have been modified


*Attempt to delete a downstream processor*
Error
[1, cb0a31ac-015b-1000-7473-873a47eb702e,
cb0b2ae4-015b-1000-35a8-9eaf6a45fc6a] is not the most up-to-date revision.
This component appears to have been modified


I don't have a 1.1.0 instance around me at the moment but I vaguely
remember being able to do that in the past.

Can someone confirm this is new and expected behavior?

Cheers


On Wed, May 3, 2017 at 5:54 AM, Andy LoPresto  wrote:

> I’ll review & merge as soon as they are available.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 2, 2017, at 3:51 PM, Bryan Bende  wrote:
>
> Thanks Drew. These seem like good candidates for the release.
>
> On Tue, May 2, 2017 at 3:42 PM, Andrew Lim 
> wrote:
>
> There are three doc updates/additions that would be great to include in
> the RC:
>
> https://issues.apache.org/jira/browse/NIFI-3701
> https://issues.apache.org/jira/browse/NIFI-3773
> https://issues.apache.org/jira/browse/NIFI-3774
>
> Sarah Olson and I have been working on these.  We should have PRs
> submitted for them very soon.
>
> -Drew
>
>
> On May 2, 2017, at 2:11 PM, Aldrin Piri  wrote:
>
> Haven't had much luck in getting our Docker efforts incorporated into
> Docker Hub.  As a result I have created an issue to track that integration
> [1] and resolved the original issue.
>
> We can evaluate our options and figure out the best path forward.  At this
> time procedures are not yet well established within ASF to support
> configuring these builds.
>
> [1] https://issues.apache.org/jira/browse/NIFI-3772
>
> On Tue, May 2, 2017 at 11:13 AM, Andrew Lim 
> wrote:
>
> I will be making updates to the Release Notes and Migration Guidance doc
> regarding the TLS 1.2 version support.  Tracked by:
>
> https://issues.apache.org/jira/browse/NIFI-3720
>
>
> -Drew
>
>
> On May 2, 2017, at 11:08 AM, Joe Witt  wrote:
>
> Those are great updates.  I'd recommend we avoid highlighting the
> versions of UI components though.
>
> Thanks
>
>
> On Tue, May 2, 2017 at 11:03 AM, Scott Aslan 
>
> wrote:
>
> Hey Bryan,
>
> Please include the following in the release notes:
>
>
> - Core UI
>- Circular references have been removed and the code modularized.
>- Upgraded Node version to 6.9.3.
>- Upgraded npm version to 3.10.10.
>- Upgraded jQuery version to 3.1.1.
>- Upgraded D3 version to 3.5.17.
>- Reduced download size by removing bundled dependencies.
> - User Experience Improvements
> - Ever wish that it was easier to align components on the canvas? Me
>too...and now you can!
>- We now provide deep links to any component(s) on the canvas. This
>will help make collaborating and sharing more natural.
>- Users will enjoy a better understanding of the scope of
>
> Controller
>
>Services through an improved experience.
>- All actions available on the operate palette are now also
>
> available
>
>under the context menu too!
>
> Thanks!
>
> On Tue, May 2, 2017 at 10:17 AM, Bryan Bende  wrote:
>
> All,
>
> Looks like we are closer than we have ever been!
>
> Down to three outstanding JIRAs...
>
> - NIFI-1833 (Azure Processors) - Active review and discussion
> occurring, appears to be close
> - NIFI-3260 (Official Docker Image) - Active discussion, looks like
> this may be moved from the release to a separate effort
> - NIFI-3765 (Status operation for NodeManager) - Active review, looks
> like it could be merged relatively soon
>
> Once these are finalized we should be able to start the RC process.
>
> I've started putting together the release notes on the Wiki (still a
> work in progress):
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.2.0
>
> If anyone knows of anything that should be highlight

Re: failed to replicate request GET /NIFI-API/flow/process-groups/root to saxx:9090

[pradeepbill]

thanks Matt, will do and let you know soon.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/failed-to-replicate-request-GET-NIFI-API-flow-process-groups-root-to-saxx-9090-tp15612p15626.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: failed to replicate request GET /NIFI-API/flow/process-groups/root to saxx:9090

[pradeepbill]

thanks Bryan, I could not open the console on
http://sabc.abcd.local:9090/nifi.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/failed-to-replicate-request-GET-NIFI-API-flow-process-groups-root-to-saxx-9090-tp15612p15625.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Closing in on a NiFi 1.2.0 release?

[Andy LoPresto]

I’ll review & merge as soon as they are available.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On May 2, 2017, at 3:51 PM, Bryan Bende  wrote:
> 
> Thanks Drew. These seem like good candidates for the release.
> 
> On Tue, May 2, 2017 at 3:42 PM, Andrew Lim  wrote:
>> There are three doc updates/additions that would be great to include in the 
>> RC:
>> 
>> https://issues.apache.org/jira/browse/NIFI-3701
>> https://issues.apache.org/jira/browse/NIFI-3773
>> https://issues.apache.org/jira/browse/NIFI-3774
>> 
>> Sarah Olson and I have been working on these.  We should have PRs submitted 
>> for them very soon.
>> 
>> -Drew
>> 
>> 
>>> On May 2, 2017, at 2:11 PM, Aldrin Piri  wrote:
>>> 
>>> Haven't had much luck in getting our Docker efforts incorporated into
>>> Docker Hub.  As a result I have created an issue to track that integration
>>> [1] and resolved the original issue.
>>> 
>>> We can evaluate our options and figure out the best path forward.  At this
>>> time procedures are not yet well established within ASF to support
>>> configuring these builds.
>>> 
>>> [1] https://issues.apache.org/jira/browse/NIFI-3772
>>> 
>>> On Tue, May 2, 2017 at 11:13 AM, Andrew Lim 
>>> wrote:
>>> 
 I will be making updates to the Release Notes and Migration Guidance doc
 regarding the TLS 1.2 version support.  Tracked by:
 
 https://issues.apache.org/jira/browse/NIFI-3720
 
 
 -Drew
 
 
> On May 2, 2017, at 11:08 AM, Joe Witt  wrote:
> 
> Those are great updates.  I'd recommend we avoid highlighting the
> versions of UI components though.
> 
> Thanks
> 
> 
> On Tue, May 2, 2017 at 11:03 AM, Scott Aslan 
 wrote:
>> Hey Bryan,
>> 
>> Please include the following in the release notes:
>> 
>> 
>> - Core UI
>>- Circular references have been removed and the code modularized.
>>- Upgraded Node version to 6.9.3.
>>- Upgraded npm version to 3.10.10.
>>- Upgraded jQuery version to 3.1.1.
>>- Upgraded D3 version to 3.5.17.
>>- Reduced download size by removing bundled dependencies.
>> - User Experience Improvements
>> - Ever wish that it was easier to align components on the canvas? Me
>>too...and now you can!
>>- We now provide deep links to any component(s) on the canvas. This
>>will help make collaborating and sharing more natural.
>>- Users will enjoy a better understanding of the scope of
 Controller
>>Services through an improved experience.
>>- All actions available on the operate palette are now also
 available
>>under the context menu too!
>> 
>> Thanks!
>> 
>> On Tue, May 2, 2017 at 10:17 AM, Bryan Bende  wrote:
>> 
>>> All,
>>> 
>>> Looks like we are closer than we have ever been!
>>> 
>>> Down to three outstanding JIRAs...
>>> 
>>> - NIFI-1833 (Azure Processors) - Active review and discussion
>>> occurring, appears to be close
>>> - NIFI-3260 (Official Docker Image) - Active discussion, looks like
>>> this may be moved from the release to a separate effort
>>> - NIFI-3765 (Status operation for NodeManager) - Active review, looks
>>> like it could be merged relatively soon
>>> 
>>> Once these are finalized we should be able to start the RC process.
>>> 
>>> I've started putting together the release notes on the Wiki (still a
>>> work in progress):
>>> https://cwiki.apache.org/confluence/display/NIFI/
>>> Release+Notes#ReleaseNotes-Version1.2.0
>>> 
>>> If anyone knows of anything that should be highlighted, feel free to
>>> mention it here and I can update the notes.
>>> 
>>> Thanks!
>>> 
>>> -Bryan
>>> 
>>> 
>>> On Thu, Apr 27, 2017 at 5:52 PM, Bryan Bende  wrote:
 Looks like we are down to just a few tickets, and all of them seem to
 have traction in terms of review and discussion.
 
 I'll keep an eye on the tickets and start trying to pull together
 anything I can to get ready for the RC process. Seems like we are
 close!
 
 On Tue, Apr 25, 2017 at 9:33 AM, Russell Bateman <
 r...@windofkeltia.com>
>>> wrote:
> Many thanks to all the industrious folk working on this tool that's
>>> become
> indispensable to so many. The community has my enduring gratitude and
> admiration!
> 
> Russ
> 
> 
> On 04/24/2017 07:16 PM, Joe Witt wrote:
>> 
>> Robert,
>> 
>> We're about 10 or so JIRAs away at this point and all of them have
 PRs
>> and progress on them.  We need to close those down and do some
>> stabilization/testing then a release candidate can happen.  So
 perhaps
>> next week we'l

Re: Closing in on a NiFi 1.2.0 release?

[Bryan Bende]

Thanks Drew. These seem like good candidates for the release.

On Tue, May 2, 2017 at 3:42 PM, Andrew Lim  wrote:
> There are three doc updates/additions that would be great to include in the 
> RC:
>
> https://issues.apache.org/jira/browse/NIFI-3701
> https://issues.apache.org/jira/browse/NIFI-3773
> https://issues.apache.org/jira/browse/NIFI-3774
>
> Sarah Olson and I have been working on these.  We should have PRs submitted 
> for them very soon.
>
> -Drew
>
>
>> On May 2, 2017, at 2:11 PM, Aldrin Piri  wrote:
>>
>> Haven't had much luck in getting our Docker efforts incorporated into
>> Docker Hub.  As a result I have created an issue to track that integration
>> [1] and resolved the original issue.
>>
>> We can evaluate our options and figure out the best path forward.  At this
>> time procedures are not yet well established within ASF to support
>> configuring these builds.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3772
>>
>> On Tue, May 2, 2017 at 11:13 AM, Andrew Lim 
>> wrote:
>>
>>> I will be making updates to the Release Notes and Migration Guidance doc
>>> regarding the TLS 1.2 version support.  Tracked by:
>>>
>>> https://issues.apache.org/jira/browse/NIFI-3720
>>>
>>>
>>> -Drew
>>>
>>>
 On May 2, 2017, at 11:08 AM, Joe Witt  wrote:

 Those are great updates.  I'd recommend we avoid highlighting the
 versions of UI components though.

 Thanks


 On Tue, May 2, 2017 at 11:03 AM, Scott Aslan 
>>> wrote:
> Hey Bryan,
>
> Please include the following in the release notes:
>
>
>  - Core UI
> - Circular references have been removed and the code modularized.
> - Upgraded Node version to 6.9.3.
> - Upgraded npm version to 3.10.10.
> - Upgraded jQuery version to 3.1.1.
> - Upgraded D3 version to 3.5.17.
> - Reduced download size by removing bundled dependencies.
>  - User Experience Improvements
>  - Ever wish that it was easier to align components on the canvas? Me
> too...and now you can!
> - We now provide deep links to any component(s) on the canvas. This
> will help make collaborating and sharing more natural.
> - Users will enjoy a better understanding of the scope of
>>> Controller
> Services through an improved experience.
> - All actions available on the operate palette are now also
>>> available
> under the context menu too!
>
> Thanks!
>
> On Tue, May 2, 2017 at 10:17 AM, Bryan Bende  wrote:
>
>> All,
>>
>> Looks like we are closer than we have ever been!
>>
>> Down to three outstanding JIRAs...
>>
>> - NIFI-1833 (Azure Processors) - Active review and discussion
>> occurring, appears to be close
>> - NIFI-3260 (Official Docker Image) - Active discussion, looks like
>> this may be moved from the release to a separate effort
>> - NIFI-3765 (Status operation for NodeManager) - Active review, looks
>> like it could be merged relatively soon
>>
>> Once these are finalized we should be able to start the RC process.
>>
>> I've started putting together the release notes on the Wiki (still a
>> work in progress):
>> https://cwiki.apache.org/confluence/display/NIFI/
>> Release+Notes#ReleaseNotes-Version1.2.0
>>
>> If anyone knows of anything that should be highlighted, feel free to
>> mention it here and I can update the notes.
>>
>> Thanks!
>>
>> -Bryan
>>
>>
>> On Thu, Apr 27, 2017 at 5:52 PM, Bryan Bende  wrote:
>>> Looks like we are down to just a few tickets, and all of them seem to
>>> have traction in terms of review and discussion.
>>>
>>> I'll keep an eye on the tickets and start trying to pull together
>>> anything I can to get ready for the RC process. Seems like we are
>>> close!
>>>
>>> On Tue, Apr 25, 2017 at 9:33 AM, Russell Bateman <
>>> r...@windofkeltia.com>
>> wrote:
 Many thanks to all the industrious folk working on this tool that's
>> become
 indispensable to so many. The community has my enduring gratitude and
 admiration!

 Russ


 On 04/24/2017 07:16 PM, Joe Witt wrote:
>
> Robert,
>
> We're about 10 or so JIRAs away at this point and all of them have
>>> PRs
> and progress on them.  We need to close those down and do some
> stabilization/testing then a release candidate can happen.  So
>>> perhaps
> next week we'll be at that point and have a release.  This is the
> right place to be watching to see it closing in.
>
> Thanks
> Joe
>
> On Mon, Apr 24, 2017 at 5:40 AM, roboh  wrote:
>>
>> Hi,
>>
>> is there any date set for 1.2.0 release yet?
>> We are running into an  NIFI-3389
>> 

Re: Closing in on a NiFi 1.2.0 release?

[Andrew Lim]

There are three doc updates/additions that would be great to include in the RC:

https://issues.apache.org/jira/browse/NIFI-3701
https://issues.apache.org/jira/browse/NIFI-3773
https://issues.apache.org/jira/browse/NIFI-3774

Sarah Olson and I have been working on these.  We should have PRs submitted for 
them very soon.

-Drew


> On May 2, 2017, at 2:11 PM, Aldrin Piri  wrote:
> 
> Haven't had much luck in getting our Docker efforts incorporated into
> Docker Hub.  As a result I have created an issue to track that integration
> [1] and resolved the original issue.
> 
> We can evaluate our options and figure out the best path forward.  At this
> time procedures are not yet well established within ASF to support
> configuring these builds.
> 
> [1] https://issues.apache.org/jira/browse/NIFI-3772
> 
> On Tue, May 2, 2017 at 11:13 AM, Andrew Lim 
> wrote:
> 
>> I will be making updates to the Release Notes and Migration Guidance doc
>> regarding the TLS 1.2 version support.  Tracked by:
>> 
>> https://issues.apache.org/jira/browse/NIFI-3720
>> 
>> 
>> -Drew
>> 
>> 
>>> On May 2, 2017, at 11:08 AM, Joe Witt  wrote:
>>> 
>>> Those are great updates.  I'd recommend we avoid highlighting the
>>> versions of UI components though.
>>> 
>>> Thanks
>>> 
>>> 
>>> On Tue, May 2, 2017 at 11:03 AM, Scott Aslan 
>> wrote:
 Hey Bryan,
 
 Please include the following in the release notes:
 
 
  - Core UI
 - Circular references have been removed and the code modularized.
 - Upgraded Node version to 6.9.3.
 - Upgraded npm version to 3.10.10.
 - Upgraded jQuery version to 3.1.1.
 - Upgraded D3 version to 3.5.17.
 - Reduced download size by removing bundled dependencies.
  - User Experience Improvements
  - Ever wish that it was easier to align components on the canvas? Me
 too...and now you can!
 - We now provide deep links to any component(s) on the canvas. This
 will help make collaborating and sharing more natural.
 - Users will enjoy a better understanding of the scope of
>> Controller
 Services through an improved experience.
 - All actions available on the operate palette are now also
>> available
 under the context menu too!
 
 Thanks!
 
 On Tue, May 2, 2017 at 10:17 AM, Bryan Bende  wrote:
 
> All,
> 
> Looks like we are closer than we have ever been!
> 
> Down to three outstanding JIRAs...
> 
> - NIFI-1833 (Azure Processors) - Active review and discussion
> occurring, appears to be close
> - NIFI-3260 (Official Docker Image) - Active discussion, looks like
> this may be moved from the release to a separate effort
> - NIFI-3765 (Status operation for NodeManager) - Active review, looks
> like it could be merged relatively soon
> 
> Once these are finalized we should be able to start the RC process.
> 
> I've started putting together the release notes on the Wiki (still a
> work in progress):
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.2.0
> 
> If anyone knows of anything that should be highlighted, feel free to
> mention it here and I can update the notes.
> 
> Thanks!
> 
> -Bryan
> 
> 
> On Thu, Apr 27, 2017 at 5:52 PM, Bryan Bende  wrote:
>> Looks like we are down to just a few tickets, and all of them seem to
>> have traction in terms of review and discussion.
>> 
>> I'll keep an eye on the tickets and start trying to pull together
>> anything I can to get ready for the RC process. Seems like we are
>> close!
>> 
>> On Tue, Apr 25, 2017 at 9:33 AM, Russell Bateman <
>> r...@windofkeltia.com>
> wrote:
>>> Many thanks to all the industrious folk working on this tool that's
> become
>>> indispensable to so many. The community has my enduring gratitude and
>>> admiration!
>>> 
>>> Russ
>>> 
>>> 
>>> On 04/24/2017 07:16 PM, Joe Witt wrote:
 
 Robert,
 
 We're about 10 or so JIRAs away at this point and all of them have
>> PRs
 and progress on them.  We need to close those down and do some
 stabilization/testing then a release candidate can happen.  So
>> perhaps
 next week we'll be at that point and have a release.  This is the
 right place to be watching to see it closing in.
 
 Thanks
 Joe
 
 On Mon, Apr 24, 2017 at 5:40 AM, roboh  wrote:
> 
> Hi,
> 
> is there any date set for 1.2.0 release yet?
> We are running into an  NIFI-3389
>    issue and
>> would
> like
> to
> know when we can expect 1.2.0 release to be available.
> 
> Thanks,
> Robert
> 
> 
> 
> --
>>

Re: how to write date to redis

[Matt Burgess]

Jython is not the most performant script engine, if you are familiar
with Groovy or Javascript I would recommend porting to one of those
for better performance.

There are some examples [1] [2] of in-progress development of Redis
processors for NiFi. Although they are fetching from Redis, not
pushing to it, I bet you could reuse a lot of the client code in your
own custom Java processor or a scripting processor. I have examples in
Groovy and Javascript for Hazelcast (albeit for fetching not pushing)
[3].

Regards,
Matt

[1] https://github.com/qntfy/nifi-redis
[2] https://github.com/ryanleary/nifi-redis
[3] http://funnifi.blogspot.com/2016/02/executescript-using-modules.html

On Tue, May 2, 2017 at 2:33 PM, Bryan Bende  wrote:
> Hello,
>
> I can't say for sure why your script isn't performing well, but I
> would guess that you would get better performance from a custom
> processor written in Java that wrapped a Redis client like Jedis.
>
> If you are interested in working on this I think it would be something
> the community would be interested in. There are probably a number of
> interesting processors that could be created given all of Redis'
> operations.
>
> Alternatively, if you provided your script, others on this list may be
> able to notice something that could improve the performance.
>
> Thanks,
>
> Bryan
>
>
> On Mon, May 1, 2017 at 10:49 PM, L Robin  wrote:
>> Hello everyone,
>> Now I want write the flowfile data to redis,I use the ExecuteScript 
>> processor with python,buth the performance is too low,Who can give me some 
>> advice！
>>
>> Thanks

Re: Not able to ingest the csv data to RDBS Database

[Bryan Bende]

In order for ReplaceText to reference ${city} and ${count} there need
to be flow file attributes available for these values, and if you
picked up a file from GetFile and went directly to ReplaceText then
they wouldn't be there.

You would likely have to do something like GetFile -> SplitText with
line count 1 -> ExtractText to get city and count in attributes ->
ReplaceText -> PutSQL.

-Bryan


On Tue, May 2, 2017 at 11:58 AM, suman@cuddle.ai
 wrote:
> Hi ,
> I have a csv file and i want to ingest the csv file data into the database.
>
> I have used getfile,RelaceText,PutSQL Processors.
>
> It's failing in replaceText Processor.
>
> File Sample :
> City,Count
> Mumbai,10
> Mumbai,10
> Pune,10
> Pune,10
>
> ReplaceText :
>
> INSERT INTO CITY(city,count) values ('${city}',${count})
>
> Please help.
>
>
>
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Not-able-to-ingest-the-csv-data-to-RDBS-Database-tp15610.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: failed to replicate request GET /NIFI-API/flow/process-groups/root to saxx:9090

[Matt Gilman]

Pradeep,

Trying increasing the cluster read timeout. It's specified in property

nifi.cluster.node.read.timeout

You'll need to update this configuration on each node of your cluster as
it's relevant for the node currently acting as the coordinator.

Thanks

Matt

On Tue, May 2, 2017 at 2:37 PM, Bryan Bende  wrote:

> Pradeep,
>
> This means the NiFi you are accessing the UI from is trying to
> replicate a request to another NiFi in the cluster running on
> sabc.abcd.local:9090 and couldn't complete the replication due to a
> read timeout.
>
> Can you verify there is a NiFi running at sabc.abcd.local:9090? Can
> you access the UI of the other NiFi at
> http://sabc.abcd.local:9090/nifi ?
>
> -Bryan
>
> On Tue, May 2, 2017 at 12:14 PM, pradeepbill 
> wrote:
> > hello folks, I see this exception, and can not open the NIFI
> console.There is
> > connectivity between the nodes in the cluster on port 9090.Any clues
> would
> > help.
> >
> > 2017-05-02 11:50:18,607 WARN [Replicate Request Thread-2]
> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request
> GET
> > /nifi-api/flow/process-groups/root to sabc.abcd.local:9090 due to {}
> > com.sun.jersey.api.client.ClientHandlerException:
> > java.net.SocketTimeoutException: Read timed out
> > at
> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(
> URLConnectionClientHandler.java:155)
> > ~[jersey-client-1.19.jar:1.19]
> > at com.sun.jersey.api.client.Client.handle(Client.java:652)
> > ~[jersey-client-1.19.jar:1.19]
> > at
> > com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(
> GZIPContentEncodingFilter.java:123)
> > ~[jersey-client-1.19.jar:1.19]
> > at
> > com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
> > ~[jersey-client-1.19.jar:1.19]
> > at
> > com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
> > ~[jersey-client-1.19.jar:1.19]
> > at
> > com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
> > ~[jersey-client-1.19.jar:1.19]
> > at
> > org.apache.nifi.cluster.coordination.http.replication.
> ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.
> java:578)
> > ~[nifi-framework-cluster-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
> > at
> > org.apache.nifi.cluster.coordination.http.replication.
> ThreadPoolRequestReplicator$NodeHttpRequest.run(
> ThreadPoolRequestReplicator.java:770)
> > ~[nifi-framework-cluster-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > [na:1.8.0_111]
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > [na:1.8.0_111]
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > [na:1.8.0_111]
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > [na:1.8.0_111]
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> > Caused by: java.net.SocketTimeoutException: Read timed out
> >
> > thanks
> > Pradeep
> >
> >
> >
> > --
> > View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/failed-to-replicate-request-
> GET-NIFI-API-flow-process-groups-root-to-saxx-9090-tp15612.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: failed to replicate request GET /NIFI-API/flow/process-groups/root to saxx:9090

[Bryan Bende]

Pradeep,

This means the NiFi you are accessing the UI from is trying to
replicate a request to another NiFi in the cluster running on
sabc.abcd.local:9090 and couldn't complete the replication due to a
read timeout.

Can you verify there is a NiFi running at sabc.abcd.local:9090? Can
you access the UI of the other NiFi at
http://sabc.abcd.local:9090/nifi ?

-Bryan

On Tue, May 2, 2017 at 12:14 PM, pradeepbill  wrote:
> hello folks, I see this exception, and can not open the NIFI console.There is
> connectivity between the nodes in the cluster on port 9090.Any clues would
> help.
>
> 2017-05-02 11:50:18,607 WARN [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/process-groups/root to sabc.abcd.local:9090 due to {}
> com.sun.jersey.api.client.ClientHandlerException:
> java.net.SocketTimeoutException: Read timed out
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
> ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.Client.handle(Client.java:652)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
> ~[jersey-client-1.19.jar:1.19]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578)
> ~[nifi-framework-cluster-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
> ~[nifi-framework-cluster-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_111]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> Caused by: java.net.SocketTimeoutException: Read timed out
>
> thanks
> Pradeep
>
>
>
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/failed-to-replicate-request-GET-NIFI-API-flow-process-groups-root-to-saxx-9090-tp15612.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: how to write date to redis

[Bryan Bende]

Hello,

I can't say for sure why your script isn't performing well, but I
would guess that you would get better performance from a custom
processor written in Java that wrapped a Redis client like Jedis.

If you are interested in working on this I think it would be something
the community would be interested in. There are probably a number of
interesting processors that could be created given all of Redis'
operations.

Alternatively, if you provided your script, others on this list may be
able to notice something that could improve the performance.

Thanks,

Bryan


On Mon, May 1, 2017 at 10:49 PM, L Robin  wrote:
> Hello everyone,
> Now I want write the flowfile data to redis,I use the ExecuteScript processor 
> with python,buth the performance is too low,Who can give me some advice！
>
> Thanks

Re: nifi-extension-utils

[Jeff]

Thanks for doing this, Bryan.

As part of the PR for NIFI-1833 (Azure Storage Blob processors), I've moved
AbstractListProcessor (and some supporting classes) into
nifi-commons/nifi-processor-utils, and had to rebase for this change.  It
was a pretty simple process.  If anyone else needs help if/when they are
affected by this, I'd be happy to lend a hand.  The rebase pretty much did
all the heavy lifting, I just had to manually move the files to the new
location for nifi-processor-utils before doing the "git clean -df".

- Jeff

On Mon, May 1, 2017 at 5:25 PM Bryan Bende  wrote:

> All,
>
> As part of NIFI-3724 which was merged to master earlier today, I made
> some slight refactoring to the organization of some utility code and
> wanted to mention it in case anyone is wondering where things moved...
>
> Previously under nifi-commons we had nifi-processor-utils and
> nifi-hadoop-utils which had become places to put abstract processors
> and processor related code. This type of utility code is a bit
> different from the rest of nifi-commons which is completely
> independent of anything under nifi-nar-bundles.
>
> So I created nifi-extension-utils under nifi-nar-bundles to contain
> code like abstract processors and things that need to reference
> controller service APIs. The structure currently looks like the
> following:
>
> nifi-nar-bundles
> - nifi-extension-utils
>   - nifi-processor-utils
>   - nifi-hadoop-utils
>   - nifi-record-utils
> - nifi-standard-record-utils
> - nifi-avro-record-utils
> - nifi-hadoop-record-utils
>
> Going forward the goal would be for nifi-commons to be independent of
> anything in nifi-nar-bundles.
>
> After rebasing you may have the residual projects still under
> nifi-commons and will want to clean up appropriately. "git clean -dn"
> will dry run the clean and then "git clean -df" will delete it.
>
> Thanks,
>
> Bryan
>

Re: Closing in on a NiFi 1.2.0 release?

[Aldrin Piri]

Haven't had much luck in getting our Docker efforts incorporated into
Docker Hub.  As a result I have created an issue to track that integration
[1] and resolved the original issue.

We can evaluate our options and figure out the best path forward.  At this
time procedures are not yet well established within ASF to support
configuring these builds.

[1] https://issues.apache.org/jira/browse/NIFI-3772

On Tue, May 2, 2017 at 11:13 AM, Andrew Lim 
wrote:

> I will be making updates to the Release Notes and Migration Guidance doc
> regarding the TLS 1.2 version support.  Tracked by:
>
> https://issues.apache.org/jira/browse/NIFI-3720
>
>
> -Drew
>
>
> > On May 2, 2017, at 11:08 AM, Joe Witt  wrote:
> >
> > Those are great updates.  I'd recommend we avoid highlighting the
> > versions of UI components though.
> >
> > Thanks
> >
> >
> > On Tue, May 2, 2017 at 11:03 AM, Scott Aslan 
> wrote:
> >> Hey Bryan,
> >>
> >> Please include the following in the release notes:
> >>
> >>
> >>   - Core UI
> >>  - Circular references have been removed and the code modularized.
> >>  - Upgraded Node version to 6.9.3.
> >>  - Upgraded npm version to 3.10.10.
> >>  - Upgraded jQuery version to 3.1.1.
> >>  - Upgraded D3 version to 3.5.17.
> >>  - Reduced download size by removing bundled dependencies.
> >>   - User Experience Improvements
> >>   - Ever wish that it was easier to align components on the canvas? Me
> >>  too...and now you can!
> >>  - We now provide deep links to any component(s) on the canvas. This
> >>  will help make collaborating and sharing more natural.
> >>  - Users will enjoy a better understanding of the scope of
> Controller
> >>  Services through an improved experience.
> >>  - All actions available on the operate palette are now also
> available
> >>  under the context menu too!
> >>
> >> Thanks!
> >>
> >> On Tue, May 2, 2017 at 10:17 AM, Bryan Bende  wrote:
> >>
> >>> All,
> >>>
> >>> Looks like we are closer than we have ever been!
> >>>
> >>> Down to three outstanding JIRAs...
> >>>
> >>> - NIFI-1833 (Azure Processors) - Active review and discussion
> >>> occurring, appears to be close
> >>> - NIFI-3260 (Official Docker Image) - Active discussion, looks like
> >>> this may be moved from the release to a separate effort
> >>> - NIFI-3765 (Status operation for NodeManager) - Active review, looks
> >>> like it could be merged relatively soon
> >>>
> >>> Once these are finalized we should be able to start the RC process.
> >>>
> >>> I've started putting together the release notes on the Wiki (still a
> >>> work in progress):
> >>> https://cwiki.apache.org/confluence/display/NIFI/
> >>> Release+Notes#ReleaseNotes-Version1.2.0
> >>>
> >>> If anyone knows of anything that should be highlighted, feel free to
> >>> mention it here and I can update the notes.
> >>>
> >>> Thanks!
> >>>
> >>> -Bryan
> >>>
> >>>
> >>> On Thu, Apr 27, 2017 at 5:52 PM, Bryan Bende  wrote:
>  Looks like we are down to just a few tickets, and all of them seem to
>  have traction in terms of review and discussion.
> 
>  I'll keep an eye on the tickets and start trying to pull together
>  anything I can to get ready for the RC process. Seems like we are
>  close!
> 
>  On Tue, Apr 25, 2017 at 9:33 AM, Russell Bateman <
> r...@windofkeltia.com>
> >>> wrote:
> > Many thanks to all the industrious folk working on this tool that's
> >>> become
> > indispensable to so many. The community has my enduring gratitude and
> > admiration!
> >
> > Russ
> >
> >
> > On 04/24/2017 07:16 PM, Joe Witt wrote:
> >>
> >> Robert,
> >>
> >> We're about 10 or so JIRAs away at this point and all of them have
> PRs
> >> and progress on them.  We need to close those down and do some
> >> stabilization/testing then a release candidate can happen.  So
> perhaps
> >> next week we'll be at that point and have a release.  This is the
> >> right place to be watching to see it closing in.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Mon, Apr 24, 2017 at 5:40 AM, roboh  wrote:
> >>>
> >>> Hi,
> >>>
> >>> is there any date set for 1.2.0 release yet?
> >>> We are running into an  NIFI-3389
> >>>    issue and
> would
> >>> like
> >>> to
> >>> know when we can expect 1.2.0 release to be available.
> >>>
> >>> Thanks,
> >>> Robert
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context:
> >>> http://apache-nifi-developer-list.39713.n7.nabble.com/
> >>> Closing-in-on-a-NiFi-1-2-0-release-tp14907p15532.html
> >>> Sent from the Apache NiFi Developer List mailing list archive at
> >>> Nabble.com.
> >
> >
> >>>
> >>
> >>
> >>
> >> --
> >> *Scott Aslan = new WebDeveloper(*
> >> *{"location": {"city": "Saint Cloud","state": "FL",
> >>"zip": "34771"},"c

Re: External ZooKeeper

[Juan Sequeiros]

Mark,

I've been using https://zookeeper.apache.org/doc/r3.4.10/ [1] and
subscribed to their user alias [2] similar to Apache NIFI, they have been
responsive when I've asked them questions.
It was easy for me to adapt to using an external zookeeper since
configuration wise its pretty much the same as setting up embedded.
Apache Zookeeper is like an onion in my opinion, with regards to supporting
it in a live environment.

[1] https://zookeeper.apache.org/doc/r3.4.10/
[2] https://zookeeper.apache.org/

On Tue, May 2, 2017 at 1:22 PM Mark Bean  wrote:

> Is there a guide, blog, FAQ or other documentation on setting up a
> dedicated external ZooKeeper (not the embeddded ZooKeeper) for a NiFi
> Cluster?
>
> Thanks,
> Mark
>

Not able to ingest the csv data to RDBS Database

[suman....@cuddle.ai]

Hi ,
I have a csv file and i want to ingest the csv file data into the database.

I have used getfile,RelaceText,PutSQL Processors.

It's failing in replaceText Processor.

File Sample :
City,Count
Mumbai,10
Mumbai,10
Pune,10
Pune,10

ReplaceText :

INSERT INTO CITY(city,count) values ('${city}',${count})

Please help.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Not-able-to-ingest-the-csv-data-to-RDBS-Database-tp15610.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

External ZooKeeper

[Mark Bean]

Is there a guide, blog, FAQ or other documentation on setting up a
dedicated external ZooKeeper (not the embeddded ZooKeeper) for a NiFi
Cluster?

Thanks,
Mark

failed to replicate request GET /NIFI-API/flow/process-groups/root to saxx:9090

[pradeepbill]

hello folks, I see this exception, and can not open the NIFI console.There is
connectivity between the nodes in the cluster on port 9090.Any clues would
help.

2017-05-02 11:50:18,607 WARN [Replicate Request Thread-2]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
/nifi-api/flow/process-groups/root to sabc.abcd.local:9090 due to {}
com.sun.jersey.api.client.ClientHandlerException:
java.net.SocketTimeoutException: Read timed out
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.Client.handle(Client.java:652)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
~[jersey-client-1.19.jar:1.19]
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578)
~[nifi-framework-cluster-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
~[nifi-framework-cluster-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.net.SocketTimeoutException: Read timed out

thanks
Pradeep



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/failed-to-replicate-request-GET-NIFI-API-flow-process-groups-root-to-saxx-9090-tp15612.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Data Load

[Matt Burgess]

Yes that sounds like your best bet, assuming you have the "Maximum
Value Column" present in the table you want to migrate.  Then a flow
might look like:

QueryDatabaseTable -> ConvertAvroToJSON -> ConvertJSONToSQL -> PutSQL

In this flow the target tables would need to be created beforehand.
You might be able to do that with pg_dump or with some fancy SQL that
you could send to PutSQL in a separate (do-ahead) flow [1].  For
multiple tables, you will need one QueryDatabaseTable for each table;
depending on the number of tables and the latency for getting/putting
rows, you may be able to share the downstream processing. If that
creates a bottleneck, you may want a copy of the above flow for each
table.  This is drastically improved in NiFi 1.2.0, as you can use
ListDatabaseTables -> GenerateTableFetch -> RPG -> Input Port ->
ExecuteSQL to perform the migration in parallel across a NiFi cluster.

Regards,
Matt

[1] 
https://serverfault.com/questions/231952/is-there-a-mysql-equivalent-of-show-create-table-in-postgres


On Tue, May 2, 2017 at 11:18 AM, Anil Rai  wrote:
> Thanks Matt for the quick reply. We are using nifi 1.0 release as of now.
> It's a postgres DB on both sides (on prem and on cloud)
> and yes incremental load is what i am looking for.
> so with that, you recommend # 2 option?
>
> On Tue, May 2, 2017 at 11:00 AM, Matt Burgess  wrote:
>
>> Anil,
>>
>> Is this a "one-time" migration, meaning you would take the on-prem
>> tables and put them on the cloud DB just once? Or would this be an
>> incremental operation, where you do the initial move and then take any
>> "new" rows from the source and apply them to the target?  For the
>> latter, there are a couple of options:
>>
>> 1) Rebuild the cloud DB periodically. You can use ExecuteSQL ->
>> [processors] -> PutSQL after perhaps deleting your target
>> DB/tables/etc.  This could be time-consuming and expensive. The
>> processors in-between probably include ConvertAvroToJSON and
>> ConvertJSONToSQL.
>> 2) Use QueryDatabaseTable or (GenerateTableFetch -> ExecuteSQL) to get
>> the source data. For this your table would need a column whose values
>> always increase, that column would comprise the value of the "Maximum
>> Value Column" property in the aforementioned processors' configuration
>> dialogs. You would need one QueryDatabaseTable or GenerateTableFetch
>> for each table in your DB.
>>
>> In addition to these current solutions, as of the upcoming NiFi 1.2.0
>> release, you have the following options:
>> 3) If the source database is MySQL, you can use the CaptureChangeMySQL
>> processor to get binary log events flowing through various processors
>> into PutDatabaseRecord to place them at the source. This pattern is
>> true Change Data Capture (CDC) versus the other two options above.
>> 4) Option #2 will be improved such that GenerateTableFetch will accept
>> incoming flow files, so you can use ListDatabaseTables ->
>> GenerateTableFetch -> ExecuteSQL to handle multiple tables with one
>> flow.
>>
>> If this is a one-time migration, a data flow tool might not be the
>> best choice, you could consider something like Flyway [1] instead.
>>
>> Regards,
>> Matt
>>
>> [1] https://flywaydb.org/documentation/command/migrate
>>
>> On Tue, May 2, 2017 at 10:41 AM, Anil Rai  wrote:
>> > I have a simple use case.
>> >
>> > DB (On Premise) and DB (On Cloud).
>> >
>> > I want to use nifi to extract data from on prem DB (huge volumes) and
>> > insert into the same table structure that is hosted on cloud.
>> >
>> > I could use ExecuteSQL on both sides of the fence (to extract from on
>> prem
>> > and insert onto cloud). What processors are needed in between (if at
>> all)?
>> > As i am not doing any transformations at allit is just extract and
>> load
>> > use case
>>

Re: Data Load

[Anil Rai]

Thanks Matt for the quick reply. We are using nifi 1.0 release as of now.
It's a postgres DB on both sides (on prem and on cloud)
and yes incremental load is what i am looking for.
so with that, you recommend # 2 option?

On Tue, May 2, 2017 at 11:00 AM, Matt Burgess  wrote:

> Anil,
>
> Is this a "one-time" migration, meaning you would take the on-prem
> tables and put them on the cloud DB just once? Or would this be an
> incremental operation, where you do the initial move and then take any
> "new" rows from the source and apply them to the target?  For the
> latter, there are a couple of options:
>
> 1) Rebuild the cloud DB periodically. You can use ExecuteSQL ->
> [processors] -> PutSQL after perhaps deleting your target
> DB/tables/etc.  This could be time-consuming and expensive. The
> processors in-between probably include ConvertAvroToJSON and
> ConvertJSONToSQL.
> 2) Use QueryDatabaseTable or (GenerateTableFetch -> ExecuteSQL) to get
> the source data. For this your table would need a column whose values
> always increase, that column would comprise the value of the "Maximum
> Value Column" property in the aforementioned processors' configuration
> dialogs. You would need one QueryDatabaseTable or GenerateTableFetch
> for each table in your DB.
>
> In addition to these current solutions, as of the upcoming NiFi 1.2.0
> release, you have the following options:
> 3) If the source database is MySQL, you can use the CaptureChangeMySQL
> processor to get binary log events flowing through various processors
> into PutDatabaseRecord to place them at the source. This pattern is
> true Change Data Capture (CDC) versus the other two options above.
> 4) Option #2 will be improved such that GenerateTableFetch will accept
> incoming flow files, so you can use ListDatabaseTables ->
> GenerateTableFetch -> ExecuteSQL to handle multiple tables with one
> flow.
>
> If this is a one-time migration, a data flow tool might not be the
> best choice, you could consider something like Flyway [1] instead.
>
> Regards,
> Matt
>
> [1] https://flywaydb.org/documentation/command/migrate
>
> On Tue, May 2, 2017 at 10:41 AM, Anil Rai  wrote:
> > I have a simple use case.
> >
> > DB (On Premise) and DB (On Cloud).
> >
> > I want to use nifi to extract data from on prem DB (huge volumes) and
> > insert into the same table structure that is hosted on cloud.
> >
> > I could use ExecuteSQL on both sides of the fence (to extract from on
> prem
> > and insert onto cloud). What processors are needed in between (if at
> all)?
> > As i am not doing any transformations at allit is just extract and
> load
> > use case
>

Re: Closing in on a NiFi 1.2.0 release?

[Andrew Lim]

I will be making updates to the Release Notes and Migration Guidance doc 
regarding the TLS 1.2 version support.  Tracked by:

https://issues.apache.org/jira/browse/NIFI-3720


-Drew


> On May 2, 2017, at 11:08 AM, Joe Witt  wrote:
> 
> Those are great updates.  I'd recommend we avoid highlighting the
> versions of UI components though.
> 
> Thanks
> 
> 
> On Tue, May 2, 2017 at 11:03 AM, Scott Aslan  wrote:
>> Hey Bryan,
>> 
>> Please include the following in the release notes:
>> 
>> 
>>   - Core UI
>>  - Circular references have been removed and the code modularized.
>>  - Upgraded Node version to 6.9.3.
>>  - Upgraded npm version to 3.10.10.
>>  - Upgraded jQuery version to 3.1.1.
>>  - Upgraded D3 version to 3.5.17.
>>  - Reduced download size by removing bundled dependencies.
>>   - User Experience Improvements
>>   - Ever wish that it was easier to align components on the canvas? Me
>>  too...and now you can!
>>  - We now provide deep links to any component(s) on the canvas. This
>>  will help make collaborating and sharing more natural.
>>  - Users will enjoy a better understanding of the scope of Controller
>>  Services through an improved experience.
>>  - All actions available on the operate palette are now also available
>>  under the context menu too!
>> 
>> Thanks!
>> 
>> On Tue, May 2, 2017 at 10:17 AM, Bryan Bende  wrote:
>> 
>>> All,
>>> 
>>> Looks like we are closer than we have ever been!
>>> 
>>> Down to three outstanding JIRAs...
>>> 
>>> - NIFI-1833 (Azure Processors) - Active review and discussion
>>> occurring, appears to be close
>>> - NIFI-3260 (Official Docker Image) - Active discussion, looks like
>>> this may be moved from the release to a separate effort
>>> - NIFI-3765 (Status operation for NodeManager) - Active review, looks
>>> like it could be merged relatively soon
>>> 
>>> Once these are finalized we should be able to start the RC process.
>>> 
>>> I've started putting together the release notes on the Wiki (still a
>>> work in progress):
>>> https://cwiki.apache.org/confluence/display/NIFI/
>>> Release+Notes#ReleaseNotes-Version1.2.0
>>> 
>>> If anyone knows of anything that should be highlighted, feel free to
>>> mention it here and I can update the notes.
>>> 
>>> Thanks!
>>> 
>>> -Bryan
>>> 
>>> 
>>> On Thu, Apr 27, 2017 at 5:52 PM, Bryan Bende  wrote:
 Looks like we are down to just a few tickets, and all of them seem to
 have traction in terms of review and discussion.
 
 I'll keep an eye on the tickets and start trying to pull together
 anything I can to get ready for the RC process. Seems like we are
 close!
 
 On Tue, Apr 25, 2017 at 9:33 AM, Russell Bateman 
>>> wrote:
> Many thanks to all the industrious folk working on this tool that's
>>> become
> indispensable to so many. The community has my enduring gratitude and
> admiration!
> 
> Russ
> 
> 
> On 04/24/2017 07:16 PM, Joe Witt wrote:
>> 
>> Robert,
>> 
>> We're about 10 or so JIRAs away at this point and all of them have PRs
>> and progress on them.  We need to close those down and do some
>> stabilization/testing then a release candidate can happen.  So perhaps
>> next week we'll be at that point and have a release.  This is the
>> right place to be watching to see it closing in.
>> 
>> Thanks
>> Joe
>> 
>> On Mon, Apr 24, 2017 at 5:40 AM, roboh  wrote:
>>> 
>>> Hi,
>>> 
>>> is there any date set for 1.2.0 release yet?
>>> We are running into an  NIFI-3389
>>>    issue and would
>>> like
>>> to
>>> know when we can expect 1.2.0 release to be available.
>>> 
>>> Thanks,
>>> Robert
>>> 
>>> 
>>> 
>>> --
>>> View this message in context:
>>> http://apache-nifi-developer-list.39713.n7.nabble.com/
>>> Closing-in-on-a-NiFi-1-2-0-release-tp14907p15532.html
>>> Sent from the Apache NiFi Developer List mailing list archive at
>>> Nabble.com.
> 
> 
>>> 
>> 
>> 
>> 
>> --
>> *Scott Aslan = new WebDeveloper(*
>> *{"location": {"city": "Saint Cloud","state": "FL",
>>"zip": "34771"},"contact": {"email":
>> "scottyas...@gmail.com ","linkedin":
>> "http://www.linkedin.com/in/scottyaslan
>> "}});*

Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

[Andrew Christianson]

Sounds good, thanks.

From: Marc P. 
Sent: Tuesday, May 2, 2017 11:12:02 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

Andy,
  That will be solved with MINIFI-226. You will only need to call
  REGISTER_RESOURCE(classname) or REGISTER_RESOURCE_AS(classname,name) in
your header file and it will automatically be loadable via the name.

  You are correct though, we can alias the namespace ( and we do in
core/Core.h). Feel free to create a processors namespace alias.

On Tue, May 2, 2017 at 10:59 AM, Andrew Christianson <
andrew.christian...@nextcentury.com> wrote:

> OK, I can work with that. Does the style guide allow for cleaning up this
> bit?
>
>   if (name
>   == 
> org::apache::nifi::minifi::processors::GenerateFlowFile::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::GenerateFlowFile>(name,
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::LogAttribute::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::LogAttribute>(name, uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::GetFile::ProcessorName) {
> processor =
> std::make_shared GetFile>(name,
>
>  uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::PutFile::ProcessorName) {
> processor =
> std::make_shared PutFile>(name,
>
>  uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::TailFile::ProcessorName) {
> processor =
> std::make_shared TailFile>(name,
>
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::ListenSyslog::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::ListenSyslog>(name, uuid);
>   } else if (name
> == org::apache::nifi::minifi::processors::ListenHTTP::ProcessorName)
> {
>   processor = std::make_shared<
>   org::apache::nifi::minifi::processors::ListenHTTP>(name, uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::InvokeHTTP::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::InvokeHTTP>(name,
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::ExecuteProcess::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::ExecuteProcess>(name,
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::AppendHostInfo::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::AppendHostInfo>(name,
> uuid);
>   } else {
> logger_->log_error("No Processor defined for %s", name.c_str());
> return nullptr;
>   }
>
> It seems very excessive to continually repeat 
> org::apache::nifi::minifi::processors.
> It looks like the style guide may allow something like this?
>
> namespace processors = org::apache::nifi::minifi::processors;
>
> Is that correct?
>
> -Andy
> 
> From: Kevin Doran 
> Sent: Tuesday, May 2, 2017 10:49:25 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)
>
> Hi Andy,
>
> I’ll let a minify-cpp veteran jump in if I have this wrong, but I believe
> we are following the Google C++ Style Guide [1]. As it is my first time
> following this style, it caused me to do a double take as well :-)
>
> The Google C++ Style Guide does specify entries under public, protected,
> and private sections are indented one space [2]
>
> [1] https://google.github.io/styleguide/cppguide.html
> [2] https://google.github.io/styleguide/cppguide.html#Class_Format
>
> -- Kevin
>
> On 5/2/17, 10:35, "Andrew Christianson"  nextcentury.com> wrote:
>
> On closer inspection, it looks like just the public/private/etc.
> specifiers are single-space indented, with the rest being two-spaces. Is
> this intentional?
> 
> From: Andrew Christianson
> Sent: Tuesday, May 2, 2017 10:33:27 AM
> To: dev@nifi.apache.org
> Subject: MiNiFi CPP indentation confusion (Re: MINIFI-244)
>
> Working on merging my code for MINIFI-244 with the latest refactoring
> where all of the classes have been put into nested namespaces. Looking at
> sibling files, it looks like several of them are indented with one space.
> This surprises me. What's the preferred indentation/code style? Two spaces?
> This would be a first for me if we're actually going for single-space
> indentation.
>
> Regards,
>
> Andy
>
>
>
>

Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

[Marc P.]

Andy,
  That will be solved with MINIFI-226. You will only need to call
  REGISTER_RESOURCE(classname) or REGISTER_RESOURCE_AS(classname,name) in
your header file and it will automatically be loadable via the name.

  You are correct though, we can alias the namespace ( and we do in
core/Core.h). Feel free to create a processors namespace alias.

On Tue, May 2, 2017 at 10:59 AM, Andrew Christianson <
andrew.christian...@nextcentury.com> wrote:

> OK, I can work with that. Does the style guide allow for cleaning up this
> bit?
>
>   if (name
>   == 
> org::apache::nifi::minifi::processors::GenerateFlowFile::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::GenerateFlowFile>(name,
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::LogAttribute::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::LogAttribute>(name, uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::GetFile::ProcessorName) {
> processor =
> std::make_shared GetFile>(name,
>
>  uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::PutFile::ProcessorName) {
> processor =
> std::make_shared PutFile>(name,
>
>  uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::TailFile::ProcessorName) {
> processor =
> std::make_shared TailFile>(name,
>
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::ListenSyslog::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::ListenSyslog>(name, uuid);
>   } else if (name
> == org::apache::nifi::minifi::processors::ListenHTTP::ProcessorName)
> {
>   processor = std::make_shared<
>   org::apache::nifi::minifi::processors::ListenHTTP>(name, uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::InvokeHTTP::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::InvokeHTTP>(name,
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::ExecuteProcess::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::ExecuteProcess>(name,
> uuid);
>   } else if (name
>   == org::apache::nifi::minifi::processors::AppendHostInfo::ProcessorName)
> {
> processor = std::make_shared<
> org::apache::nifi::minifi::processors::AppendHostInfo>(name,
> uuid);
>   } else {
> logger_->log_error("No Processor defined for %s", name.c_str());
> return nullptr;
>   }
>
> It seems very excessive to continually repeat 
> org::apache::nifi::minifi::processors.
> It looks like the style guide may allow something like this?
>
> namespace processors = org::apache::nifi::minifi::processors;
>
> Is that correct?
>
> -Andy
> 
> From: Kevin Doran 
> Sent: Tuesday, May 2, 2017 10:49:25 AM
> To: dev@nifi.apache.org
> Subject: Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)
>
> Hi Andy,
>
> I’ll let a minify-cpp veteran jump in if I have this wrong, but I believe
> we are following the Google C++ Style Guide [1]. As it is my first time
> following this style, it caused me to do a double take as well :-)
>
> The Google C++ Style Guide does specify entries under public, protected,
> and private sections are indented one space [2]
>
> [1] https://google.github.io/styleguide/cppguide.html
> [2] https://google.github.io/styleguide/cppguide.html#Class_Format
>
> -- Kevin
>
> On 5/2/17, 10:35, "Andrew Christianson"  nextcentury.com> wrote:
>
> On closer inspection, it looks like just the public/private/etc.
> specifiers are single-space indented, with the rest being two-spaces. Is
> this intentional?
> 
> From: Andrew Christianson
> Sent: Tuesday, May 2, 2017 10:33:27 AM
> To: dev@nifi.apache.org
> Subject: MiNiFi CPP indentation confusion (Re: MINIFI-244)
>
> Working on merging my code for MINIFI-244 with the latest refactoring
> where all of the classes have been put into nested namespaces. Looking at
> sibling files, it looks like several of them are indented with one space.
> This surprises me. What's the preferred indentation/code style? Two spaces?
> This would be a first for me if we're actually going for single-space
> indentation.
>
> Regards,
>
> Andy
>
>
>
>

Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

[Andrew Christianson]

OK, I can work with that. Does the style guide allow for cleaning up this bit?

  if (name
  == 
org::apache::nifi::minifi::processors::GenerateFlowFile::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::GenerateFlowFile>(name, uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::LogAttribute::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::LogAttribute>(name, uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::GetFile::ProcessorName) {
processor =
std::make_shared(name,
 uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::PutFile::ProcessorName) {
processor =
std::make_shared(name,
 uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::TailFile::ProcessorName) {
processor =
std::make_shared(name,
  uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::ListenSyslog::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::ListenSyslog>(name, uuid);
  } else if (name
== org::apache::nifi::minifi::processors::ListenHTTP::ProcessorName) {
  processor = std::make_shared<
  org::apache::nifi::minifi::processors::ListenHTTP>(name, uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::InvokeHTTP::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::InvokeHTTP>(name, uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::ExecuteProcess::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::ExecuteProcess>(name, uuid);
  } else if (name
  == org::apache::nifi::minifi::processors::AppendHostInfo::ProcessorName) {
processor = std::make_shared<
org::apache::nifi::minifi::processors::AppendHostInfo>(name, uuid);
  } else {
logger_->log_error("No Processor defined for %s", name.c_str());
return nullptr;
  }

It seems very excessive to continually repeat 
org::apache::nifi::minifi::processors. It looks like the style guide may allow 
something like this?

namespace processors = org::apache::nifi::minifi::processors;

Is that correct?

-Andy

From: Kevin Doran 
Sent: Tuesday, May 2, 2017 10:49:25 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

Hi Andy,

I’ll let a minify-cpp veteran jump in if I have this wrong, but I believe we 
are following the Google C++ Style Guide [1]. As it is my first time following 
this style, it caused me to do a double take as well :-)

The Google C++ Style Guide does specify entries under public, protected, and 
private sections are indented one space [2]

[1] https://google.github.io/styleguide/cppguide.html
[2] https://google.github.io/styleguide/cppguide.html#Class_Format

-- Kevin

On 5/2/17, 10:35, "Andrew Christianson"  
wrote:

On closer inspection, it looks like just the public/private/etc. specifiers 
are single-space indented, with the rest being two-spaces. Is this intentional?

From: Andrew Christianson
Sent: Tuesday, May 2, 2017 10:33:27 AM
To: dev@nifi.apache.org
Subject: MiNiFi CPP indentation confusion (Re: MINIFI-244)

Working on merging my code for MINIFI-244 with the latest refactoring where 
all of the classes have been put into nested namespaces. Looking at sibling 
files, it looks like several of them are indented with one space. This 
surprises me. What's the preferred indentation/code style? Two spaces? This 
would be a first for me if we're actually going for single-space indentation.

Regards,

Andy

Re: Closing in on a NiFi 1.2.0 release?

[Joe Witt]

Those are great updates.  I'd recommend we avoid highlighting the
versions of UI components though.

Thanks


On Tue, May 2, 2017 at 11:03 AM, Scott Aslan  wrote:
> Hey Bryan,
>
> Please include the following in the release notes:
>
>
>- Core UI
>   - Circular references have been removed and the code modularized.
>   - Upgraded Node version to 6.9.3.
>   - Upgraded npm version to 3.10.10.
>   - Upgraded jQuery version to 3.1.1.
>   - Upgraded D3 version to 3.5.17.
>   - Reduced download size by removing bundled dependencies.
>- User Experience Improvements
>- Ever wish that it was easier to align components on the canvas? Me
>   too...and now you can!
>   - We now provide deep links to any component(s) on the canvas. This
>   will help make collaborating and sharing more natural.
>   - Users will enjoy a better understanding of the scope of Controller
>   Services through an improved experience.
>   - All actions available on the operate palette are now also available
>   under the context menu too!
>
> Thanks!
>
> On Tue, May 2, 2017 at 10:17 AM, Bryan Bende  wrote:
>
>> All,
>>
>> Looks like we are closer than we have ever been!
>>
>> Down to three outstanding JIRAs...
>>
>> - NIFI-1833 (Azure Processors) - Active review and discussion
>> occurring, appears to be close
>> - NIFI-3260 (Official Docker Image) - Active discussion, looks like
>> this may be moved from the release to a separate effort
>> - NIFI-3765 (Status operation for NodeManager) - Active review, looks
>> like it could be merged relatively soon
>>
>> Once these are finalized we should be able to start the RC process.
>>
>> I've started putting together the release notes on the Wiki (still a
>> work in progress):
>> https://cwiki.apache.org/confluence/display/NIFI/
>> Release+Notes#ReleaseNotes-Version1.2.0
>>
>> If anyone knows of anything that should be highlighted, feel free to
>> mention it here and I can update the notes.
>>
>> Thanks!
>>
>> -Bryan
>>
>>
>> On Thu, Apr 27, 2017 at 5:52 PM, Bryan Bende  wrote:
>> > Looks like we are down to just a few tickets, and all of them seem to
>> > have traction in terms of review and discussion.
>> >
>> > I'll keep an eye on the tickets and start trying to pull together
>> > anything I can to get ready for the RC process. Seems like we are
>> > close!
>> >
>> > On Tue, Apr 25, 2017 at 9:33 AM, Russell Bateman 
>> wrote:
>> >> Many thanks to all the industrious folk working on this tool that's
>> become
>> >> indispensable to so many. The community has my enduring gratitude and
>> >> admiration!
>> >>
>> >> Russ
>> >>
>> >>
>> >> On 04/24/2017 07:16 PM, Joe Witt wrote:
>> >>>
>> >>> Robert,
>> >>>
>> >>> We're about 10 or so JIRAs away at this point and all of them have PRs
>> >>> and progress on them.  We need to close those down and do some
>> >>> stabilization/testing then a release candidate can happen.  So perhaps
>> >>> next week we'll be at that point and have a release.  This is the
>> >>> right place to be watching to see it closing in.
>> >>>
>> >>> Thanks
>> >>> Joe
>> >>>
>> >>> On Mon, Apr 24, 2017 at 5:40 AM, roboh  wrote:
>> 
>>  Hi,
>> 
>>  is there any date set for 1.2.0 release yet?
>>  We are running into an  NIFI-3389
>>     issue and would
>> like
>>  to
>>  know when we can expect 1.2.0 release to be available.
>> 
>>  Thanks,
>>  Robert
>> 
>> 
>> 
>>  --
>>  View this message in context:
>>  http://apache-nifi-developer-list.39713.n7.nabble.com/
>> Closing-in-on-a-NiFi-1-2-0-release-tp14907p15532.html
>>  Sent from the Apache NiFi Developer List mailing list archive at
>>  Nabble.com.
>> >>
>> >>
>>
>
>
>
> --
> *Scott Aslan = new WebDeveloper(*
> *{"location": {"city": "Saint Cloud","state": "FL",
> "zip": "34771"},"contact": {"email":
> "scottyas...@gmail.com ","linkedin":
> "http://www.linkedin.com/in/scottyaslan
> "}});*

Re: Closing in on a NiFi 1.2.0 release?

[Scott Aslan]

Hey Bryan,

Please include the following in the release notes:


   - Core UI
  - Circular references have been removed and the code modularized.
  - Upgraded Node version to 6.9.3.
  - Upgraded npm version to 3.10.10.
  - Upgraded jQuery version to 3.1.1.
  - Upgraded D3 version to 3.5.17.
  - Reduced download size by removing bundled dependencies.
   - User Experience Improvements
   - Ever wish that it was easier to align components on the canvas? Me
  too...and now you can!
  - We now provide deep links to any component(s) on the canvas. This
  will help make collaborating and sharing more natural.
  - Users will enjoy a better understanding of the scope of Controller
  Services through an improved experience.
  - All actions available on the operate palette are now also available
  under the context menu too!

Thanks!

On Tue, May 2, 2017 at 10:17 AM, Bryan Bende  wrote:

> All,
>
> Looks like we are closer than we have ever been!
>
> Down to three outstanding JIRAs...
>
> - NIFI-1833 (Azure Processors) - Active review and discussion
> occurring, appears to be close
> - NIFI-3260 (Official Docker Image) - Active discussion, looks like
> this may be moved from the release to a separate effort
> - NIFI-3765 (Status operation for NodeManager) - Active review, looks
> like it could be merged relatively soon
>
> Once these are finalized we should be able to start the RC process.
>
> I've started putting together the release notes on the Wiki (still a
> work in progress):
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.2.0
>
> If anyone knows of anything that should be highlighted, feel free to
> mention it here and I can update the notes.
>
> Thanks!
>
> -Bryan
>
>
> On Thu, Apr 27, 2017 at 5:52 PM, Bryan Bende  wrote:
> > Looks like we are down to just a few tickets, and all of them seem to
> > have traction in terms of review and discussion.
> >
> > I'll keep an eye on the tickets and start trying to pull together
> > anything I can to get ready for the RC process. Seems like we are
> > close!
> >
> > On Tue, Apr 25, 2017 at 9:33 AM, Russell Bateman 
> wrote:
> >> Many thanks to all the industrious folk working on this tool that's
> become
> >> indispensable to so many. The community has my enduring gratitude and
> >> admiration!
> >>
> >> Russ
> >>
> >>
> >> On 04/24/2017 07:16 PM, Joe Witt wrote:
> >>>
> >>> Robert,
> >>>
> >>> We're about 10 or so JIRAs away at this point and all of them have PRs
> >>> and progress on them.  We need to close those down and do some
> >>> stabilization/testing then a release candidate can happen.  So perhaps
> >>> next week we'll be at that point and have a release.  This is the
> >>> right place to be watching to see it closing in.
> >>>
> >>> Thanks
> >>> Joe
> >>>
> >>> On Mon, Apr 24, 2017 at 5:40 AM, roboh  wrote:
> 
>  Hi,
> 
>  is there any date set for 1.2.0 release yet?
>  We are running into an  NIFI-3389
>     issue and would
> like
>  to
>  know when we can expect 1.2.0 release to be available.
> 
>  Thanks,
>  Robert
> 
> 
> 
>  --
>  View this message in context:
>  http://apache-nifi-developer-list.39713.n7.nabble.com/
> Closing-in-on-a-NiFi-1-2-0-release-tp14907p15532.html
>  Sent from the Apache NiFi Developer List mailing list archive at
>  Nabble.com.
> >>
> >>
>



-- 
*Scott Aslan = new WebDeveloper(*
*{"location": {"city": "Saint Cloud","state": "FL",
"zip": "34771"},"contact": {"email":
"scottyas...@gmail.com ","linkedin":
"http://www.linkedin.com/in/scottyaslan
"}});*

Re: Data Load

[Matt Burgess]

Anil,

Is this a "one-time" migration, meaning you would take the on-prem
tables and put them on the cloud DB just once? Or would this be an
incremental operation, where you do the initial move and then take any
"new" rows from the source and apply them to the target?  For the
latter, there are a couple of options:

1) Rebuild the cloud DB periodically. You can use ExecuteSQL ->
[processors] -> PutSQL after perhaps deleting your target
DB/tables/etc.  This could be time-consuming and expensive. The
processors in-between probably include ConvertAvroToJSON and
ConvertJSONToSQL.
2) Use QueryDatabaseTable or (GenerateTableFetch -> ExecuteSQL) to get
the source data. For this your table would need a column whose values
always increase, that column would comprise the value of the "Maximum
Value Column" property in the aforementioned processors' configuration
dialogs. You would need one QueryDatabaseTable or GenerateTableFetch
for each table in your DB.

In addition to these current solutions, as of the upcoming NiFi 1.2.0
release, you have the following options:
3) If the source database is MySQL, you can use the CaptureChangeMySQL
processor to get binary log events flowing through various processors
into PutDatabaseRecord to place them at the source. This pattern is
true Change Data Capture (CDC) versus the other two options above.
4) Option #2 will be improved such that GenerateTableFetch will accept
incoming flow files, so you can use ListDatabaseTables ->
GenerateTableFetch -> ExecuteSQL to handle multiple tables with one
flow.

If this is a one-time migration, a data flow tool might not be the
best choice, you could consider something like Flyway [1] instead.

Regards,
Matt

[1] https://flywaydb.org/documentation/command/migrate

On Tue, May 2, 2017 at 10:41 AM, Anil Rai  wrote:
> I have a simple use case.
>
> DB (On Premise) and DB (On Cloud).
>
> I want to use nifi to extract data from on prem DB (huge volumes) and
> insert into the same table structure that is hosted on cloud.
>
> I could use ExecuteSQL on both sides of the fence (to extract from on prem
> and insert onto cloud). What processors are needed in between (if at all)?
> As i am not doing any transformations at allit is just extract and load
> use case

Re: Closing in on a NiFi 1.2.0 release?

[Bryan Bende]

Quick update...

I was looking through PRs and noticed that NIFI-3594 (Encrypted Prov
Repo) had already received a +1 and just needed an update to resolve
conflicts with master.  I spoke to Andy offline and he is planning to
resolve the conflicts shortly and we can include this in 1.2.0.

Also, NIFI-3726 (CompareFuzzyHash processor) appears to be very close
as well and has already had some review and updates made. Andy plans
to do a final review and hopefully merge this as well.

Thanks,

Bryan




On Tue, May 2, 2017 at 10:17 AM, Bryan Bende  wrote:
> All,
>
> Looks like we are closer than we have ever been!
>
> Down to three outstanding JIRAs...
>
> - NIFI-1833 (Azure Processors) - Active review and discussion
> occurring, appears to be close
> - NIFI-3260 (Official Docker Image) - Active discussion, looks like
> this may be moved from the release to a separate effort
> - NIFI-3765 (Status operation for NodeManager) - Active review, looks
> like it could be merged relatively soon
>
> Once these are finalized we should be able to start the RC process.
>
> I've started putting together the release notes on the Wiki (still a
> work in progress):
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.2.0
>
> If anyone knows of anything that should be highlighted, feel free to
> mention it here and I can update the notes.
>
> Thanks!
>
> -Bryan
>
>
> On Thu, Apr 27, 2017 at 5:52 PM, Bryan Bende  wrote:
>> Looks like we are down to just a few tickets, and all of them seem to
>> have traction in terms of review and discussion.
>>
>> I'll keep an eye on the tickets and start trying to pull together
>> anything I can to get ready for the RC process. Seems like we are
>> close!
>>
>> On Tue, Apr 25, 2017 at 9:33 AM, Russell Bateman  
>> wrote:
>>> Many thanks to all the industrious folk working on this tool that's become
>>> indispensable to so many. The community has my enduring gratitude and
>>> admiration!
>>>
>>> Russ
>>>
>>>
>>> On 04/24/2017 07:16 PM, Joe Witt wrote:

 Robert,

 We're about 10 or so JIRAs away at this point and all of them have PRs
 and progress on them.  We need to close those down and do some
 stabilization/testing then a release candidate can happen.  So perhaps
 next week we'll be at that point and have a release.  This is the
 right place to be watching to see it closing in.

 Thanks
 Joe

 On Mon, Apr 24, 2017 at 5:40 AM, roboh  wrote:
>
> Hi,
>
> is there any date set for 1.2.0 release yet?
> We are running into an  NIFI-3389
>    issue and would like
> to
> know when we can expect 1.2.0 release to be available.
>
> Thanks,
> Robert
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Closing-in-on-a-NiFi-1-2-0-release-tp14907p15532.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>>>
>>>

Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

[Kevin Doran]

Hi Andy,

I’ll let a minify-cpp veteran jump in if I have this wrong, but I believe we 
are following the Google C++ Style Guide [1]. As it is my first time following 
this style, it caused me to do a double take as well :-)

The Google C++ Style Guide does specify entries under public, protected, and 
private sections are indented one space [2]

[1] https://google.github.io/styleguide/cppguide.html
[2] https://google.github.io/styleguide/cppguide.html#Class_Format 

-- Kevin

On 5/2/17, 10:35, "Andrew Christianson"  
wrote:

On closer inspection, it looks like just the public/private/etc. specifiers 
are single-space indented, with the rest being two-spaces. Is this intentional?

From: Andrew Christianson
Sent: Tuesday, May 2, 2017 10:33:27 AM
To: dev@nifi.apache.org
Subject: MiNiFi CPP indentation confusion (Re: MINIFI-244)

Working on merging my code for MINIFI-244 with the latest refactoring where 
all of the classes have been put into nested namespaces. Looking at sibling 
files, it looks like several of them are indented with one space. This 
surprises me. What's the preferred indentation/code style? Two spaces? This 
would be a first for me if we're actually going for single-space indentation.

Regards,

Andy

Data Load

[Anil Rai]

I have a simple use case.

DB (On Premise) and DB (On Cloud).

I want to use nifi to extract data from on prem DB (huge volumes) and
insert into the same table structure that is hosted on cloud.

I could use ExecuteSQL on both sides of the fence (to extract from on prem
and insert onto cloud). What processors are needed in between (if at all)?
As i am not doing any transformations at allit is just extract and load
use case

Re: MiNiFi CPP indentation confusion (Re: MINIFI-244)

[Andrew Christianson]

On closer inspection, it looks like just the public/private/etc. specifiers are 
single-space indented, with the rest being two-spaces. Is this intentional?

From: Andrew Christianson
Sent: Tuesday, May 2, 2017 10:33:27 AM
To: dev@nifi.apache.org
Subject: MiNiFi CPP indentation confusion (Re: MINIFI-244)

Working on merging my code for MINIFI-244 with the latest refactoring where all 
of the classes have been put into nested namespaces. Looking at sibling files, 
it looks like several of them are indented with one space. This surprises me. 
What's the preferred indentation/code style? Two spaces? This would be a first 
for me if we're actually going for single-space indentation.

Regards,

Andy

MiNiFi CPP indentation confusion (Re: MINIFI-244)

[Andrew Christianson]

Working on merging my code for MINIFI-244 with the latest refactoring where all 
of the classes have been put into nested namespaces. Looking at sibling files, 
it looks like several of them are indented with one space. This surprises me. 
What's the preferred indentation/code style? Two spaces? This would be a first 
for me if we're actually going for single-space indentation.

Regards,

Andy

Re: [CALL FOR ACTION] Peer reviews & testing of new code

[Joseph Niemiec]

Thanks for pointing out that *contributions *of Reviewing code are still
valid and that its not a *committer *only job!

On Mon, May 1, 2017 at 7:17 PM, Andre  wrote:

> Dev,
>
> Those following the git commit log probably noticed that there has been a
> lot of action by some of the developers behind some significant
> improvements in 1.2.0.
>
> As these developers get busy I would like to remind the rest of the list
> about importance of our contribution around code review and *testing* of
> [proposed] changes to the NiFi code base.
>
> So long history short. No matter if you have committer privilege to the
> NiFi git, you can still contribute to this great community.
>
> You can still contribute by providing extra testing of the introduced code
> (I am myself running SNAPSHOT in prod-like conditions with the new
> provenance WAL and it is so fast that I reckon you should be doing the
> same...).
>
>
> You can contribute by helping with the peer review process. [1]
>
>
> So if you have some cycles available and is keen be more active in our
> vibrant community, feel free to get involved!
>
>
>
> Cheers!
>
>
>
> [1] I admit that some of these PRs is daunting to the non-initiated (myself
> included), but trust me when I say that some are perfectly within reach of
> those that are familiar with NiFi and have also dealt with things like:
> HDFS, Azure Cloud, Websockets, etc etc etc. (To be honest, sometimes the
> contribution may be as simple as spell checking!!)
>



-- 
Joseph

Re: Closing in on a NiFi 1.2.0 release?

[Bryan Bende]

All,

Looks like we are closer than we have ever been!

Down to three outstanding JIRAs...

- NIFI-1833 (Azure Processors) - Active review and discussion
occurring, appears to be close
- NIFI-3260 (Official Docker Image) - Active discussion, looks like
this may be moved from the release to a separate effort
- NIFI-3765 (Status operation for NodeManager) - Active review, looks
like it could be merged relatively soon

Once these are finalized we should be able to start the RC process.

I've started putting together the release notes on the Wiki (still a
work in progress):
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.2.0

If anyone knows of anything that should be highlighted, feel free to
mention it here and I can update the notes.

Thanks!

-Bryan


On Thu, Apr 27, 2017 at 5:52 PM, Bryan Bende  wrote:
> Looks like we are down to just a few tickets, and all of them seem to
> have traction in terms of review and discussion.
>
> I'll keep an eye on the tickets and start trying to pull together
> anything I can to get ready for the RC process. Seems like we are
> close!
>
> On Tue, Apr 25, 2017 at 9:33 AM, Russell Bateman  
> wrote:
>> Many thanks to all the industrious folk working on this tool that's become
>> indispensable to so many. The community has my enduring gratitude and
>> admiration!
>>
>> Russ
>>
>>
>> On 04/24/2017 07:16 PM, Joe Witt wrote:
>>>
>>> Robert,
>>>
>>> We're about 10 or so JIRAs away at this point and all of them have PRs
>>> and progress on them.  We need to close those down and do some
>>> stabilization/testing then a release candidate can happen.  So perhaps
>>> next week we'll be at that point and have a release.  This is the
>>> right place to be watching to see it closing in.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Mon, Apr 24, 2017 at 5:40 AM, roboh  wrote:

 Hi,

 is there any date set for 1.2.0 release yet?
 We are running into an  NIFI-3389
    issue and would like
 to
 know when we can expect 1.2.0 release to be available.

 Thanks,
 Robert



 --
 View this message in context:
 http://apache-nifi-developer-list.39713.n7.nabble.com/Closing-in-on-a-NiFi-1-2-0-release-tp14907p15532.html
 Sent from the Apache NiFi Developer List mailing list archive at
 Nabble.com.
>>
>>