Re: Nifi Database repository backup & restore

[Koji Kawamura]

Hi Ganesh,

What did you mean by following statement? Would you elaborate what is
expected and how it behaved actually?
> Nifi is not processing flow from the point where it got stopped or crashed .

Some processor needs "state" get restored in addition to FlowFiles.
States are stored in Zookeeper or in local file system. Does your test
recover these files, too?

Thanks,
Koji

On Thu, Oct 10, 2019 at 3:10 PM Ganesh, B (Nokia - IN/Bangalore)
 wrote:
>
> Hi ,
>
> we are trying to test a disaster recovery with the backup/restore feature.
> We could see that the flowfile and database volume are restored but Nifi is 
> not processing flow from the point where it got stopped or crashed .
>
> Any one have tested the above scenario ?
>
> Thanks & Regards,
> Ganesh.B

Re: Jira contributor access

[Koji Kawamura]

Hi Seokwon,

I've added you contributor role. Looking forward to see your contribution!

Thanks,
Koji

On Thu, Oct 10, 2019 at 7:51 AM Seokwon Yang  wrote:
>
> Hello,
>
> I would like to contribute to the nifi codebase. Please add me (Jira username 
> : sjyang18) as a contributor.
>
> Thanks
>
> Seokwon
>

Re: Can CaptureChangeMySQL be scheduled to all nodes instead of primary node?

[Koji Kawamura]

Hi Lei,

I don't know any NiFi built-in feature to achieve that.
To distribute CaptureChangeMySQL load among nodes, I'd deploy separate
standalone NiFi (or even MiNiFi Java) in addition to the main NiFi
cluster for the main data flow.

For example, if there are 5 databases and 3 NiFi nodes, deploy a 3
node NiFi cluster with an InputPort.
And also, run standalone NiFi/MiNiFi processes on each node, too,
node-a (datasource 1 and 2), node-b (datasource 3 and 4), node-c
(datasource 5) then use RemoteProcessGroup to send captured data to
the main NiFi cluster.

This approach may be harder to maintain, but feasible.

Thanks,
Koji

On Wed, Oct 9, 2019 at 3:06 PM wangl...@geekplus.com.cn
 wrote:
>
> I am using CaptureChangeMySQL to replicate the database.
> There are many data sources and so there're many  CaptureChangeMySQL 
> processors.
> The CaptureChangeMySQL throws same slave id error  if scheduled on all nodes. 
> So it can only be scheduled on primary node. This causes  very heavy load on 
> the primary node.
>
> Is there any method than i can  distribute the CaptureChangeMySQL processors 
> to all nodes instead of only to primary node?
>
> Thanks,
> Lei
>
> 
> wangl...@geekplus.com.cn

Re: PULL ProvenanceEvent

[Adam Taft]

Nissim,

Just to be clear, you are trying to distinguish between processors which
are actively "pulling" data (GetXYZ) vs. processors which just "listen" for
data (ListenXYZ)?  Is that your basic vision?

GetFile => PULL
GetHTTP => PULL
ListenHTTP => RECEIVE
ListenTCP => RECEIVE

Could you clarify what advantages this would have in terms of data
provenance?  What would you use this new event type for specifically?  What
are you missing now? Do you have a use case that needs this, or are you
just generally trying to round out the provenance event types for sake of
completeness?  I honestly don't know a use case where you care whether you
polled for the data or listened for it.  The provenance model today just
cares that you received the data, not so much how you received it.

You're right that this proposal will affect many processors and the
internal visualization tools, etc.  However even more important to realize,
this change would affect many other downstream consumers of provenance data
which aren't necessarily in the stock NiFi distribution.  For example, any
third-party/custom ReportingTask that handles provenance data would need to
be updated with this change.  There's probably need for a strong vision to
help demonstrate the value for this vs. the cost of the cascading effects
related to this change.

Thanks,
Adam


On Thu, Oct 10, 2019 at 4:02 PM Nissim Shiman 
wrote:

> Hello Team,
>
> The ProvenanceEventType class does a good job capturing possible events,
> but the PULL event doesn't seem to fall nicely into any of the existing
> types.
>
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=blob;f=nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java
> RECEIVE is the closest, but RECEIVE is passive and doesn't capture the
> active action of a PULL
>
> And... maybe it would fall into FETCH, but FETCH is more focused on
> contents of an existing flow file being overwritten.
>
> What does the community think about a new PULL event type,
> or
>  using FETCH for PULL, and having what FETCH does now be a new event such
> as REUSE
>
> NOTE: a new PULL event would have a cascading effect of many processors
> that currently are emitting RECEIVE's being modified to be PULL
> (i.e. So GetFile would no longer be a RECEIVE, but rather a PULL), but
> would more accurately capture the event.
>
> Thanks,
> Nissim Shiman
>
>

Re: Jira contributor access

[Bahlul Haider]

Hi Andy,

Thank you very much for quickly adding me as a contributor to the project.
I am excited to join nifi community and hope to make some contribution to
the codebase.

To get familiar with the pull request process, I assigned an easy "newbie"
issue [NIFI-6755] "Chron Trigger Tutorial" link broken in NiFi User Guide
 to myself and created
[PR-3802] NIFI-6755
"Chron Trigger Tutorial" link broken in NiFi User Guide
. It would be great if someone
from the community can take a look at the PR to help me understand the
process.

Thanks,
Bahlul

On Wed, Oct 9, 2019 at 2:50 PM Andy LoPresto  wrote:

> Welcome. I have added you to the contributors role, Bahlul.
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Oct 9, 2019, at 2:31 PM, Bahlul Haider 
> wrote:
> >
> > Hello,
> >
> > I will like to contribute to the nifi codebase. Please add me (Jira
> > username: bahlulh) as a contributor.
> >
> > Thanks,
> > Bahlul
>
>

-- 
Thanks,
Bahlul

PULL ProvenanceEvent

[Nissim Shiman]

Hello Team,

The ProvenanceEventType class does a good job capturing possible events, but 
the PULL event doesn't seem to fall nicely into any of the existing types.
https://gitbox.apache.org/repos/asf?p=nifi.git;a=blob;f=nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java
RECEIVE is the closest, but RECEIVE is passive and doesn't capture the active 
action of a PULL

And... maybe it would fall into FETCH, but FETCH is more focused on contents of 
an existing flow file being overwritten.

What does the community think about a new PULL event type, 
or
 using FETCH for PULL, and having what FETCH does now be a new event such as 
REUSE

NOTE: a new PULL event would have a cascading effect of many processors that 
currently are emitting RECEIVE's being modified to be PULL
(i.e. So GetFile would no longer be a RECEIVE, but rather a PULL), but would 
more accurately capture the event.

Thanks,
Nissim Shiman

Re: Maven Build Error - nifi-properties-loader sub-project test failures

[Adam Taft]

Yeah, I've been suspicious that maybe Maven 3.3.9 is too old to build
NiFi.  We say[1] that Maven 3.1.0+ is required, but that's a really old
version.  It's possible or even likely that some Maven plugins used in the
NiFi build are expecting services from newer Maven versions.

I don't have a suggestion here that doesn't involve someone's time. It
would be ideal we could narrow down and update the required minimum Java &
Maven configuration, so as to update the Quickstart guide that Joe
originally linked[1].  Is this JIRA ticket worthy? I'll file one if so.

Adam

[1] https://nifi.apache.org/quickstart.html


On Thu, Oct 10, 2019 at 7:57 AM Aram Openden  wrote:

> Adam,
>
> Thanks so much for your excellent suggestion.
>
> Probably should have checked my versions 1st (honestly didn't even think of
> it yesterday).
>
> Upgrading my Java-JDK install version to *1.8.0_222* and Maven to *3.6.2*
> did indeed *fix my build issue*!
>
> Aram S. Openden
> aram.open...@gmail.com
>
>
>
> On Thu, Oct 10, 2019 at 1:10 AM Adam Taft  wrote:
>
> > Aram,
> >
> > Just to rule out the obvious ...  Can you update your Maven and Java
> > versions, which would include:
> > - Maven 3.6.2
> > - Java 1.8.0_222
> >
> > Also, are you including a MAVEN_OPTS environment to increase your JVM
> > memory in Maven?
> >
> > $> export MAVEN_OPTS="-Xms1g -Xmx3g"
> >
> > Thanks,
> > Adam
> >
> > On Wed, Oct 9, 2019 at 1:31 PM Aram Openden 
> > wrote:
> >
> > > Hoping someone on this dev mailing list can help with the following
> maven
> > > build failures issue. I am hoping to contribute a new suggested custom
> S3
> > > Processor that I have been working on.
> > >
> > > But, I need to be able to get the build to work locally before I even
> > start
> > > adding in my changes.
> > >
> > > I am trying to run the main NiFi project build with maven locally on
> the
> > > master branch *without having made any local code changes*, with the
> > latest
> > > updates (master is at commit 9a496fe9d NIFI-6751: - Fixing the
> identifier
> > > on the user table(in other words this is the latest code on
> master):
> > >
> > > $ git branch -v
> > > * master 9a496fe9d NIFI-6751: - Fixing the identifier on the user
> table.
> > In
> > > a previous task, this was changed to utilize the URI but that does not
> > work
> > > with other code interacting with this table.
> > >
> > > $ git status
> > > On branch master
> > > Your branch is up-to-date with 'origin/master'.
> > > nothing to commit, working tree clean
> > >
> > >
> > > My local maven env is as follows (running on Mac OS Mojave):
> > >
> > > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > > 2015-11-10T11:41:47-05:00)
> > > Maven home: /usr/local/Cellar/maven/3.3.9/libexec
> > > Java version: 1.8.0_111, vendor: Oracle Corporation
> > > Java home:
> > > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre
> > > Default locale: en_US, platform encoding: UTF-8
> > > OS name: "mac os x", version: "10.14.6", arch: "x86_64", family: "mac"
> > >
> > >
> > > Looking for any help you can provide on what I should be doing to get
> the
> > > maven build to pass locally.
> > > I am getting test failures that look like this:
> > >
> > > INFO]
> > >
> 
> > > [INFO] Building nifi-properties-loader 1.10.0-SNAPSHOT
> > > [INFO]
> > >
> 
> > > [INFO]
> > > [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @
> > > nifi-properties-loader ---
> > > [INFO] Deleting
> > >
> > >
> >
> /Users/aramo.penden/workspaces/WFfH/data-gov-beta/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/target
> > > [INFO]
> > > [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven-version)
> @
> > > nifi-properties-loader ---
> > > [INFO]
> > > [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven) @
> > > nifi-properties-loader ---
> > > [INFO]
> > > [INFO] --- buildnumber-maven-plugin:1.4:create (default) @
> > > nifi-properties-loader ---
> > > [INFO]
> > > [INFO] --- maven-remote-resources-plugin:1.5:process
> > > (process-resource-bundles) @ nifi-properties-loader ---
> > > [INFO]
> > > [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @
> > > nifi-properties-loader ---
> > > [INFO] Using 'UTF-8' encoding to copy filtered resources.
> > > [INFO] skip non existing resourceDirectory
> > >
> > >
> >
> /Users/aramo.penden/workspaces/WFfH/data-gov-beta/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/resources
> > > [INFO] Copying 3 resources
> > > [INFO]
> > > [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @
> > > nifi-properties-loader ---
> > > [INFO] Changes detected - recompiling the module!
> > > [INFO] Compiling 9 source files to
> > >
> > >
> >
> 

Nifi Database repository backup & restore

[Ganesh, B (Nokia - IN/Bangalore)]

Hi ,

we are trying to test a disaster recovery with the backup/restore feature.
We could see that the flowfile and database volume are restored but Nifi is not 
processing flow from the point where it got stopped or crashed .

Any one have tested the above scenario ?

Thanks & Regards,
Ganesh.B