Re: Developing NiFi custom processors in NetBeans

[Otto Fowler]

8 Months ago, the nifi parent pom was change to have https: for all apache
repos:
https://github.com/apache/nifi/commit/e6c843f465da98fd7dc640e53214e4aeefa61b7d
If you are getting this error, I wonder if you have out of date versions of
the archetype, or that you are specifying an old nifi version or something?


On February 5, 2020 at 23:04:42, Otto Fowler (ottobackwa...@gmail.com)
wrote:

screen

Re: Developing NiFi custom processors in NetBeans

[Otto Fowler]

You should log a jira on this and set it to a high priority.




On February 5, 2020 at 10:45:42, Valentina Ivanova (valentina.ivan...@ri.se)
wrote:

Hi everyone!

We faced another issue when trying to create custom processors with
NetBeans 8.2.
I try to create a new project in the following way File-> New Project ->
Maven -> Project from Maven Archetype and then selecting
nifi-processor-bundle-archetype from the available archetypes. Then on the
next screen filling in the requested fields. Finally when pressing finish
to create it I get the following (see below).

After some investigation I figured out that file can't be downloaded:
http://repo.maven.apache.org/maven2/org/apache/nifi/nifi/1.11.0/nifi-1.11.0.pom
as the protocol needs to be set to HTTPS. So I created a custom repository
(and built the index) which uses the HTTPS protocol and try to create the
project using it. However, it seems the HTTP version is still used, so
creating the project fails again.

When I browse the Maven repositories in the Services tab and try to
download certain org.apache.nifi poms I get error messages for some
dependencies are broken which is not clear how to fix.

So I wonder if anyone has had the same issues and how to resolve them.
(I tried to use maven from command line but other issues appear amd I would
prefer the UI as I am a maven beginner.)

Many thanks & all the best

Valentina



cd C:\Users\valentinaiv\Desktop\NetBeansProjects\ReadProtoBuf;
"JAVA_HOME=C:\\Program Files\\AdoptOpenJDK\\jdk-8.0.212.03-hotspot" cmd /c
"\"\"C:\\Program Files\\NetBeans 8.2\\java\\maven\\bin\\mvn.bat\"
-DarchetypeGroupId=org.apache.nifi
-DarchetypeArtifactId=nifi-processor-bundle-archetype
-DarchetypeVersion=1.11.0
-DarchetypeRepository=https://repo.maven.apache.org/maven2/
-DgroupId=rise.koda

-DartifactId=mavenproject2 -Dversion=1.0-SNAPSHOT
-Dpackage=rise.koda.mavenproject2
-Dbasedir=C:\\Users\\valentinaiv\\Desktop\\NetBeansProjects\\ReadProtoBuf
-DartifactBaseName=read -DnifiVersion=1.11.0 -Darchetype.interactive=false
-Dmaven.ext.class.path=\"C:\\Program Files\\NetBeans
8.2\\java\\maven-nblib\\netbeans-eventspy.jar\" --batch-mode
archetype:generate\""
Scanning for projects...


Building Maven Stub Project (No POM) 1


>>> maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom >>>

<<< maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom <<<

--- maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom ---
Generating project in Batch mode
No archetype found in remote catalog. Defaulting to internal catalog
Archetype not found in any catalog. Falling back to central repository.
Add a repository with id 'archetype' in your settings.xml if archetype's
repository is elsewhere.
Downloading:
http://repo.maven.apache.org/maven2/org/apache/nifi/nifi/1.11.0/nifi-1.11.0.pom

BUILD FAILURE

Total time: 1.991s
Finished at: Wed Feb 05 16:37:23 CET 2020
Final Memory: 14M/243M

Failed to execute goal
org.apache.maven.plugins:maven-archetype-plugin:3.1.2:generate
(default-cli) on project standalone-pom: The desired archetype does not
exist (org.apache.nifi:nifi-processor-bundle-archetype:1.11.0) -> [Help 1]

To see the full stack trace of the errors, re-run Maven with the -e switch.
Re-run Maven using the -X switch to enable full debug logging.

For more information about the errors and possible solutions, please read
the following articles:
[Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Re: COnsumekakarecord error messages

[Mike Thomsen]

I think what Pierre might be getting at is that someone or something else
besides NiFi might have dropped invalid data into your Kafka queue. If NiFi
is writing to Kafka and consuming from it too, odds are low (but not
impossible) that it's NiFi. I'd consider setting up a little script to read
the impacted topic and see if anything looks off. Could be someone did
something like connect with the console producer and try to put a message
in there to test something.

On Wed, Feb 5, 2020 at 5:09 PM KhajaAsmath Mohammed 
wrote:

> I am using json record reader and json record set writer in consume kafka.
> Do I need to change these record readers just to confirm about the corrupt
> data? Can you help me what readers should I use in that case ?
>
> Sent from my iPhone
>
> On Feb 5, 2020, at 3:47 PM, Pierre Villard 
> wrote:
>
> 
> Hi,
>
> It seems like you have a message that cannot be parsed correctly by your
> record processor. It is probably not matching your schema or containing
> some corrupted data? You could use a the non record ConsumeKafka processor
> to get the message without doing any parsing and see what the message looks
> like.
>
> HTH
> Pierre
>
> Le mer. 5 févr. 2020 à 16:27, KhajaAsmath Mohammed <
> mdkhajaasm...@gmail.com> a écrit :
>
>> Hi,
>>
>> I was able to consume messages from consumekafkarecord processor.
>> Suddenly I started getting below error. any solutions for this please?
>>
>> 
>>
>>
>> Thanks,
>> Asmath
>>
>

Re: Execute Script from Configuration in a Generic Flow

[Mike Thomsen]

What you probably should do for security sake is use flowfile attributes
and/or dynamic properties on ExecuteScript to dynamic identify a script on
the file system to execute and what parameters to give it for configuring
it.

On Wed, Feb 5, 2020 at 6:45 PM Mike Thomsen  wrote:

> > What are all possible ways to achieve this functionality?
>
> An eval function. Don't know the right way to do it with Nashorn, but in
> Groovy you could accomplish this like this:
>
> def ff = session.get()
> if (ff) {
>   def attr_val = ff.getAttribute("myScript")
>   def result = Eval.me(attr_val)
> }
>
> That said, I'd definitely get signoffs from other parties on how this fits
> in because if the people don't sanitize the config files, you've just
> created a remote code execution hole in your ETL pipeline.
>
> Hope that helps.
>
> On Wed, Feb 5, 2020 at 2:15 PM Anurag Sharma 
> wrote:
>
>> Hi,
>>
>>
>> We are trying to create a generic data flow where configuration will be
>> received as JSON. Configuration will have a JavaScript to be executed.
>>
>>
>>
>> We are stuck on how to execute the script we received in configuration,
>> as the ExecuteScript processor needs either scriptPath or scriptBody. But
>> in our case script will be in flowFile's attribute.
>>
>>
>>
>> What are all possible ways to achieve this functionality?
>>
>>
>>
>> Along with this, the script should not be able to access the 'session'
>> and 'context' variables available in the ExecuteScript processor. So how to
>> sanitise the script before execution?
>>
>>
>> Regards
>>
>> Anurag
>>
>

Re: Execute Script from Configuration in a Generic Flow

[Mike Thomsen]

> What are all possible ways to achieve this functionality?

An eval function. Don't know the right way to do it with Nashorn, but in
Groovy you could accomplish this like this:

def ff = session.get()
if (ff) {
  def attr_val = ff.getAttribute("myScript")
  def result = Eval.me(attr_val)
}

That said, I'd definitely get signoffs from other parties on how this fits
in because if the people don't sanitize the config files, you've just
created a remote code execution hole in your ETL pipeline.

Hope that helps.

On Wed, Feb 5, 2020 at 2:15 PM Anurag Sharma 
wrote:

> Hi,
>
>
> We are trying to create a generic data flow where configuration will be
> received as JSON. Configuration will have a JavaScript to be executed.
>
>
>
> We are stuck on how to execute the script we received in configuration, as
> the ExecuteScript processor needs either scriptPath or scriptBody. But in
> our case script will be in flowFile's attribute.
>
>
>
> What are all possible ways to achieve this functionality?
>
>
>
> Along with this, the script should not be able to access the 'session' and
> 'context' variables available in the ExecuteScript processor. So how to
> sanitise the script before execution?
>
>
> Regards
>
> Anurag
>

Re: COnsumekakarecord error messages

[KhajaAsmath Mohammed]

I am using json record reader and json record set writer in consume kafka. Do I 
need to change these record readers just to confirm about the corrupt data? Can 
you help me what readers should I use in that case ?

Sent from my iPhone

> On Feb 5, 2020, at 3:47 PM, Pierre Villard  
> wrote:
> 
> 
> Hi,
> 
> It seems like you have a message that cannot be parsed correctly by your 
> record processor. It is probably not matching your schema or containing some 
> corrupted data? You could use a the non record ConsumeKafka processor to get 
> the message without doing any parsing and see what the message looks like.
> 
> HTH
> Pierre
> 
>> Le mer. 5 févr. 2020 à 16:27, KhajaAsmath Mohammed  
>> a écrit :
>> Hi,
>> 
>> I was able to consume messages from consumekafkarecord processor. Suddenly I 
>> started getting below error. any solutions for this please?
>> 
>> 
>> 
>> 
>> Thanks,
>> Asmath

Re: Execute Script from Configuration in a Generic Flow

[Emanuel Oliveira]

Hi,

processor ReplaceText (.*) with ${yourattribute} - replaces FF content with
expression  (example 1 attribute)
But looking into ExecuteScript documentation
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-scripting-nar/1.5.0/org.apache.nifi.processors.script.ExecuteScript/index.html
Seems only "Script File" and "Module Directory" supports Expression
Language: true (aka use of attributes etc).
Sadly "Script Body" doesnt support  Expression Language, meaning you cant
load script body from ${yourattribute}.

You may try the workaround of saving script as local file and then set both
directory + script file dynamically as they support expression.

Hope this helps,
Emanuel O.




Best Regards,
*Emanuel Oliveira*



On Wed, Feb 5, 2020 at 7:15 PM Anurag Sharma 
wrote:

> Hi,
>
>
> We are trying to create a generic data flow where configuration will be
> received as JSON. Configuration will have a JavaScript to be executed.
>
>
>
> We are stuck on how to execute the script we received in configuration, as
> the ExecuteScript processor needs either scriptPath or scriptBody. But in
> our case script will be in flowFile's attribute.
>
>
>
> What are all possible ways to achieve this functionality?
>
>
>
> Along with this, the script should not be able to access the 'session' and
> 'context' variables available in the ExecuteScript processor. So how to
> sanitise the script before execution?
>
>
> Regards
>
> Anurag
>

Execute Script from Configuration in a Generic Flow

[Anurag Sharma]

Hi,


We are trying to create a generic data flow where configuration will be
received as JSON. Configuration will have a JavaScript to be executed.



We are stuck on how to execute the script we received in configuration, as
the ExecuteScript processor needs either scriptPath or scriptBody. But in
our case script will be in flowFile's attribute.



What are all possible ways to achieve this functionality?



Along with this, the script should not be able to access the 'session' and
'context' variables available in the ExecuteScript processor. So how to
sanitise the script before execution?


Regards

Anurag

Re: NiFi import processor group created with previous NiFi version does not work

[Mike Thomsen]

FYI, when you are building processors that reference controller service
packages (ex Record API services), you'll want to make sure that you
parameterize the Maven dependency version like ${nifi.version} so that when
you do upgrade versions it's a simple case of this to rebuild:

mvn clean install -Dnifi.version=1.11.1

On Wed, Feb 5, 2020 at 6:19 AM Valentina Ivanova 
wrote:

> Hi Jozef!
>
> Many thanks for your reply! Just tested it and that is the case.
> Everything works when I remove the disabled processor.
>
> Many thanks again & all the best
>
> Valentina
> --
> *From:* josef.zahn...@swisscom.com
> *Sent:* Wednesday, February 05, 2020 12:04
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi import processor group created with previous NiFi
> version does not work
>
> Hi Valentina,
>
>
>
> has your template/canvas disabled processors? I’ve already faced a bug
> which caused the same error as yours.
>
> https://issues.apache.org/jira/projects/NIFI/issues/NIFI-6958
>
>
>
> Cheers Josef
>
>
>
>
>
> *From: *Valentina Ivanova 
> *Reply to: *"users@nifi.apache.org" 
> *Date: *Wednesday, 5 February 2020 at 11:10
> *To: *"users@nifi.apache.org" 
> *Subject: *NiFi import processor group created with previous NiFi version
> does not work
>
>
>
> Hi everyone!
>
>
>
> We are investigating the following issue.
>
> I have a dev NiFi instance with version 1.10.0 where I created a process
> group template with approximately 15 processors. I can upload that template
> to the production NiFi with version 1.11.0 and then start version control
> and commit it to the nifi registry.
>
> However when I delete that template from the canvas and try to import it
> from the registry I get the following error message - No Processor with ID
>  belongs to this Process Group.
>
> If I create the processor group template in the production NiFi 1.11.0
> then I can delete it and import it back to the canvas without issues.
>
> Do you have any idea why this happens? Is this any versioning issues in
> connection to the NiFi processors? Help is very appreciated.
>
>
>
> Thanks & all the  best
>
>
>
> Valentina
>

Developing NiFi custom processors in NetBeans

[Valentina Ivanova]

Hi everyone!

We faced another issue when trying to create custom processors with NetBeans 
8.2.
I try to create a new project in the following way File-> New Project -> Maven 
-> Project from Maven Archetype and then selecting 
nifi-processor-bundle-archetype from the available archetypes. Then on the next 
screen filling in the requested fields. Finally when pressing finish to create 
it I get the following (see below).

After some investigation I figured out that file can't be downloaded:
http://repo.maven.apache.org/maven2/org/apache/nifi/nifi/1.11.0/nifi-1.11.0.pom
as the protocol needs to be set to HTTPS. So I created a custom repository (and 
built the index) which uses the HTTPS protocol and try to create the project 
using it. However, it seems the HTTP version is still used, so creating the 
project fails again.

When I browse the Maven repositories in the Services tab and try to download 
certain org.apache.nifi poms I get error messages for some dependencies are 
broken which is not clear how to fix.

So I wonder if anyone has had the same issues and how to resolve them.
(I tried to use maven from command line but other issues appear amd I would 
prefer the UI as I am a maven beginner.)

Many thanks & all the best

Valentina



cd C:\Users\valentinaiv\Desktop\NetBeansProjects\ReadProtoBuf; 
"JAVA_HOME=C:\\Program Files\\AdoptOpenJDK\\jdk-8.0.212.03-hotspot" cmd /c 
"\"\"C:\\Program Files\\NetBeans 8.2\\java\\maven\\bin\\mvn.bat\" 
-DarchetypeGroupId=org.apache.nifi 
-DarchetypeArtifactId=nifi-processor-bundle-archetype -DarchetypeVersion=1.11.0 
-DarchetypeRepository=https://repo.maven.apache.org/maven2/ -DgroupId=rise.koda 
-DartifactId=mavenproject2 -Dversion=1.0-SNAPSHOT 
-Dpackage=rise.koda.mavenproject2 
-Dbasedir=C:\\Users\\valentinaiv\\Desktop\\NetBeansProjects\\ReadProtoBuf 
-DartifactBaseName=read -DnifiVersion=1.11.0 -Darchetype.interactive=false 
-Dmaven.ext.class.path=\"C:\\Program Files\\NetBeans 
8.2\\java\\maven-nblib\\netbeans-eventspy.jar\" --batch-mode 
archetype:generate\""
Scanning for projects...


Building Maven Stub Project (No POM) 1


>>> maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom >>>

<<< maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom <<<

--- maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom ---
Generating project in Batch mode
No archetype found in remote catalog. Defaulting to internal catalog
Archetype not found in any catalog. Falling back to central repository.
Add a repository with id 'archetype' in your settings.xml if archetype's 
repository is elsewhere.
Downloading: 
http://repo.maven.apache.org/maven2/org/apache/nifi/nifi/1.11.0/nifi-1.11.0.pom

BUILD FAILURE

Total time: 1.991s
Finished at: Wed Feb 05 16:37:23 CET 2020
Final Memory: 14M/243M

Failed to execute goal 
org.apache.maven.plugins:maven-archetype-plugin:3.1.2:generate (default-cli) on 
project standalone-pom: The desired archetype does not exist 
(org.apache.nifi:nifi-processor-bundle-archetype:1.11.0) -> [Help 1]

To see the full stack trace of the errors, re-run Maven with the -e switch.
Re-run Maven using the -X switch to enable full debug logging.

For more information about the errors and possible solutions, please read the 
following articles:
[Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Re: NiFi import processor group created with previous NiFi version does not work

[Valentina Ivanova]

Hi Jozef!

Many thanks for your reply! Just tested it and that is the case. Everything 
works when I remove the disabled processor.

Many thanks again & all the best

Valentina

From: josef.zahn...@swisscom.com
Sent: Wednesday, February 05, 2020 12:04
To: users@nifi.apache.org
Subject: Re: NiFi import processor group created with previous NiFi version 
does not work


Hi Valentina,



has your template/canvas disabled processors? I’ve already faced a bug which 
caused the same error as yours.

https://issues.apache.org/jira/projects/NIFI/issues/NIFI-6958



Cheers Josef





From: Valentina Ivanova 
Reply to: "users@nifi.apache.org" 
Date: Wednesday, 5 February 2020 at 11:10
To: "users@nifi.apache.org" 
Subject: NiFi import processor group created with previous NiFi version does 
not work



Hi everyone!



We are investigating the following issue.

I have a dev NiFi instance with version 1.10.0 where I created a process group 
template with approximately 15 processors. I can upload that template to the 
production NiFi with version 1.11.0 and then start version control and commit 
it to the nifi registry.

However when I delete that template from the canvas and try to import it from 
the registry I get the following error message - No Processor with ID  
belongs to this Process Group.

If I create the processor group template in the production NiFi 1.11.0 then I 
can delete it and import it back to the canvas without issues.

Do you have any idea why this happens? Is this any versioning issues in 
connection to the NiFi processors? Help is very appreciated.



Thanks & all the  best



Valentina

Re: NiFi import processor group created with previous NiFi version does not work

[Josef.Zahner1]

Hi Valentina,

has your template/canvas disabled processors? I’ve already faced a bug which 
caused the same error as yours.
https://issues.apache.org/jira/projects/NIFI/issues/NIFI-6958

Cheers Josef


From: Valentina Ivanova 
Reply to: "users@nifi.apache.org" 
Date: Wednesday, 5 February 2020 at 11:10
To: "users@nifi.apache.org" 
Subject: NiFi import processor group created with previous NiFi version does 
not work

Hi everyone!

We are investigating the following issue.
I have a dev NiFi instance with version 1.10.0 where I created a process group 
template with approximately 15 processors. I can upload that template to the 
production NiFi with version 1.11.0 and then start version control and commit 
it to the nifi registry.
However when I delete that template from the canvas and try to import it from 
the registry I get the following error message - No Processor with ID  
belongs to this Process Group.
If I create the processor group template in the production NiFi 1.11.0 then I 
can delete it and import it back to the canvas without issues.
Do you have any idea why this happens? Is this any versioning issues in 
connection to the NiFi processors? Help is very appreciated.

Thanks & all the  best

Valentina


smime.p7s
Description: S/MIME Cryptographic Signature

NiFi import processor group created with previous NiFi version does not work

[Valentina Ivanova]

Hi everyone!

We are investigating the following issue.
I have a dev NiFi instance with version 1.10.0 where I created a process group 
template with approximately 15 processors. I can upload that template to the 
production NiFi with version 1.11.0 and then start version control and commit 
it to the nifi registry.
However when I delete that template from the canvas and try to import it from 
the registry I get the following error message - No Processor with ID  
belongs to this Process Group.
If I create the processor group template in the production NiFi 1.11.0 then I 
can delete it and import it back to the canvas without issues.
Do you have any idea why this happens? Is this any versioning issues in 
connection to the NiFi processors? Help is very appreciated.

Thanks & all the  best

Valentina