Obtaining Process Group Version Number

[Dan S]

My team is trying to obtain the version numbers of all the process groups
they have in a cluster. There are two REST endpoints which I thought could
solve this issue but it turns out neither one does the entire job by itself.

/flow/process-groups/{id}/status?recursive=true
allows for obtaining all the process groups but it does not have the
version number
while
/process-groups/{id}/process-groups allows for obtaining the version number
of the process groups but it is not recursive so only the top level process
groups are retrieved.

Is there another endpoint that can be recursive to retrieve all the process
groups and their version numbers?
If there is no endpoint, can there be a ticket created which I would be
glad to work on  for including the version number in
/flow/process-groups/{id}/status?recursive=true and/or making
/process-groups/{id}/process-groups  recursive?

Re: Processing JSON Files Embedded with Base64 Encoded Files

[Mike Thomsen]

Can you be more specific on what you're trying to accomplish since the
RecordPath API has base64 encode/decode functions for stuff like decoding
inline base64 content?

On Tue, Aug 15, 2023 at 9:45 AM Reid, Chris 
wrote:

> Hello.
>
> I understand that there's currently no Apache NiFi processor that can
> process JSON files embedded with Base64 encoded files.
>
> I'd like to know if it's possible to develop a custom Apache NiFi
> processor to process JSON files embedded with Base64 files?
>
> Thank you.
>
> Chris
>

[VOTE] Release Apache NiFi 1.23.1 (RC1)

[David Handermann]

Team,

I am pleased to be calling this vote for the source release of Apache NiFi
1.23.1.

Please review the following guide for how to verify a release candidate
build:

https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The source being voted upon and the convenience binaries are available on
the Apache Distribution Repository:

https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.1/

The build artifacts are available on the Apache Nexus Repository:

https://repository.apache.org/content/repositories/orgapachenifi-1234

Git Tag: nifi-1.23.1-RC1
Git Commit ID: f739e9dd2d476b3c0df5a806ff1dffd24be52916
GitHub Commit Link:
https://github.com/apache/nifi/commit/f739e9dd2d476b3c0df5a806ff1dffd24be52916

Checksums of nifi-1.23.1-source-release.zip:

SHA256: 01f5f67a9f9703232f6fe260f6c1da73f3e3a0764b8e8464f915cfad168278e6
SHA512:
f8a010ad5a5dd1f71fe04e5d2bf1c6637d2d0a8a7c580a0ff4dbd76c12c2e5ab4ac43e1f5314f9fca85cebe1606bd5e7ae0a8b62f577ddc68696ebd0155d

Release artifacts are signed with the following key:

https://people.apache.org/keys/committer/exceptionfactory.asc

KEYS file available here:

https://dist.apache.org/repos/dist/release/nifi/KEYS

51 issues were closed/resolved for this release:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353541

Release note highlights can be found here:

https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.1

The vote will be open for 72 hours.

Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build from source, and test. Then
please vote:

[ ] +1 Release this package as nifi-1.23.1
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: Processing JSON Files Embedded with Base64 Encoded Files

[Joe Witt]

Also depending on the setup you could first use
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.EncodeContent/index.html
in decode mode then do normal json handling.

Thanks

On Tue, Aug 15, 2023 at 7:42 AM Yolanda Davis 
wrote:

> Hello Chris,
>
> Depending on the structure of your Json file, you may be able to achieve
> what you need using processors such as JoltTransformJSON
> <
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.22.0/org.apache.nifi.processors.standard.JoltTransformJSON/index.html
> >
> paired with NiFi's expression language which supports base64
> <
> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64encode
> >
> encoding and decoding.  JoltTransform is very useful if you're looking to
> convert or transform incoming JSON data to another json structure. Also
> EvaluateJsonPath
> <
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.22.0/org.apache.nifi.processors.standard.EvaluateJsonPath/index.html
> >
> may
> be useful as well if you're looking to extract data first from json then
> potentially decode/encode
>
> Hope this helps!
>
> -yolanda
>
> On Tue, Aug 15, 2023 at 9:45 AM Reid, Chris 
> wrote:
>
> > Hello.
> >
> > I understand that there's currently no Apache NiFi processor that can
> > process JSON files embedded with Base64 encoded files.
> >
> > I'd like to know if it's possible to develop a custom Apache NiFi
> > processor to process JSON files embedded with Base64 files?
> >
> > Thank you.
> >
> > Chris
> >
>
>
> --
> --
> yolanda.m.da...@gmail.com
> @YolandaMDavis
>

Re: Processing JSON Files Embedded with Base64 Encoded Files

[Yolanda Davis]

Hello Chris,

Depending on the structure of your Json file, you may be able to achieve
what you need using processors such as JoltTransformJSON

paired with NiFi's expression language which supports base64

encoding and decoding.  JoltTransform is very useful if you're looking to
convert or transform incoming JSON data to another json structure. Also
EvaluateJsonPath

may
be useful as well if you're looking to extract data first from json then
potentially decode/encode

Hope this helps!

-yolanda

On Tue, Aug 15, 2023 at 9:45 AM Reid, Chris 
wrote:

> Hello.
>
> I understand that there's currently no Apache NiFi processor that can
> process JSON files embedded with Base64 encoded files.
>
> I'd like to know if it's possible to develop a custom Apache NiFi
> processor to process JSON files embedded with Base64 files?
>
> Thank you.
>
> Chris
>


-- 
--
yolanda.m.da...@gmail.com
@YolandaMDavis

Processing JSON Files Embedded with Base64 Encoded Files

[Reid, Chris]

Hello.

I understand that there's currently no Apache NiFi processor that can process 
JSON files embedded with Base64 encoded files.

I'd like to know if it's possible to develop a custom Apache NiFi processor to 
process JSON files embedded with Base64 files?

Thank you.

Chris

Re: JoltTransformJSON EL when using file

[Matthew Hawkins]

Hi Matt,

The tag will be ${firstname}, the spec is correct.
(well, I also get lastname, but it's unimportant)

Weird thing is this failed similarly with a file based input on the Record
based jolt processor as well on a completely different system.

Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
source with include-graph,include-media,include-rules,include-sql-reporting

In the second case I was trying to get ${now():toNumber()} into a json
record using either default or modify-overwrite-beta (tried both). The
following success processor failed on reading schema as the literal
${now...} was a string not a long and the JVM refused the type cast. If I
insert the transform directly then it still fails with a type cast problem
from generic Object :/ (that was midnight last night, and I tossed my hands
in the air and turned the server off)

I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
postgresql. Trying to be a good nifi citizen and use record based
processing where possible. Since I don't run Hadoop at home, it's all
manual schemas and using postgres as the perm data store. I'm using json as
mqtt does it intrinsically and I can jolt transform other data into json
form and then store it easily in postgres. Well, that was the idea 

Kr,

On Sun, 13 Aug 2023, 10:44 Matt Burgess,  wrote:

> Just to follow up, I added a unit test to put EL in a JOLT spec and it
> worked. I noticed you referred to "attrname" in your post but your
> spec refers to "firstname", is that a typo?
>
> Regards,
> Matt
>
> On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess  wrote:
> >
> > I added file support to JoltTransformJSON under NIFI-4957 [1], a first
> > glance at the code seems like it should work, but I'll try to
> > reproduce it and follow up, thanks for bringing this to our attention!
> >
> > Regards,
> > Matt
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-4957
> >
> > On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins 
> wrote:
> > >
> > > Hi devs,
> > >
> > > Using 1.23.0 I have a simple transform adding some flowfile attributes
> into
> > > the content.
> > > When putting the spec directly to the processor it works fine. If I
> have
> > > the spec in an external file however it seems to put in the literal
> string
> > > ${attrname} - ie the EL didn't process.
> > >
> > > Have I done something incorrectly or should I be now asking for an
> account
> > > to log a bug?
> > >
> > > Sample spec run using Chain DSL:
> > >
> > > [{
> > >   "operation": "shift",
> > >   "spec": {
> > > "@": "values"
> > >   },
> > >   {
> > > "operation": "default",
> > > "spec": {
> > >   "firstname": "${firstname}"
> > > }
> > >   }
> > > ]
> > >
> > > Kind regards,
> > > Matthew
>