Re: AccessApi CreateDownloadToken deprecated

[Daniel Chaffelson]

Thanks David, that makes good sense and I just couldn't find the relevant
Jira for myself earlier.

On Mon, Jan 23, 2023 at 2:24 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Hi Dan,
>
> NiFi 1.15.0 included the removal of One-Time Password Authentication as
> described in NIFI-8931 [1].
>
> Depending on the authentication strategy configured, the Access Token REST
> API can be used to create a Bearer Token that can be used for a longer
> duration. Configuring and authorizing X.509 Client Certificates is another
> option for programmatic access, but there is no direct replacement for
> One-Time Password Authentication.
>
> Regards,
> David Handermann
>
> [1] https://issues.apache.org/jira/browse/NIFI-8931
>
> On Mon, Jan 23, 2023 at 3:37 AM Daniel Chaffelson 
> wrote:
>
> > Hi folks,
> > A user has created an Issue ticket[1] on NiPyAPI after upgrading NiFi
> from
> > 1.12 to 1.16.
> > It seems that sometime in this window the AccessApi has had the
> > CreateDownloadToken function deprecated. This function appears to have
> > issued a one-time auth token to download flowfile content, and I cannot
> see
> > a replacement for this particular functionality.
> > I can see that a user with the correct authz against Flowfile content
> would
> > be able to use the FlowfileQueuesApi to DownloadFlowFileContent once
> their
> > client had a bearer token, but this would be subtly different from a
> > one-time token indicated in this older function.
> >
> > Can anyone advise on this deprecation and if the functionality moved
> > elsewhere?
> >
> > [1] https://github.com/Chaffelson/nipyapi/issues/324
> >
>

AccessApi CreateDownloadToken deprecated

[Daniel Chaffelson]

Hi folks,
A user has created an Issue ticket[1] on NiPyAPI after upgrading NiFi from
1.12 to 1.16.
It seems that sometime in this window the AccessApi has had the
CreateDownloadToken function deprecated. This function appears to have
issued a one-time auth token to download flowfile content, and I cannot see
a replacement for this particular functionality.
I can see that a user with the correct authz against Flowfile content would
be able to use the FlowfileQueuesApi to DownloadFlowFileContent once their
client had a bearer token, but this would be subtly different from a
one-time token indicated in this older function.

Can anyone advise on this deprecation and if the functionality moved
elsewhere?

[1] https://github.com/Chaffelson/nipyapi/issues/324

Re: Using nifi in separated networks

[Daniel Chaffelson]

This is a very interesting area of integration investigation Marc, thank
you for sharing your work!

I looked into this a little after conversations with folks in security
applications, and I wonder if you investigated approaches to tracking and
reporting/handling packet loss and error rates in this?
The interest was in reasoning about loss rates, and the completeness of
received data - something with a simple merge>put>diode>get>unpack would
not manage I think.

I was looking at Longhair , and similar
reed-solomon approaches, as a method of breaking down arbitrary files and
transmitting for reconstitution over diodes that may have lossy behavior in
Field scenarios.
I also looked a little into transmitting manifests for downstream
reconciliation, but this unravelled to be more complex an operation than
would suit a pure NiFi implementation, so I started on the path of
Kafka/Flink as a streaming-reconciliation service but quickly realised i
was creating a monster without commercial interest :)
Both approaches are easier for fewer larger files than millions of tiny
messages in terms of practicality, and if you had very reliable diode
transmission the overhead of ecc/reconciliation may not be worthwhile.
Other implementations I had seen (like ZeroMQ radio/dish or blindFTP)
seemed to talk about provable delivery as a potential requirement, but I
only found the more simplistic 'my network is reliable and any packet loss
is negligible anyway' approaches. I suspect the implementations of these
more robust approaches are reserved for commercial offerings...

Anyway, I appreciate that you may not be able to share more details on
this, but you reminded me of enjoying the investigation when I looked at it
so I thought I'd say thanks for that.

On Tue, Aug 3, 2021 at 2:55 AM Phil H  wrote:

> Adam, that's true, although if your data size is larger than network
> MTU there can be some disconnect there.
>
> Connection per flow file is pretty slow for sustained high traffic
> flows though (can't recall the establishment times off the top of my
> head, but they are non-trivial).
>
> On Tue, Aug 3, 2021 at 8:39 AM Adam Taft  wrote:
> >
> > Just spitballing a little here. If you set the configuration of the
> PutTCP
> > processor property "Connection per Flowfile" to 'true' and you leave the
> > "Outgoing Message Delimiter" as blank (none), then I don't think you have
> > the delimiter problem that you both are describing. I could be wrong
> though?
> >
> > I would consider it a bug if you couldn't send a "raw"
> connection-oriented
> > object over PutTCP.  With that processor, the goal would be to: a) open a
> > socket, b) dump whatever binary you have prepared over it, c) close the
> > socket to signal completion of transfer. If PutTCP doesn't work this way
> > (byte-for-byte), it should probably be flagged as a bug (its original
> > intention was exactly this use case).
> >
> > That being said, I still think custom FlowFile serialization might be
> > something that is outside of the concern of the transport. I personally
> > think serializing/deserializing is a different concern from transport.
> > Arguably, sometimes the semantics of the transport protocol requires you
> to
> > prepare the message itself in a protocol accommodating way (HTTP being an
> > obvious example of this, or packet ordering in Marc's UDP example). But a
> > new JSON flowfile serialization seems like it could be a separate
> > processor, not commingled into an existing one.
> >
> > MergeContent / UnpackContent work in tandem and have a "FlowFile Stream
> v3"
> > format that can serialize/deserialize multiple flowfiles together into a
> > single byte stream. This allows transport over any protocol, including
> > file-based, socket-based, etc.
> >
> > Marc: Your mention of performance is, of course, appropriate for the
> scale
> > that you're talking about (Gbps). Maybe there's some performance
> > improvements that could be garnered from your work applicable to the
> > "standard" processors I mentioned. And I definitely didn't mean to imply
> > you were doing "anything wrong". Just legitimately curious as to your
> > thought process and design approach.
> >
> > OK, I'll step off a little, because I might be probing too hard here.
> But I
> > was legitimately curious about the intention of the proposed processor as
> > it relates to the mentioned Diode device.
> >
> > Thanks,
> >
> > Adam
> >
> >
> > On Mon, Aug 2, 2021 at 4:15 PM Phil H  wrote:
> >
> > > Hi Marc,
> > >
> > > Thanks for the additional info.  Just so you know you’re not the only
> > > one, I’ve also had to re-implement a ListenTCP alternative to get
> > > around the byte delimeter issue for binary and multiline text data.
> > >
> > > Phil
> > >
> > >
> > > On Tue, Aug 3, 2021 at 6:59 AM Marc  wrote:
> > > >
> > > > Hi Adam,
> > > >
> > > > more or less it is a ‚merge', puttcp, listentcp and unpack. I hope
> that
> > > I am not wrong but the nifi 

Re: Issue in automate deployment of Flow from one registry bucket to another using Nipy API for flow having Parameter Context

[Daniel Chaffelson]

Hi Deepak,
I'll help take a look, can you please bump NiFi logging up to Debug and
reproduce the error, then post the information as an Issue on the NiPy repo
for me?
https://github.com/Chaffelson/nipyapi/issues
I have only recently started looking at Parameter Contexts in the NiPy
wrapper functionality, and it is not yet tested with Registry functions, so
this is a new error to me.
If I get time this weekend I'll make a start on it, but it may have to wait
until Monday (Pandemic Childcare).

On Fri, Nov 6, 2020 at 1:01 PM deepak.gu...@spglobal.com <
deepak.gu...@spglobal.com> wrote:

> HI Team,
>
>
>
> We are facing the below mentioned issue while deploying the Nifi flow from
> one Nifi Registry bucket to another bucket using Nipy API, if flow contains
> any parameter context. However, it is working fine if flow doesn’t contain
> any parameter context.
>
>
>
> Issue is while calling nipyapi.versioning.deploy_flow_version API with
> following error message
>
>
>
> raise ApiException(http_resp=r)
>
> nipyapi.nifi.rest.ApiException: (500)
>
> Reason: Internal Server Error
>
> HTTP response headers: HTTPHeaderDict({'Date': 'Tue, 20 Oct 2020 07:55:30
> GMT', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Security-Policy':
> "frame-ancestors 'self'", 'X-XSS-Protection': '1; mode=block',
> 'Strict-Transport-Security': 'max-age=3154', 'Content-Type':
> 'text/plain', 'Vary': 'Accept-Encoding', 'Content-Length': '79', 'Server':
> 'Jetty(9.4.19.v20190610)'})
>
> HTTP response body: An unexpected error has occurred. Please check the
> logs for additional details.
>
>
>
> There is no error/exception in any other additional logs.
>
>
>
> It seems that there is some issue in exporting and importing the flow from
> source bucket to destination bucket using Nipy API. As after importing the
> flow in destination bucket, it is also giving error while importing it at
> destination Nifi instance using Nifi UI from destination bucket.
>
> And it is working fine while importing the flow through destination Nifi
> UI from source bucket. But we are not getting any error in 
> nipyapi.versioning.export_flow_version
> and nipyapi.versioning.import_flow_version API calls
>
>
>
> Could you please look into it and suggest if it is not possible through
> Nipy API if flow contains any parameter context. Or is there any other way
> to do it.
>
> I have attached the source code for the reference.
>
>
>
> We are using following  versions:
>
>
>
> Docker Nifi – 1.11.4
>
> Docker Nifi Registry – nifi-registry-0.7.0
>
> Nipy API - 0.14.3
>
>
>
>
>
> Regards,
>
> Deepak Gupta
>
>
>
> --
>
> The information contained in this message is intended only for the
> recipient, and may be a confidential attorney-client communication or may
> otherwise be privileged and confidential and protected from disclosure. If
> the reader of this message is not the intended recipient, or an employee or
> agent responsible for delivering this message to the intended recipient,
> please be aware that any dissemination or copying of this communication is
> strictly prohibited. If you have received this communication in error,
> please immediately notify us by replying to the message and deleting it
> from your computer. S Global Inc. reserves the right, subject to
> applicable local law, to monitor, review and process the content of any
> electronic message or information sent to or from S Global Inc. e-mail
> addresses without informing the sender or recipient of the message. By
> sending electronic message or information to S Global Inc. e-mail
> addresses you, as the sender, are consenting to S Global Inc. processing
> any of your personal data therein.
>

Re: Collaboration for Azure features

[Daniel Chaffelson]

I would check how search functionality works in two cases; do you get
results from channels you are not a member of, do you get results from
archived channels.
If both are true, then having side channels doesn't prevent the most common
way of finding specific information, though it would be good to ensure that
general chat remains in generally channels to encourage community cohesion.

On Fri, 17 Apr 2020, 19:06 Bryan Bende,  wrote:

> Personally I have always wished there was just a single channel with
> threaded convos, rather than needing to monitor many channels and see the
> same question cross posted multiple times, but since we are well past that
> point, I am not really opposed. The only issue I could see is where to draw
> the line, we could easily create channels for many areas development and
> have it get out of hand.
>
> On Fri, Apr 17, 2020 at 1:54 PM Pierre Villard <
> pierre.villard...@gmail.com>
> wrote:
>
> > Hi,
> >
> > On my side, I think this is a good idea if that helps having groups of
> > collaboration focusing on specific areas for NiFi.
> >
> > If having too much channels in our Slack is a concern, I guess a
> #dev_cloud
> > channel could be an intermediate solution (assuming we expect to see
> > #dev_aws and #dev_gcp now or in the future). We could then re-evaluate
> > based on how much people are interacting there.
> >
> > Pierre
> >
> > Le ven. 17 avr. 2020 à 18:28, Ken Swanson  a
> écrit
> > :
> >
> > > Is there a place to vote on channel proliferation in Slack? I might
> have
> > > missed that.
> > >
> > > -Ken
> > >
> > > On Fri, Apr 17, 2020 at 11:03 AM Joe Witt  wrote:
> > >
> > > > Muazma,
> > > >
> > > > My initial reaction was strongly against this as it creates even more
> > > > channels and our slack in my view already has too many.
> > > >
> > > > However, we definitely do want groups of extensions to be well
> > > > managed/groomed by contributors and users alike.  So in that regard
> > this
> > > is
> > > > a good focus area.  We want to get extension bundles into their own
> > > project
> > > > structure and out of the main nifi build.  So this is probably a
> > helpful
> > > > step in that direction.
> > > >
> > > > I'm fine with it.
> > > >
> > > > I'd let folks comment and see if there is consensus.  Dont think we
> > need
> > > a
> > > > vote or anything.  Our channel proliferation on slack hasnt had any
> > votes
> > > > either.
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > >
> > > > On Fri, Apr 17, 2020 at 11:20 AM Muazma Zahid 
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I wanted to float an idea for creating a dedicated #dev_azure
> channel
> > > in
> > > > > the NiFi Slack where people could chime in and collaborate on new
> > > > features.
> > > > > Let me know if this is something that can be done in the community
> as
> > > we
> > > > > are adding many new features and we can get feedback and
> collaborate
> > > with
> > > > > others interested.
> > > > >
> > > > > Thanks,
> > > > > Muazma
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release Apache NiFi 1.11.1 (rc1)

[Daniel Chaffelson]

+1 nonbinding

Built from specified commit with tests
Generated updated NiPyAPI client and completed tests on updated Dockers

On Sun, 2 Feb 2020, 04:08 Aldrin Piri,  wrote:

> +1, binding
>
> notes:
> * signature good
> * hashes look good
> * tagged commit matches
> * full build with contrib check correct
> * verified associated docker images functioning correctly
> * ran some simple flows and verified site to site
>
> On Sat, Feb 1, 2020 at 7:22 AM Arpad Boda 
> wrote:
>
> > +1
> >
> > Verified signature, hashes
> > Built, started NiFi
> > Created a flow
> > Transmitted flowfiles from MiNiFI
> >
> > Although I had to skip execution of tests because of this bug:
> > https://issues.apache.org/jira/browse/NIFI-7089
> > The bug also appears on master as well, so it doesn't block this patch
> > release.
> >
> >
> > On Fri, Jan 31, 2020 at 9:11 PM Joe Witt  wrote:
> >
> > > Hello,
> > >
> > > I am pleased to be calling this vote for the source release of Apache
> > NiFi
> > > 1.11.1.
> > >
> > > The source zip, including signatures, digests, etc. can be found at:
> > > https://repository.apache.org/content/repositories/orgapachenifi-1156
> > >
> > > The source being voted upon and the convenience binaries can be found
> at:
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.11.1/
> > >
> > > A helpful reminder on how the release candidate verification process
> > works:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > >
> > > The Git tag is nifi-1.11.1-RC1
> > > The Git commit ID is d22858d045fb3e5343a87d362855810963aa8556
> > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=d22858d045fb3e5343a87d362855810963aa8556
> > >
> > > Checksums of nifi-1.11.1-source-release.zip:
> > > SHA256:
> 41878981689e08c51ed7bb921c46f6328c1e6b6c94460a2382069bd042ab7112
> > > SHA512:
> > >
> > >
> >
> d309342732196225ffdbac3eaeed3c9b949bcee00509fb6d073618d869c6db56b426f5867849a8ae752d7acc8c6dd18a14cb377655774d86f91d736350eb2041
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/joewitt.asc
> > >
> > > KEYS file available here:
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 9 issues were closed/resolved for this release:
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12346906
> > >
> > > Release note highlights can be found here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.11.1
> > >
> > > The vote will be open for 72 hours.
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build
> > > from source, and test. Then please vote:
> > >
> > > [ ] +1 Release this package as nifi-1.11.1
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> > >
> >
>

Re: Programmatically passing secrets to nifi processor.

[Daniel Chaffelson]

While I haven't implemented it yet, it would be a relatively easy addition
to NiPyAPI alongside the existing Variable and Registry functionality.
If there is good community demand I can prioritise it, or someone may
choose to contribute it.

On Mon, 25 Nov 2019, 20:12 Bahlul Haider,  wrote:

> Friendly Monday morning ping.
>
> I am looking for a way to programmatically setting/updating parameters in
> newly created nifi cluster.
>
> Thanks,
> Bahlul
>
> On Fri, Nov 22, 2019 at 1:28 PM Bahlul Haider 
> wrote:
>
> > Hey Bryan,
> >
> > I was able to manually add parameters in a flow and was able to use them.
> > However I did not find a way to programmatically create the parameter
> > context values in new nifi cluster. Any help regarding this is greatly
> > appreciated.
> >
> > Happy Friday :-)
> >
> > Thanks,
> > Bahlul
> >
> > On Thu, Nov 21, 2019 at 4:57 PM Bahlul Haider 
> > wrote:
> >
> >> Thank you very much Bryan for your quick reply.
> >>
> >> I actually started with 1.10.0. But when I tried to pass variables to
> the
> >> flow using nifi CLI I was getting an error (see my comments in NIFI
> >> 6814: Nifi REST api returns incompatible date format for NiFi Client DTO
> >> <
> https://issues.apache.org/jira/projects/NIFI/issues/NIFI-6860?filter=allopenissues
> >).
> >> The regression was later fixed.
> >>
> >> I will try parameters in 1.10.
> >>
> >> Thanks,
> >> Bahlul
> >>
> >> On Thu, Nov 21, 2019 at 11:10 AM Bryan Bende  wrote:
> >>
> >>> Hello,
> >>>
> >>> If you can use the latest 1.10.0 release there is a new feature called
> >>> parameters that will greatly help with this.
> >>>
> >>> You can use a new syntax #{my.secret} which will remain in the flow
> >>> when saved to registry.
> >>>
> >>> Then you can use CLI commands for setting the value of the parameter
> >>> named "my.secret".
> >>>
> >>> Thanks,
> >>>
> >>> Bryan
> >>>
> >>> On Thu, Nov 21, 2019 at 2:04 PM Bahlul Haider 
> >>> wrote:
> >>> >
> >>> > Hello,
> >>> >
> >>> > I am working on automatically starting a flow from registry in newly
> >>> > created nfi clusters. To start the flow, I need to provide some
> secret
> >>> > values to some of the processors. What is the recommended way of
> >>> > programmatically provide secret of a processor?
> >>> >
> >>> > So far, I have tried NiFi CLI to pass secrets as variables to NiFi
> >>> (e.g.,
> >>> > {secretName, secretValue}) and reference ${secretName} in the
> >>> processor (I
> >>> > am aware that NiFi does yet support encryption for secrets in
> >>> variables).
> >>> > However, I am unable to create a new version of the flow (from the
> >>> browser)
> >>> > in the registry with ${secretName} as the property value (NiFi thinks
> >>> it is
> >>> > a secret and does not include it in the new version). I was hoping
> that
> >>> > NiFi CLI have a way to update flow in registry with ${secretName} as
> >>> secret
> >>> > value. However, I did not find a way to to do it.
> >>> >
> >>> > I am a newbie to nifi. Any help regarding this is greatly
> appreciated.
> >>> I
> >>> > have seen there are several similar open issues in nifi. I am willing
> >>> to
> >>> > work on those issues if my scenario is not supported.
> >>> >
> >>> > Thanks,
> >>> > Bahlul
> >>>
> >>
> >>
> >> --
> >> Thanks,
> >> Bahlul
> >>
> >>
> >
> > --
> > Thanks,
> > Bahlul
> >
> >
>
> --
> Thanks,
> Bahlul
>

Re: Hive backward compatibily

[Daniel Chaffelson]

Hi Milan,
Apologies for a link as an answer, but I'm currently traveling and
responding from mobile.

I worked through a similar problem in NiFi-1.1 and shared details of my
method which hopefully should help you.

https://stackoverflow.com/questions/39200903/apache-nifi-hive-processors-with-hive-1-1-cdh-5-7-1

Cheers,
Dan.

On Mon, 5 Mar 2018, 18:21 Matt Burgess,  wrote:

> Milan,
>
> Unfortunately no, without code changes the Hive processors are not
> compatible with Hive 1.1.x.  There were API changes both to the Java code
> and the Thrift code (the latter adds the client_protocol field you
> mentioned), so you can't change the version of the Hive dependencies and
> get the NAR to compile. You'd have to make the code changes yourself and
> build with the appropriate version of Hive. There are some vendor-specific
> profiles (hortonworks, cloudera, mapr) that you can use when specifying
> vendor-specific Hive versions.
>
> Regards,
> Matt
>
>
> On Mon, Mar 5, 2018 at 12:29 PM, Milan Das  wrote:
>
> > Hello NIFI Dev,
> >
> > I am trying to use SelectHiveQL with Hive version hive-1.1.0 (it is part
> > of the Cloudera 5.13.1)
> >
> > https://www.cloudera.com/documentation/enterprise/
> > release-notes/topics/cdh_vd_cdh_package_tarball_513.html#tarball_5131
> >
> >
> >
> > Seems like because SelectHiveQL uses hive-1.2.1 client library I am
> > getting following error. My NIFI version is 1.4.0
> >
> >
> >
> > Caused by: java.sql.SQLException: Could not establish connection to
> > jdbc:hive2://ip-NNN-NN-N-NNN.ec2.internal:1/default;
> > principal=hive/_h...@.interset.com: Required field 'client_protocol' is
> > unset! Struct:TOpenSessionReq(client_protocol:null,
> > configuration:{use:database=default})
> >
> > at org.apache.hive.jdbc.HiveConnection.openSession(
> > HiveConnection.java:594)
> >
> > at org.apache.hive.jdbc.HiveConnection.(
> > HiveConnection.java:192)
> >
> >
> >
> > hive.version-> pom.xml
> >
> > https://github.com/apache/nifi/blob/rel/nifi-1.4.0/pom.xml#L105
> >
> >
> >
> >
> >
> >
> >
> > Wondering if there is a way for backward compatibility.
> >
> >
> >
> > Regards,
> >
> >
> >
> > [image: graph]
> >
> > *Milan Das*
> > Sr. System Architect
> >
> > email: m...@interset.com
> > mobile: +1 678 216 5660
> >
> > [image: dIn icon] 
> >
> > www.interset.com
> >
> >
> >
> >
> >
> >
> >
> >
> >
>

Re: Policies for root Process Group.

[Daniel Chaffelson]

Daniel,
What is your timeframe for needing this working?
I ask because the contributors to NiPyApi are very close to a release which
will provide by secured environment support, and wrappers to set this kind
of policy via a Python Client SDK for NiFi.
https://github.com/Chaffelson/nipyapi/tree/pr/43  is the branch this code
is currently sitting on, I would draw your attention to the
nipyapi.demo.secured_connection.py script which creates a secured Docker
instance, and then we are putting access/policy tools into
nipyapi.security.py, but you can also use the API commands directly via
nipyapi.nifi.*, such as nipyapi.nifi.PolicyApi().
You can also get the root flow with:
nipyapi.nifi.FlowApi().get_flow('root') as another example.

If you have specific requirements in this area please feel free to raise an
issue on the repo and we'll see if we can assist you.

Thanks,
Dan.

On Tue, Feb 27, 2018 at 2:46 PM Bryan Bende  wrote:

> Making a call to "/process-groups/root" should retrieve the root
> process group which should then have an id element.
>
>
> On Mon, Feb 26, 2018 at 5:20 PM, Daniel Hernandez
>  wrote:
> > Thanks Matt,
> >
> > I get now what is the problem, in order to exhaust all my possibilities I
> > may ask, is there a way using the API to get the root UUID from the
> > flow.xml.gz file? Because I see the file there after running the tests.
> >
> > Thanks,
> >
> >
> > On Mon, Feb 26, 2018 at 3:26 PM, Daniel Hernandez <
> > daniel.hernan...@civitaslearning.com> wrote:
> >
> >> Hi Matt,
> >>
> >> Thanks for your answer.
> >>
> >> Do you know if there is a way to preconfigure this value when running
> >> Nifi's Docker image? I am making the calls from an integration test that
> >> runs a docker container with the Nifi server. I already check and the
> value
> >> under  in the flow.xml.gz file changes everytime I deploy
> >> the container, I guess it is created at startup.  Is it possible that I
> can
> >> change my docker image to get a fix root group value?
> >>
> >> Thanks,
> >>
> >> Daniel
> >>
> >> On Mon, Feb 26, 2018 at 11:35 AM, Daniel Hernandez  >> civitaslearning.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> I am currently working on calling the Nifi REST API to get the 'root'
> >>> process group and setting it as parent for a new process-group.
> >>>
> >>> However I am getting the next messages:
> >>>
> >>> Attempting GET request to: JerseyWebTarget {
> >>> https://127.0.0.1:8443/nifi-api/process-groups/root }
> >>> 2018-02-26 11:06:55.341 DEBUG  --- [   main]
> >>> c.c.p.n.c.i.b.BootApiClient  :
> >>> 2018-02-26 11:06:55.341 DEBUG  --- [   main]
> >>> c.c.p.n.c.i.b.BootApiClient  : Received 403 response from
> GET
> >>> to JerseyWebTarget {
> https://127.0.0.1:8443/nifi-api/process-groups/root
> >>> }
> >>>
> >>>
> com.civitaslearning.platform.nifi.client.invoker.boot.exception.NifiForbiddenException:
> >>> No applicable policies could be found. Contact the system
> administrator.
> >>>
> >>> This is the content of my authorizations.xml file:
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>>  >>> resource="/flow" action="R">
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>>  >>> resource="/restricted-components" action="W">
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>>  >>> resource="/tenants" action="R">
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>>  >>> resource="/tenants" action="W">
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>>  >>> resource="/policies" action="R">
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>>  >>> resource="/policies" action="W">
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>>  >>> resource="/controller" action="R">
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>>  >>> resource="/controller" action="W">
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>>  >>> resource="/process-groups/root" action="R">
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>>  >>> resource="/process-groups/root" action="W">
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>> And this is the content of authorizations.xml
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>> file-access-policy-provider
> >>>
> >>> org.apache.nifi.authorization.FileAccessPolicyProvide
> >>> r
> >>>
> >>> file-user-group-prov
> >>> ider
> >>>
> >>> ./conf/authorizations.xm
> >>> l
> >>>
> >>> CN=civitas,
> >>> OU=ApacheNifi
> >>>
> >>> 
> >>>
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>> managed-authorizer
> >>>
> >>> org.apache.nifi.authorization.StandardManagedAuthoriz
> >>> er
> >>>
> >>> file-access-policy-p
> >>> rovider
> >>>
> >>> 
> >>>
> >>> 
> >>>
> >>>
> >>> And users.xml
> >>>
> >>>
> >>> 
> >>>
> >>> 
> 

Re: Re: 答复: Re: Is there a REST API to run a dataflow on demand?

[Daniel Chaffelson]

Hi Boying,

I have been working on a NiFi Python Client SDK that might help you here,
as the goal is to be able to replicate everyday actions taken in the NiFi
GUI as well as extending it for CICD/SDLC work.
For example with the following commands you would:

   1. get the reference object for a processor
   2. stop it if it is running
   3. change the scheduling period to 3s (or most other parameters)
   4. start it again


import nipyapi
processor_state_1 = nipyapi.canvas.get_processor('MyProcessor')
nipyapi.canvas.schedule_processor(processor, scheduled=False)
update = nipyapi.nifi.ProcessorConfigDTO(
scheduling_period='3s'
)
processor_state_2 = nipyapi.canvas.update_processor(processor, update)
nipyapi.canvas.schedule_processor(processor, scheduled=True)

If you need a different set of steps then please let me know and perhaps I
can help.
Those commands are currently in the master branch awaiting release:
https://github.com/Chaffelson/nipyapi

Thanks,
Dan

On Thu, Feb 22, 2018 at 7:41 AM  wrote:

> Thanks very much, I'll try your suggestions.
>
>
>
> 发件人:
> James Wing 
> 收件人:
> NiFi Dev List 
> 日期:
> 2018/02/22 14:05
> 主题:
> Re: 答复: Re: Is there a REST API to run a dataflow on demand?
>
>
>
> The NiFi API can be used to start and stop processors or process groups,
> and this might solve your use case.  But NiFi does not have an API to run
> a
> processor only once, immediately, separate from its configured schedule. I
> have solved similar problems in the past by creating two separate upstream
> sources - one for scheduled operation, and one for ad-hoc operation.
> GenerateFlowFile, GetFile, or similar processors can be used to inject a
> flowfile where you need to kick off the flow.
>
> Thanks,
>
> James
>
> On Wed, Feb 21, 2018 at 5:57 PM,  wrote:
>
> > Thanks a lot.
> >
> > But I want to know if there is a REST API that triggers a dataflow on
> > demand?
> > I don't find the API in the page.
> >
> >
> >
> >
> > 发件人:
> > Charlie Meyer 
> > 收件人:
> > dev@nifi.apache.org
> > 日期:
> > 2018/02/22 09:36
> > 主题:
> > Re: Is there a REST API to run a dataflow on demand?
> >
> >
> >
> > Yep, when you make the changes in the UI, open developer tools in your
> > browser and see what calls to the nifi api it is making then mimic those
> > with code.
> >
> > The nifi team also kindly publishes
> > https://nifi.apache.org/docs/nifi-docs/rest-api/index.html which help a
> > lot.
> >
> > Best of luck!
> >
> > -Charlie
> >
> > On Wed, Feb 21, 2018 at 7:34 PM,  wrote:
> >
> > > Hi, team,
> > >
> > > We set up several NiFi dataflows for data processing.
> > > These dataflows are configured to run once per day in the midnight.
> > >
> > > But sometimes, some dataflows are failed,I want to run the dataflow
> > again
> > > immediately after fixing the issue instead of waiting for running it
> in
> > > the midnight to
> > > make sure that the issue is really fixed.
> > >
> > > The only way I know to do this is to change the time of running the
> > > dataflow to the 5 mintutes from now for example
> > > and then change it back to midnight.
> > >
> > > It's a little inconvenient.
> > >
> > > Is there any REST API that I can use to trigger the dataflow on demand
> > > i.e. without change the time back and forth?
> > >
> > > Thanks
> > >
> > > Boying
> > >
> > >
> > >
> > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、
> 对
> > 外
> > > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知
> 发
> > 件
> > > 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。
> > >
> > >
> > > This email message may contain confidential and/or privileged
> > information.
> > > If you are not the intended recipient, please do not read, save,
> > forward,
> > > disclose or copy the contents of this email or open any file attached
> to
> > > this email. We will be grateful if you could advise the sender
> > immediately
> > > by replying this email, and delete this email and any attachment or
> > links
> > > to this email completely and immediately from your computer system.
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> >
> > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对
> 外
> > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发
> 件
> > 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。
> >
> >
> > This email message may contain confidential and/or privileged
> information.
> > If you are not the intended recipient, please do not read, save,
> forward,
> > disclose or copy the contents of this email or open any file attached to
> > this email. We will be grateful if you could advise the sender
> immediately
> > by replying this email, and delete this email and any attachment or
> links
> > to this email completely and immediately from your computer system.
> >
> >
> >
> >
>
>
>
>
>
>
> 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对外
> 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发件
> 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。
>
>
> This email message may contain confidential and/or privileged information.
> If you are not the 

Re: [EXT] Re: Embedded Nifi

[Daniel Chaffelson]

Charlie,
This was merged last week to provide an easy method to get around the host
header issue with running 1.5.0 docker.
Essentially it allows you to specify the host:port via environment
variables.
https://github.com/apache/nifi/pull/2439


On Tue, Feb 6, 2018 at 12:12 AM Charlie Meyer <
charlie.me...@civitaslearning.com> wrote:

> Hi Vincent,
>
> We do this for our test cases using
> https://github.com/palantir/docker-compose-rule. It works fairly well, but
> is a bit heavy weight to get tests started up locally. Also, with 1.5.0,
> the host header issue surfaces here and makes running in docker a pain.
>
> Hope that helps a bit!
>
> On Mon, Feb 5, 2018 at 6:07 PM, Vincent Russell <vincent.russ...@gmail.com
> >
> wrote:
>
> > Perhaps Dan, but integrating docker into an automated test might prove a
> > little challenging.
> >
> > I'll check it out though.
> >
> > Thanks,
> >
> > On Wed, Jan 31, 2018 at 3:00 AM, Daniel Chaffelson <chaffel...@gmail.com
> >
> > wrote:
> >
> > > Vincent,
> > > I do something along these lines in Python to test NiFi automation
> work.
> > > NiPyApi creates the requisite Docker container(s) for the test suite,
> > > procedurally creates the NiFi/Registry objects (Process Groups,
> > Processors,
> > > Buckets, Flows, etc), recursively tests itself, then tears itself down.
> > >
> > > I know this is not 'embedded', but could automated and containerised be
> > > close enough to meet your needs?
> > >
> > > Thanks,
> > > Dan
> > >
> > > On Wed, Jan 24, 2018 at 2:22 AM Peter Wicks (pwicks) <
> pwi...@micron.com>
> > > wrote:
> > >
> > > > Vincent,
> > > >
> > > > Embedded NiFi still has a long ways to go to be really useful, in my
> > > > opinion; and I don't know if anyone is actively working on those
> > > > improvements.
> > > >
> > > > The PR Andy mentioned simply allows you to startup NiFi inside your
> > > > process instead of running it directly from a startup script, but
> that
> > > > doesn't mean you magically have access to all of NiFi's internals
> > > (someone
> > > > can correct me if I'm wrong). If you want to actually interact with
> > your
> > > > new NiFi instance you will still need to use the REST API.
> > > >
> > > > Thanks,
> > > >   Peter
> > > >
> > > > -Original Message-
> > > > From: Vincent Russell [mailto:vincent.russ...@gmail.com]
> > > > Sent: Tuesday, January 23, 2018 03:07
> > > > To: dev@nifi.apache.org
> > > > Subject: [EXT] Re: Embedded Nifi
> > > >
> > > > Thanks Andy,
> > > >
> > > > This looks a great first step.   It would be nice to have a builder
> > > pattern
> > > > and the ability to download the "executable" from a nexus or the
> local
> > > > filesystem like embedded elastic, but perhaps that might be better in
> > > some
> > > > third party library.
> > > >
> > > > https://github.com/allegro/embedded-elasticsearch
> > > >
> > > > -Vincent
> > > >
> > > > On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto <alopre...@apache.org
> >
> > > > wrote:
> > > >
> > > > > Vincent,
> > > > >
> > > > > I plan to merge this pull request [1] for NIFI-4424 [2] by Peter
> > > > > Horvath today. Does this satisfy your requirements?
> > > > >
> > > > > [1] https://github.com/apache/nifi/pull/2251
> > > > > [2] https://issues.apache.org/jira/browse/NIFI-4424
> > > > >
> > > > > Andy LoPresto
> > > > > alopre...@apache.org
> > > > > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* PGP
> > > > > Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > > > >
> > > > > On Jan 21, 2018, at 7:35 AM, Vincent Russell
> > > > > <vincent.russ...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > Devs,
> > > > >
> > > > > Does an embedded nifi exist that would start a nifi with a provided
> > > > > workflow?   I am aware of the Mock framework, but I am looking for
> > > > > something for integration tests.
> > > > >
> > > > > Thanks,
> > > > > Vincent
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [EXT] Re: Embedded Nifi

[Daniel Chaffelson]

Vincent,
I do something along these lines in Python to test NiFi automation work.
NiPyApi creates the requisite Docker container(s) for the test suite,
procedurally creates the NiFi/Registry objects (Process Groups, Processors,
Buckets, Flows, etc), recursively tests itself, then tears itself down.

I know this is not 'embedded', but could automated and containerised be
close enough to meet your needs?

Thanks,
Dan

On Wed, Jan 24, 2018 at 2:22 AM Peter Wicks (pwicks) 
wrote:

> Vincent,
>
> Embedded NiFi still has a long ways to go to be really useful, in my
> opinion; and I don't know if anyone is actively working on those
> improvements.
>
> The PR Andy mentioned simply allows you to startup NiFi inside your
> process instead of running it directly from a startup script, but that
> doesn't mean you magically have access to all of NiFi's internals (someone
> can correct me if I'm wrong). If you want to actually interact with your
> new NiFi instance you will still need to use the REST API.
>
> Thanks,
>   Peter
>
> -Original Message-
> From: Vincent Russell [mailto:vincent.russ...@gmail.com]
> Sent: Tuesday, January 23, 2018 03:07
> To: dev@nifi.apache.org
> Subject: [EXT] Re: Embedded Nifi
>
> Thanks Andy,
>
> This looks a great first step.   It would be nice to have a builder pattern
> and the ability to download the "executable" from a nexus or the local
> filesystem like embedded elastic, but perhaps that might be better in some
> third party library.
>
> https://github.com/allegro/embedded-elasticsearch
>
> -Vincent
>
> On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto 
> wrote:
>
> > Vincent,
> >
> > I plan to merge this pull request [1] for NIFI-4424 [2] by Peter
> > Horvath today. Does this satisfy your requirements?
> >
> > [1] https://github.com/apache/nifi/pull/2251
> > [2] https://issues.apache.org/jira/browse/NIFI-4424
> >
> > Andy LoPresto
> > alopre...@apache.org
> > *alopresto.apa...@gmail.com * PGP
> > Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > On Jan 21, 2018, at 7:35 AM, Vincent Russell
> > 
> > wrote:
> >
> > Devs,
> >
> > Does an embedded nifi exist that would start a nifi with a provided
> > workflow?   I am aware of the Mock framework, but I am looking for
> > something for integration tests.
> >
> > Thanks,
> > Vincent
> >
> >
> >
>

Re: Programmatically adding processors

[Daniel Chaffelson]

Hi Phil,
It's not part of the official NiFi project, but I have been building a
python client to do exactly the kind of changes in NiFi that you describe.
You can find it at https://github.com/Chaffelson/nipyapi
And the function to update a processor's config via API call is here:
https://github.com/Chaffelson/nipyapi/blob/18142f2a03aebac14be75b607eb35074efa6aa7f/nipyapi/canvas.py#L377
There are also calls to stop and start the processor in order to effect the
change.

Happy to receive feature requests and PRs.

Thanks,
Dan.

On Tue, Jan 30, 2018 at 2:01 PM Chad Zobrisky  wrote:

> Phil,
>
> The chrome dev tools option is definitely helpful with building request
> order and looking for the calls you need to make.  There is also nifi rest
> docs[1] available online.
>
> 1. https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>
> Chad
>
> chad@nifi.rocks
>
>
>
> On Mon, Jan 29, 2018 at 9:47 PM Phil H  wrote:
>
> > Thanks Robert!
> >
> > Cheers,
> > Phil
> >
> > > On 30 Jan 2018, at 13:35, Robert R. Bruno  wrote:
> > >
> > > In the past, I've used the Chrome developer tools network view to see
> the
> > > api calls made by the NiFi UI.  Just manually go through the steps you
> > want
> > > to automate in the UI and view the REST calls made.
> > >
> > > Hope that helps.
> > >
> > > Robert
> > >
> > >> On Mon, Jan 29, 2018, 21:31 Phil H  wrote:
> > >>
> > >> Hi there,
> > >>
> > >> I need to add new properties/routes to RouteOnAttribute and subsequent
> > >> PutFile processors based on an external input (a basic web interface).
> > >>
> > >> Hoping someone can point me in the right direction in terms of either
> > >> config files to change, or API calls to make to simulate what the NiFi
> > UI
> > >> would send through.
> > >>
> > >> Thanks in advance,
> > >> Phil
> >
>